
How a sysadmin can become a programmer - sea6ear
http://cuddletech.com/blog/?p=817
======
Aqueous
Great advice except for this bit:

"Step 3: Scripting isn’t programming."

I have to disagree that scripting isn't programming. To me it is no less
programming than using a library, except for you are interacting with that
library through stdin and stdout instead of a direct function call. If there
are no libraries available for a single user tool, then execing is your only
option. UNIX is designed to pipeline many different small tools that do one
thing very well together. What's the difference between doing that at the
command line vs doing it in a script? Sometimes it's just the most efficient
way to program something. And with the use of exec tools can get quite
powerful quite fast.

~~~
pacala
> What's the difference between doing that at the command line vs doing it in
> a script?

Testing. Clean consistent APIs. Dependency bloat.

~~~
toomuchtodo
Because my 50-150 line bash scripts need that sort of overhead.

~~~
njharman
Perhaps not. But that is why they are scripting and not programming.
Programming is a large discipline. Writing code is small part.

Scripting is not programming in same way clicking through install wizard is
not system administration.

~~~
waps
There is still a massive difference between developers and sysadmins even in
the devops era. You can make a caricature of both professions to point out the
difference.

A sysadmin and a developer walk into a bar. They decide each to make a website
doing order management and return in a month to demonstrate both solutions.

First the developer gets his turn. There's a massive version history, which is
shown. After 15 minutes of browsing through commit messages and pointing out
test cases and "look how easy it is", everyone in the cafe requests to be
shown the actual site. After pointing out there's more LoC dedicated to unit
testing the code than there is actual code, the programmer gives in and opens
a web browser. "Could not connect to site". The programmer forgot to start his
solution's web server. He logs into a shell, gives the command. After 3 tries
he gets the command right, and it won't start. It is quickly noticed that
there is a bug on the top of the main function preventing it from starting. He
quickly fixes it. The website has every feature you could possibly want and
more than a few you wouldn't want, and looks beautiful, if a bit "full". And
every individual feature works beautifully and perfectly as designed, and then
2 or 3 features have to work together and an hourglass appears. After 5
minutes they give up the wait, and the programmer digs into the code and
concludes that it was completely unreasonable for the sysadmin to ask for a
demonstration of picking a product and while a mail notification service was
enabled, something which was never in the requirements ! Getting a report out
of the system takes the better part of the day. Before the demo can go further
the web server crashes and not only did the programmer lose the entire order
database, but the versioning system, including all the code, and surprisingly
the crash also managed to take out the entire contents of the programmer's
gmail inbox, X-COM savegames and his drink is suddenly empty. Only the drink's
disappearance can reasonably be blamed on the sysadmin.

Second, the devops/sysadmin turn comes round. His website has been running for
a long time, in fact he says he didn't bother restarting it to add the last 10
features he added to the site. The programmer (and everyone who isn't the
sysadmin) has serious trouble distinguishing the website from a terminal
window, or indeed, from an unpowered laptop. Tiny gray letters on a black
background list the orders, as confirmed by an inspection of the sysadmins
laptop screen with a magnifier glass and extreme brightness settings. They are
preceeded with 10 letters that can be either small caps, or capitals
illustrating the various properties an order might have. Oh and it can also be
one of 2378 other letters that indicate various things, but even the sysadmin
himself has forgotten most of them. There is one test, which starts up the
system, and runs 1000 orders through them. The "RoundFloat" function in the
code turns out to return a string, "dog", but is never called. There is no
reporting functionality, insists the programmer. The sysadmin disagrees. Look
! echo 'cat orders/ __/??d _.2013._.xyzz2??3?? | sed " _& (_)&^#@ _\ &^\\\@#"
| grep | perl -pe "lqwtq8_(&()lj;lkd" | mail jef@boem.com' >> /etc/cron.daily
adds daily reporting of all orders in the asian region over $220 to the system
! The sysadmin insists that this is simple, as he got it right the first time.
When a person standing at the bar is asked to enter an order into the system,
he tries for 10 minutes, then knifes the sysadmin in frustration, before
retreating to an optician for damage control. The "web server", which turns
out to be a bash script, has never spent more than .0003 picoseconds serving
the page. The page ? Yes, there is only one after all. Why would an order
tracking system need more than one page ? It turns out the webpage page is
text/raw, not text/html, after which a baffled sysadmin enquires what's wrong
with that. An inspection of the versioning system reveals 3 commit messages
"start", "middle" and "end", and nothing else. It's replicated to 5 different
datacenters, though. Look ! An offsite backup was initiated as soon as an
order was entered. The programmer, still angry about his crashed server, takes
an axe to the sysadmin's laptop, which surprisingly remains operational after
being hacked in 10 pieces, set on fire, and fed to a local dog. The sysadmin
declares that he's very proud of the system's features, you know, both of
them.

The whole thing ends in a barfight between the two, the sysadmin insisting
that a non-working website has 0 features, even if the programmer implemented
3000 "well-tested" ones, chuckling at the "tested" word. The programmer was
heard saying that a single black screen does not an order system make, before
breaking a beer bottle and attacking, with help of the person that was asked
to enter an order into "the black hole".

------
karlkatzke
I'm a sysadmin who used to be a coder. I know a lot of people who have gone in
either direction.

The reason I stopped coding is that I got into it to solve problems, but I got
tired of spending what felt like 90% of my time writing unit tests, wrestling
with frameworks, and solving problems with upgrades to the huge webs of
dependencies for even basic projects. In other words, I got fed up with what
coding has become. It's no longer hacking. It's toolchain management.

I'd always been a full stack developer, so I just started focusing more on the
sysadmin side and gradually transitioned over there. I can manage toolchains
just fine if I want to focus on that. My coding experience is handy when we're
trying to troubleshoot a bug that only shows up on production ... uncommon,
because our dev and QA groups are GREAT, but it still happens. I not only know
where all the logs are, but I'm able to read the code and make sense of stack
traces from the different languages we use.

Those who think what I've done is ridiculous don't understand how scripted
everything that happens in the sysadmin world these days. Everything in
infrastructure/operations is scripted except the troubleshooting -- and even
that stuff gets scripted as soon as it's done so that it doesn't pop up again.
Yes, the native language of these tasks is bash, but ruby and python are
becoming really popular. So's haskell, for those who can speak it. What's
disappointing me right now is that all of the chef stuff we're doing is
starting to get complicated enough to have a big toolchain (thankfully managed
right now with bundler) and starting to grow unit tests... Didn't I make a
career change to get away from this crap?

I think I'll go be a DBA next. ;)

~~~
hosh
That's a weird association between "unit tests" and "toolchain management".
Did you feel it was a tedious part of development or what?

~~~
ams6110
To me, unit tests are the ultimate in tedium. I appreciate the value, but
writing them is certainly about the dullest part of the job.

~~~
hosh
Thanks for the response :-)

I know a fair number of developers who find ops pretty crazy, considering that
a good chunk of it can be automated. So I suppose the other side of tedium is
the unnecessary risk in having to hand-perform things repeatedly.

~~~
karlkatzke
houk got most of the answer. The other answer is that ops, unlike dev, still
gets to do things the fast way if it makes sense. So if it's faster to hit up
and enter a bunch of times? Do it. Dev doesn't get to make that decision most
of the time these days, and that's just sad and completely against the spirit
of hacking.

~~~
hosh
There's hacking, and then there's polish. You can't make a product out of a
hack. Though some companies offer Hack Days to get things going.

------
oneeyedpigeon
This whole article is full of prejudice, narrow-mindedness, and over-
generalisation. I'm not a sys-admin myself, but I have known some, and they
certainly didn't fit this stereotype.

~~~
shawn-furyan
Yeah, I worked for 2 years as a sysadmin, and now I'm working on building a
website in one of those scripting languages that apparently don't count as
programming, so perhaps I have a better perspective than most on this. The
strange conceit of this article is that all sysadmins apparently want to be
programmers but just lack the self confidence to actually do it. I think that
this is pretty wide of the mark. If anything, sysadmins know how much more
stable and reliable mature tools are than the sort of half baked solutions the
author is recommending that they push themselves to develop. When you spend
your entire day supporting fragile solutions, your first inclination is not
usually going to be to put more of those into the world.

~~~
CodeMage
_now I 'm working on building a website in one of those scripting languages
that apparently don't count as programming_

The article didn't claim that scripting languages don't count as programming.
Despite the heading "scripting isn't programming", it doesn't even really
claim that scripting isn't programming: it claims that if you switch from
shell scripting language to a compiled language, but you still do most of the
stuff by spawning child processes, you're most likely using the wrong tool for
the job.

~~~
shawn-furyan
Forgive me for countering hyperbole with hyperbole. She did baldly say in the
main content of her article "Scripting isn't programming". It's not a
hyperbolic title, it's in the body of the article. She also doesn't counter
that assertion. The article is hyperbolic through and through, and I don't
care to argue its finer semantic points since the original article was neither
nuanced nor subtle.

------
CrLf
Let's just, for a moment, entertain this radical notion that many sysadmins
are actually programmers that just don't want to do software development as a
profession. I know that, for most people here, this is very strange: who would
choose to avoid the exciting world of programming having the skills to do so?
But indeed, it happens.

For someone that likes to solve problems, 95% of software development is mind-
boggingly boring. It's mostly looking at the same codebase day-in and day-out,
while having to deal with shifting requirements. All of this to end up with
some application that you mostly wouldn't use yourself. This, if you're lucky
that your project doesn't get cancelled before it sees the light of day, or it
doesn't get delayed so much that you get so bored that you move on before
seeing it to completion.

And the consultancy... Parachuting into projects at clients and being
extracted by helicopter when "it's done" (which is usually when it meets
minimum requirements to receive payment, not when it is at all in a state to
be proud of).

Really, 95% of software development isn't glamorous stuff at NASA or the next
big social network(1).

Systems administration, provided you are self-motivated, can actually code
your way out of a hole, and are willing to deal with the occasional user,
provides plenty of space to solve problems with programming. With
_programming_ , not IDE and framework juggling.

Sure, it can be boring at times and can be stressful at others, and you have
95% chance of being in the exact same environment where you wouldn't like to
be coding in. But at least you have more autonomy and can actually _solve_
problems.

I'm a programmer by training and a systems administrator by profession. I
would most likely rather be programming than taking care of systems and
databases, but only if I found something in those remaining 5% that I would
actually enjoy doing.

Otherwise, it would just burn me out from programming altogether, and I like
programming...

(1) Actually, developing the next big social network is probably as boring as
everything else I said. Who in hell wants another social network?

~~~
j45
Well said. Programming is solving problems with (hopefully) thinking and
problem solving skills that improve every time you do something.
Differentiating between coding/engineering/programming/scripting is all a moot
point where it counts -- creating value and solving problems.

No matter how perfectly you do or don't architect something, it will have a
failing point, just like the limits of hardware when you buy them. There are
some very general thinks you can do to be kind to your future self
architecturally, but just as many you can do to hurt yourself.

I think edw519 said it best, clever architecture can be far more important
than clever code.

------
dsr_
Hi, I'm a systems administrator. I'm reasonably senior. I can tell you that
pretty much all sysadmins code. In overlapping sets, some are good at it, some
don't need to be good at it, and some like solving problems that way. But
being a sysadmin is a different job from being a software developer, and
people who are good at one and not the other are much more common than those
who are good at both.

The nature of systems administration work falls into two broad categories. The
first is small problem work: a user has a problem, a machine signals an alert,
something isn't working. The second is, of course, large project work: users
need a new service, the alerting system needs to be replaced, new
infrastructure needs to be built. Most of the time, but not always, the small
problems need and have immediate solutions, and the large projects are not as
time-critical and require some amount of research, budgeting, selection,
configuration and support before they can be deemed complete.

A sysadmin who has specialized into a particular problem domain will need the
skills appropriate to that domain. Sometimes that means picking up a new
configuration syntax -- which may be a complete language. Sometimes it means
building better communications skills, so that you can diagnose what your
users are complaining about. Whatever it is, you can assume that the people
who specialize in an area that doesn't require much programming, don't do
much.

Generalist sysadmins, on the other hand, need to be competent at a wide
variety of skills, including two or three languages. That doesn't mean that
they need to be UX designers (but they need to recognize a UI failure) and it
doesn't mean that they need to be database administrators (but they need to
understand how to back up and restore the databases they support, and figure
out the replication system and find single points of failure). They do need to
be able to write short programs to automate what they would otherwise do
manually, to figure out what the debug logs are really complaining about, and
to understand computer systems from electrical and environmental needs through
virtualization systems and configuration management.

Finally, there are sysadmins who are also developers, who announce that they
don't like any of the existing mail programs and so they will write a new one;
who determine that there's no configuration management system that tracks and
removes subsystems as cleanly as they would like, and so a new one will be
needed; who write filesystems or music organizers or browser extensions
because that's what they need and want and are good at.

The industry, nebulous quagmire that it is, needs specialists, generalists,
and dev-ops hybrids. Making false generalizations merely reveals a bizarre
lack of awareness.

~~~
WestCoastJustin
I'd just like to mention something about specialists vs generalists. Over my
career, as a sysadmin, I've noticed this trend. At smallish organizations
(i.e. startups), you'll see almost exclusively generalists, because the
company simply is not large enough to have a team of specialists running
around. As the company grows (think fortune 500 companies, universities,
government, etc), these generalists will have to make the transition into
specialists, say for example into, security, networks, email, linux admins,
windows admins, storage, racking/stacking/cabling, etc. You might even have
these enterprise architects popping up.

I have also seen less sysadmin programming (whatever that is) at the larger
companies, because they will purchase prepackaged products that fit their
need.

~~~
ownagefool
To counter, I worked at a small web hosting company between 2007 and 2010 who
were responsbile for creating Europe's first cloud computing infrastructure
and the worlds second, according to them at least.

They had about 9 sysadmins go through their doors, a maximum of 7 at any one
time with a max of 4 coders. Of those system guys I'd hazard a guess that
around 50% of them couldn't code at all and only really two could get anything
beyond procedural scripting, if that.

Not to disrespect these guys because some of them were between good and really
good at what they did do, but there were also decisions such as to rely on
expensive SANs and proprietary software to tackle issues that could have been
and later were replaced by internal code because, in my opinion, these guys
were unable to comprehend automation on a scale that a developer could unless
they already knew of software available to do such a thing.

After that I worked for a massive hosting company with much more segregation
of departments and as far as I'm aware operations didn't code despite the
deparment being a lot larger than the previous company.

It's funny because back then I was arguing with both these companies that
operations should be driven from an automation stand point utilising devs
skills but it wasn't exactly a great point since I was the only developer on
staff for much of that time in the first company and the second wasn't
interested at all.

For the record, I'd like to be a devops but I'm pretty sure I get shut down
from those roles from a percieved lack of operations experience and buzzwords.
Much of that because I'm actually honest and my experience of sysadmins are
that they were gatekeepers unwilling to so much as let me login to some of
their systems. Meh.

I'm sure theres a lot of sysadmins out there who a very capable software
developers but I think it's a stretch attempting to call often simple
configuration languages and extremely short glue scripts programming. But hey,
as long as you're adding value, who cares what we're calling it.

------
wisty
Most sysadmins aren't LAMP (or LNMP, or LNPR, or whatever the flavor of the
day) programmers, who know how to make a todo list "pop". They aren't
javascript ninjas, who actually know how to interact with the DOM. They don't
know how build models of black holes, or self-driving cars, or make enemy
sprites hunt you down in an interesting way, or debug a GUI.

Sysadmins _do_ program. Their programs tend to be flatter (scripts, not
frameworks), but that doesn't mean they aren't programmers. They probably
can't build great frameworks, but not a lot of programmers can (but sadly,
they try - every big corporate project seems to grow into a badly thought out
framework).

If you pick any domain that's not "compiler design", you can probably make a
good case that they aren't really programmers.

~~~
michaelochurch
_Most sysadmins aren 't LAMP (or LNMP, or LNPR, or whatever the flavor of the
day) programmers, who know how to make a todo list "pop". They aren't
javascript ninjas, who actually know how to interact with the DOM. They don't
know how build models of black holes, or self-driving cars, or make enemy
sprites hunt you down in an interesting way, or debug a GUI._

Most software engineers couldn't do any of those tasks to the typical, silo-
enforcing business asshole's liking (i.e. it needs to be done yesterday). That
shouldn't matter. What matters is _the ability to learn_.

------
hhw
Sure, most people working as sysadmins can't code. Then again, most people
working as programmers can't code either. People who can really write good
code are few and far between.

The very best sysadmins are also good coders, and the very best coders are
also good sysadmins. The foremost experts on any operating system are the
developers that work on them after all. You can never fully understand an
operating system without reading the code, and you can't write the most
efficient code if you don't fully understand the entire stack the application
runs on, operating system included.

As for using tools instead of writing code, what it comes down to is what's
the best solution for the problem at hand. A good sysadmin doesn't need to
memorize ways to use tools; they know each of the tools well enough to make
full use of them. If a task can be done with a one-liner, and has a fixed
amount of work to be done, why would you spend more time to implement the same
in a programming language, when the execution time is insignificant relative
to the development time? Considering the slower higher level languages that
are popular nowadays vs the standard tools written in low level C, the
execution time may actually end up slower despite the execution savings in
running a single process instead of many.

~~~
tete
> The very best sysadmins are also good coders, and the very best coders are
> also good sysadmins.

I can agree on the first, but the second while surely true in cases is maybe
wrong for people doing something completely different, like are more into the
academic areas, in really abstract levels, doing big data stuff in completely
managed environment.

Think about planes. Someone who crafts and maintains them and is close to the
metal, maybe is a really good pilot too, because he knows planes in and out,
but while really good pilots probably know a lot about planes, they may be
focusing on stuff that is really, really far away from crafting one. Still
they could be among the most excellent pilots.

But then who is who, might depend on whom you ask.

I just really find it funny and that's something that kinda came across in the
article, how because sysadmins are frequently good in a field of programming
that by programmers is considered a field of masters, which are operating
systems and all this low level stuff that many programmers have to get to at
some point. In a way sysadmins have to do and shell scripts are pretty high
level. But it also really shows how it is a lot of two different mindsets.

------
dobbsbob
I've never met an openbsd/freebsd, solaris or old school unix sysadmin that
didn't have a total mastery of C/++ and scripting like perl. By total mastery
I mean debugging and even reverse engineering binaries looking for something
that was eating memory, and often completely rewriting some major program
parts to add security like old smtp and sendmail. The Sr admin I worked for
could quote K&R by memory and was a much better dev than the software
engineers there just he was a total neckbeard and didn't want to deal with
"politics" so confined himself to the sysadmin dungeon. He made more money
than them too

Just look at openbsd-dev mailing list. Most of those guys are sysadmins, like
the guy who wrote OpenBGP from scratch, the pf maintainers... basically all
Berkley forks were created by sysadmins.

~~~
4ad
Do you see many OpenBSD or "old school" sysadmins? I mostly see kids that
haven't touched anything but Debian and Ubuntu, think RHEL is too old school
and never heard of fork(2).'

Good sysadmins obviously exist, but in my experience they are a shrinking
minority.

~~~
dobbsbob
I think sysadmins period are shrinking. Most companies are just signing up to
cloud products and using feudal security. The only jobs I've seen locally here
advertising for sysadmins required that you had fluency in C/python.
Everything else is just a $14/hr desktop tech support position.

------
incision
As a set of bullets, I can agree with parts of this list, but the supporting
paragraphs of weird assumption and baffling generalization, not nearly as
much.

The phrase "I'm not a coder" can mean all kinds things coming from a sysadmin,
context is import.

If a sysadmin truly lacks the ability/confidence to write useful code, I'd
expect has more to do with the typical nature of the job and organizational
divisions than prejudices, hang-ups or fickle behavior of the sysadmins
themselves.

A role which is too often reactive doesn't leave much time for development.
Likewise, development towards the goals of a sysadmin (automation, insight,
reliability) too often lead sysadmins to unemployment due to misunderstanding
as the relationships between sysadmins who "don't do anything" and systems
that don't break.

I expect new businesses, particularly the type common on HN could be
exceptions, but in many organizations the preferred language of system
administration and those of software development don't align. As a result,
serious sysadmin code written in say Python as opposed to corporate standard
of .NET/Java is putting the sysadmin on an island and likely to be seen as a
threat/liability.

------
jrussbowman
Couldn't help but write a response -
[http://joerussbowman.tumblr.com/post/54602688494/why-
sysadmi...](http://joerussbowman.tumblr.com/post/54602688494/why-sysadmins-
dont-code)

~~~
hosh
This divide is a really fascinating to me. I feel like I'm watching a National
Geographic special.

"We don’t want to learn the latest IDE and develop best practices for working
with a version control system with other developers. Code reviews, strategy
meetings, QA reviews… what? No thanks, let the programmers get that stuff
done. They have 7 hours to commit to this project today, I have 45 minutes and
that’s only if all my KLO goes well."

Why do you feel this way? (Not a rhetorical question). Looking at it
superficially, I would guess that:

(1) You don't like the engineering part of software development, finding it
too tedious or something? (2) You feel like you're more badass for having to
get shit done in a much smaller time frame.

I read through some of the other responses and it seems to fall into this
general pattern.

~~~
incision
Not the parent and I don't totally agree with his position, but I can
certainly relate.

 _> (1) You don't like the engineering part of software development, finding
it too tedious or something?_

I find the engineering of software fascinating. I've found the glacial pace
and ceremony of most software development teams I've been exposed to
unbearable.

I can't imagine this is intrinsic to software development, more likely a clash
of cultures and eras.

For example, during my longest stint in a sysadmin role (~10 years) I
supported and migrated between a continuously shifting mix of OS/2, Windows,
AIX, Solaris, NetWare and all sorts of Linux speaking SNA, IPX, TCP/IP on EoC,
TR etc.

I did this while accommodating and often clashing with internal developers who
had been working the same codebase the entire time.

Again, I don't mean to present this as something necessarily true of all
development vs administration as a whole, but I've seen it and heard similar
experiences related enough to think that it's not uncommon.

 _> (2) You feel like you're more badass for having to get shit done in a much
smaller time frame._

Well, "badass" seems a bit condescending.

When a sysadmin needs to get things done in 20 or 45 minutes, there's a good
chance his job depends on it. Though they generally wouldn't choose to put in
positions like that, I have found that people who excel in the role are the
ones who can stay cool, thrive under intense pressure.

~~~
hosh
Interesting. Seems to me, the main thing is the pacing.

So what about, for example, video game development? It's infamous for
development to be in continuous state of crunch time.

~~~
incision
Coincidentally, one of if not the best programmer I've worked with was
previously a programming sysadmin for a big name in the games industry.

His stories corroborate what is often heard about gaming companies - that the
continuous crunch time isn't necessary so much a matter of exploiting an
endless supply of young, enthusiastic talent.

I'd certainly see myself having issues with that kind of environment.

The fast iterating development culture of many Internet-era companies
certainly sound nice, but never having worked in one, I'm not sure how of the
hype to believe.

~~~
hosh
I don't work for a game company, but yeah, I agree that the crunch time
doesn't work well. I've read articles about the correlation between that kind
of crunch time programming, and the general maturity of the artform. That is,
the guys (I remember the industry being dominated by males) who raise a family
and enter that stage of their life end up dropping out of the gaming industry.
Thus, games tend to be made by guys who haven't had that life experience,
willing to go into crunch mode, etc.

I brought it up though as an example of pacing. I'm trying to discern whether
it is the pacing itself, or whether it is the programming jobs you have tried
have been with organizations that have a lot of red tape, or whether it is
simply the engineering discipline that you prefer not to participate in.

As for the Internet-era companies, the fast iteration is something you tend to
see more in startups rather than startups that have matured into big business.
Iteration for the sake of iteration itself doesn't work very well either (John
Boyd "fighter pilot agility" vs. Agile Manifesto). It's also where devops is
coming out from, and proliferating back into mainstream IT.

------
nnq
The way I see it, in:

 _Age 0_ , all computer users where actually _users_ and _sys admins_ and
_programmers_ and _EEs_ , then, when computers stopped being room-sized
monsters, in

 _Age 1_ , all users were _users_ and _sys admins_ and _programmers_ (the Lisp
machines era is probably at the end of this age), then, when software became
actually usable by most non-programmers, in

 _Age 2_ , all users were _users_ and _sys admins_ (the UNIX era falls in this
age), then, in

 _Age 3_ , users started to be only _users_ \- or, more exactly, people who
were not programmers and sys admins started to actually use computers on a
large scale (the current era of Windows and Macs and everything else) - since
sys admins separated themselves from the users group long after the
programmers have, they are "philogenetically" closer to users than to
programmers ...also, this is why I think the "Linux desktop" dream took so
long to rise up, becase Unix systems by design ask users to at least be sys
admins too, and reward them to the max when they are sys admins and coders
too.

 _Now, if anyone has any idea of what comes next, please share the
enlightening thoughts :)_

The only witty continuations I can think of is: Age N - humans are no longer
"using" the machines and they haven't done this for a long long time (well,
they do, but only in the sense that a plant uses the sun, water and earth to
survive), they are being used by them, grown in "human natural reservations"
for scientific study purposes and historical education purposes for machines
to remember how the beings that preceded them were like ...but hopefully
someone else has a brighter view on the future.

~~~
ams6110
You missed the mainframe era where from at least the 1970s forward you had
office workers by the score who were only users.

~~~
nnq
You're right, I think one may say that mainframes did the Age 2 -> 3
transition first, as it was kind of necessary for their use care, and then PCs
recapitulated history...

------
tehwalrus
small point: the author isn't saying "scripting languages aren't programming
languages", nor are they saying "you cannot write programs in scripting
languages" \- they're saying that _scripting_ \- a particular style of
program-writing - isn't programming, or put better "proper programming" which
is arguably true, especially if you're writing bash or IPython scripts, where
you have access to and make heavy use of shell commands.

That style of programming is very very different to regular compiled programs,
and practicing one and never the other _will_ lead to a bias, and relative
incompetence in the other. This is why I'm rubbish at chaining unix commands
together, because whenever I'm confronted by a problem that might need
something that complex, I just use python. I have the reverse problem to the
rhetorical sysadmin in the article.

~~~
popee
The point of sysadmin scripting is to automate stupid and repetitive jobs.
They are payed for that kind of programming. Also, it can be viewed as kind of
_hacking_ -> word that is not popular in most of enterprise environments. You
know it works, leave it alone :-)

~~~
Cobbler
Exactly, because sysadmins have time and luxury combined with training in
programming, they know how to make their jobs easiy.

------
tmcw
Why Programmers Can't Use Apostrophes Correctly

~~~
rythie
Or control comment-spam

------
popee
Some of best programmers i have met are, watch this out, sysadmins. And no,
they will not do your job :-)

On the other hand most of developers' knowledge is like swiss cheese, because
they don't know enough about underlying machine, they only want to deal with
abstractions and algorithms, so for example trivial things like where is
located config can be really tricky for them. As you can see, one could do
same article about developers, but what's the point on spitting on people you
work with? Ah, yeah, frustration :-)

------
stephengillie
This actually describes how I have approached system administration over the
past few years. I'm just now trying to change that. I'm embarrassed to say,
but I'm writing HTML for the first time -- in 2013, not back in 2003 when all
the cool kids were.

OK so I'm not a cool kid anymore. But I can be again, I just have to slog
through some tomes and actually build something cool. Isn't that what being a
nerd is all about?

~~~
alrs
The cool kids were writing HTML in 1993.

[http://www.w3.org/MarkUp/draft-ietf-iiir-
html-01.txt](http://www.w3.org/MarkUp/draft-ietf-iiir-html-01.txt)

------
od2m
Hi, I have been a Solaris/SGI admin (2 years) and software engineer (10 years)
in my life. The reason why sysadmins can't write software is this: Every day
sysadmin stuff like pulling cable, installing OS's, fixing printers does not
require someone whose super intelligent. But real IT work-- setting up domain
controllers, running a global corporate network, running a cloud service, etc
takes a fair amount of brains yet a lot of companies compensate their IT
departments as if they're well paid cable monkeys.

The skill sets for a good programmer and a good sysadmin overlap somewhat. But
you can make MUCH MORE MONEY writing software. In the early 2000's when I was
a sysadmin jobs in my area were paying 40k-60k a year. Programming jobs were
paying 60k-100k. So basically, poorly payed IT positions select out anyone who
could actually do a good job at them because those very people can make more
money writing software

------
ancarda
>SysAdmin’s are purists and have a lot of ego. Using a module or library is
akin to cheating.

I'm a programmer and I know how this feels. It's not about "cheating". To me,
I view it as "I can't be bothered to learn /how/ this works, I'll just use a
library". I can make a half-assed HTTP library, but in production, I'd
obviously use a real library. The important thing is I understand on a more
intimate level how HTTP works. That is invaluable in debugging as when I ask
myself "why isn't this working", I can think back to my own code and it often
helps to shape my understanding. I can think how the machine thinks as I
taught it how.

I'll often study other systems while building my own. It's all part of the
learning process to me.

It also provides deep appreciation for the effort that's poured into
libraries.

I've also done the same thing on the web; I used XMLHttpRequest and DOM apis
for years before I started using jQuery. I've never had a single complaint but
I've seen programmers who have gone right into jQuery and surrounded
themselves with abstraction and have had no idea how the system is behaving
and they get confused and frustrated. This leads people like me to view
libraries as bad; they are preventing people from learning how their systems
work.

Programmers today seem so lazy. I'm not really sure if that's the right word.
I see people using HTML 5 Boilerplate and they never seem to ask "Why is the
meta charset before the title tag?". I did. Now I know about UTF-7 XSS
vulnerabilities. I spent most of my day reading up on charset vulnerabilities.
With that knowledge, I patched several projects of mine. Sure, people using
Boilerplate would be protected but they wouldn't really know what from.

Perhaps I just think differently to other programmers. Maybe they just want to
get on with their work while I obsess over "how does this work?!"

~~~
vacri
It's not just that it's cheating, it's that it's less portable as well. A
self-contained script can be thrown anywhere. One that needs a library needs
to have the library taken with it. One thing that 'sysadmins' encounter than
'programmers' don't is a wide variety of systems with a varied install base.
The more portable your coding, the easier time you have as you progress.

Throw in that programmers work on more monolithic projects, getting the
libraries for those is (kind of) a one-time affair, but with sysadmins working
on smaller, multitudinous projects, you can end up with a raft of different
libraries to manage.

The real answer, as always, is 'be appropriate'. Often libraries are called
for, but sometimes they're to be avoided. It depends on the use case, or even
the library itself (we're currently using an abandoned one in production...
and there's some crossing-fingers going on as we don't have the resources to
fix it if there's breakage...)

------
vonskippy
And the flip side, why coders can't do the simplest of sysadmin tasks.

I've meet coders who couldn't spell "DHCP" let alone tell you what it does or
how it works.

They're two completely different job descriptions - why should one need to do
the other?

~~~
alephnil
They are different but also related.

Programmers that don't know basics about network are very often poor
programmers as well. Most programmers must be able to set up build servers,
test environments etc that they need. Of cause the sysadmin can do it for
them, but in that case they have to wait until he have time, and even then
they will get poorer understanding of the system. Also a lot of programmers
need to know network basics because their application has a network component.
It is also important for a programmer to understand what he need to make a
system administration friendly, like easy to install and upgrade, sensible
logs when things go wrong etc. They don't need to be able to take over the
sysadmin's job the next day, but should understand what the job is about.

Sysadmins on the other hand need to at least be able to script common tasks
and write plugins for monitoring software and be able do some customization of
the systems. They don't need to be able to, or use the methods for developing
large complex system, but I expect a good sysadmin to know programming and
scripting in some form.

------
Hovertruck
Why not link to the blog post directly? This link will break as soon as the
author writes another blog post about devops.

[http://cuddletech.com/blog/?p=817](http://cuddletech.com/blog/?p=817)

------
contingencies
When a sigh will not suffice, abstract it.

------
shirro
Part of it is just experience. As a sysadmin I was jumping from one task to
the next learning as I went. I never got the chance to be really crazily good
at any one thing. No sooner would I learn something and apply it and I was off
to put out another fire in a totally different area.

------
tete
I can so much agree. I pretty much went that way and a lot of it is true and I
wished I had done that earlier. Also I would recommend everyone to go this
way, because it makes some problems easier to solve for you, even if you stay
a sysadmin.

Other than that, I really wasn't a good sysadmin, actually I used to be a
really bad sysadmin, but still came from that direction. This however brings
me to another thing: Sysadmins aren't ever valued enough.

Everyone treats Sysadmins as second class IT people that can't code too well.
However, that's from my point of view not true at all. They usually code
"script" way faster. Their set of tools simply are different and designed for
people trying to reach different goals.

But there is more to this. As a programmer, you have your cozy, nice
environment, your nice specifications, your nice libraries, your basic
knowledge of data structures, etc and you have an easy way to be like "uh you
can do it that way. It's even cooler".

As a sysadmin you usually don't have anything to rely at all. You don't learn
too much at universities, that can be of help and while as a software
developer you basically have to just write code and make that work as a
sysadmin you are dealing with unpredictable systems, have maybe a single
system with multiple users, need to make sure none of these processes do
something bad, yet can do everything they may need to do, need to trace down
an error that is _somewhere_ , don't have a debugger, can't just quickly run
the code on your test system, have a way harder time to trace things down, but
then it doesn't have to be the one system, but could be a whole network of
such systems, the whole internet can be the source and you are pretty much
always in the battle field, in live systems and lots of attackers could
potentially attack every single thing and you basically need to know every
protocol, and every piece of software, every network connection and by heart.
It could be something very high level causing the problem, a faulty program,
the hardware, physics, whatever and you have to try to master chaos every
single day, while programmers usually just work with something simple and
often see it as a super hard problem when they are dealing with some input
from the outside.

I don't know, but it feels weird, when the bigger amount of math basically
causes your pay to be higher, when that very thing is actually making your job
easier.

I am a programmer myself. I love coding. I love my job and I kinda like to
brag a bit with it, but actually compared to many other jobs, like sysadmins
we are really overpayed, while way too often complain about bad sysadmins.
Being a sysadmin isn't just "apt-get install apache", but way more and way
intense.

Another thing that's also funny and strange is that scripting thing. The same
thing that often causes sysadmin to start out with ugly code is the thing that
makes sysadmin advance extremely quickly when it comes to code quality. Once
they really are programmers they know that a lot of code can be complex and
since they have skills in using many tools and not just use them, but use them
correctly (unlike most programmers tend to) they will usually turn out to
write high quality (less bug prone) code with the right amount of
defensiveness and in a way that brings some kind of order into chaos.

It's a hard way, but sysadmins are maybe a bit too much perfectionists to
realize that they would be good programmers and that hardly is recognized by
people starting out as programmers, who would make really, really awful
sysadmins btw. and no just because you set up a unix system and a firewall it
doesn't make you a sysadmin.

I actually even think that the devop thing is going partly into a wrong
direction. It works, but only if you are using Ubuntu. No, really. That's also
not a good thing in general, because it could actually push IT backwards.
Diversity is sadly not valued enough. Using the right tool can really push you
forward. And while we often seek for tools used by most people it ultimately
leads to stalling. Going into a direction where we have Ubuntu instead of many
Linux distributions, many Unices, many Operating Systems isn't good. There are
reasons for the creation of more than one and they didn't use to be as naive
as they frequently are today.

Also Sysamdins are very humble. One can actually see that by version numbers.
It's maybe a bit silly of an example, but they (and programmers working with
them) are extremely conservative about making a version sound like it could be
stable. If they try their hard to find anything to change or make better and
really can't think of anything then they will maybe call it 0.9 Alpha. On the
other hand, if they find something that is called 12.13.4.0 then the zero at
the end makes them really suspicious about the software being too unstable
(there is enough chaos already). If you know such software then it is probably
written by a person somehow related to sysadmin stuff.

However, there is one thing that would make Sysadmins really great
programmers: They are super pragmatic. They a´have a good sense of realism.
They know what's necessary and what isn't, as long as they are not afraid of
being punished for non-perfect/beautiful code. That means that they are good
at writing release-ready (feature-/stability wise) code.

However none of that is true for anyone who just switched. It's just hard to
learn what the right amount of quality really is and that it also is about
"beautiful code".

Well, so much about my experience. If you know a sysadmin that likes to code,
maybe help him and be nice. They are always complained about when something
doesn't work (even when it isn't their fault) and whenever something works
great it is the developers, marking people and so on who are praised.

------
junkilo
my response: [http://eleventymedia.com/](http://eleventymedia.com/)

------
michaelochurch
I really can't stand all of these silos that seem to exist for the benefit of
MBA-culture idiots who don't understand the first thing about computers, at
the expense of programmers.

"You can't X. You're a Y!" Well, _fuck you_.

All of this warring-camps idiocy, "hire a real X" nonsense, and all of these
damn silos, just exist so a bunch of people who aren't smart enough to manage
technology can feel like they understand shit. They don't. They hire some
$785/hour consultant whose diagnosis is, "you need better system
administration". So they go out and hire a sys-admin or two with a "track
record" and the problem is "magically" solved.

Programming was supposed to be the magic skill that allowed one to move
fluidly about the industrial/technological economy. It showed that you could
grapple with abstraction, learn quickly on your own, and get hard things done
and do them well. It conferred _mobility_ and job security.

Then those useless "be a team player and syngerize the verticals while the
rubber hits the road" monkey-morons came in and had to fucking siloize
everything, just to play a divide-and-conquer game, create the sense that we
were incapable of handling "the big picture", and reduce our bargaining power.
This distracts us from the fact that those people aren't at all necessary.

Good business people go out and solve important business problems; the bad
ones meddle in our affairs for intimidation's sake.

~~~
incision
_> Then those useless "be a team player and syngerize the verticals while the
rubber hits the road" monkey-morons came in and had to fucking siloize
everything, just to play a divide-and-conquer game, create the sense that we
were incapable of handling "the big picture", and reduce our bargaining power.
This distracts us from the fact that those people aren't at all necessary._

So true.

I worked in a largely silo-free environment up until about 2007. New
management transformed an effective ~25 person team into half a dozen kingdoms
of 30+ each in a little over one year.

I left and discovered just how unique my environment had been prior to the
regime change. It's hard for me to imagine how any organization which has gone
down this path can hope to reverse it.

It's desirable to both the managers who use it make themselves relevant and an
unfortunate majority of workers who take comfort in their pigeon-holes.

