
I'll Probably Never Hire Another Pure SysAdmin - nemesisj
http://peebs.org/ill-probably-never-hire-another-pure-sysadmin
======
wccrawford
With any non-trivial assortment of servers, you need a dedicated sysadmin. The
security issues alone should dictate this, but your developers don't need to
be worrying about sysadmin stuff. They need to be coding.

I totally agree with proper deployment procedures. But having them doesn't
eliminate the sysadmin at all. Sysadmin should be deploying standard packages
that are necessary for the system, along with updates for them. Developers
should be deploying in-house code, and updates for that.

I worked at a company that hired a sysadmin too early. When he quit, I ended
up doing his duties and mine, too. Of course, I automated 99% of it and
everyone was happy. That was really early in the life of the company.

Later, when it came necessary to hire one, they didn't. They'd expect a
developer to deal with the issues, and were always dissatisfied with the time
it took, and the lackluster results. Eventually it got so bad they hired a
dedicated sysadmin again. Now, unlike the past, they actually needed one and
he was busy all the time. Developers stopped being unproductive for weeks at a
time, working on projects they didn't understand and couldn't complete
satisfactorily.

For a startup, yes, skip the sysadmin at first. But once you have a complex
system, hire one. It'll save your company.

~~~
illumin8
+1 - you can automate deployment all you want, but when something goes wrong
with your production site (and it inevitably will), do you want to pull your
top developers off of whatever new features they are working on and ask them
to troubleshoot and fix production?

Sysadmins will always be a necessary evil when you are dealing with hundreds
of servers, physical, virtual, or cloud. The tools and process will change. We
have gone from physical Unix boxes (Sun, HP, etc) to virtual Linux boxes, to
cloud servers, but sysadmins are still doing the heavy lifting of designing
and building highly scalable systems.

I would say that sysadmins should look at moving into systems architecture
roles, since one sysadmin will be able to manage hundreds of systems rather
than a few dozen in the future. You really need architecture, networking, and
performance tuning experience in order to succeed when you are managing more
than a handful of systems.

------
tptacek
People on this thread seem to be objecting to the idea that agile companies
should avoid hiring people who manage their systems. But that's not what he's
saying!

My read is that he's saying that the days of the "system administrator" with a
job description that begins and ends with tar flags and Apache vhost
configuration are numbered.

He's not saying sysadmins are being replaced by Puppet and Chef; he's saying
they're being replaced by a new kind of sysadmin that can be expected to
extend Puppet and Chef.

This seems closely related to a point Patrick McKenzie makes all the time,
that being the difference between a "programmer" and "someone who provides
business value by means of applying specialized skills with programming
languages". Don't be a "programmer".

~~~
jvehent
So, we're playing with the terms ? That's it ?

As a sysadmin, I'm fine with the concept of continual improvement, there is
nothing worse than a static industry. The job is changing every day and
this.is.good.

What I understand from this post is that startup people don't want to deal
with sysadmin anymore. It makes a lot of sense, considering the amount of
time/energy/money you need to run a stable infrastructure. But at the same
time, heroku, ec2 & al. will never be able to cover everybody's needs. So
while I'm convinced that a dev or devops will be fine deploying his webapp to
some random cloud, I'm also convinced they will come crying when massive load
hits them and they can't tune the system.

The sysadmin is not only the guy who configure new servers, it's also the guy
who explains the new dev dude that, no, he can't do 15 inner joins on large
tables to create the home page because it will stress the IOs so bad that the
whole website will collapse.

Ease of use != flexibility.

~~~
tytso
It very much is about the terms. 20-25 years ago a sysadmin was expected to be
able to fix C bugs, and have skills not that far removed from programming in
assembly (i.e., sendmail.cf files :-).

About ten years ago I remember talking to some old timers who were surprised
that a lot of the younger sysadmins no longer even did any perl progamming,
but rather did things like program Cisco rounters and set up Oracle servers.

At Google we don't have sysadmins at all, but rather "Site Reliability
Engineers", and they are expected to be able to answer some fairly deep C/C++
data structure and algorithms questions. All of those skills are necessary to
run a Google-sized infrastructure, and it's quite a far cry from the sysadmins
that don't even write perl scripts any more -- and yet feel perfectly free to
call themselves "sysadmins".

When I first started, I both wore a pager and wrote C programs --- including
the 'ninit' program that was described in the Unix haters handbook. The fact
that I was one of the people who might be woken up at 3am when the mail server
melted down due to named core dumping informed how I wrote my C programs
(i.e., very defensively), and is a prime example of the dev/ops concept.

~~~
jvehent
So we agree on the definition of what a sysadmin, systems engineers, devops or
whatever people want to call it should be. A sysadmin must program, that's a
given. When did we forget that exactly ?

But why are so many devs shooting at the sysadmins these days ? I never read
anything from sysadmins complaining about the astonishingly low level of
knowledge of a lot of devs. And they are numerous, willingly ignoring even the
most basic performance issues and vomiting thousands of lines of codes,
hundreds of tons of libraries, to simple problems.

It seems that devs are frustrated to not be able to push their random app to
the web in 5 minutes, and throw their love to PaaS like an apple geek
discovering the new iphone, blaming the sysadmins for not providing the same
thing at their own company without even understanding (or trying to
understand) how operating systems work, and how their own code should be
built.

The real issue here is that when devs and sysadmins DON'T work together, and
ignore each others problems, constantly blaming each other, then the result
sucks. When the sysadmins are 20% devs, and the devs 20% sysadmins, and they
drink their coffees together in the morning, you don't see that kind of
problem. They built good, reliable, scalable infrastructure, whether it's in
the cloud or in the basement of their house.

~~~
ryanpers
probably because at nearly every place I've worked at the sysadmin/ops group
was the #1 obstacle to getting things done. Sure some of it was "saving you
from yourself", but a lot of was pure obstinatance for no other reason than I
think sysadmins just cant turn it off anymore... much as your mom says "dont
frown your face might get stuck", it appears a lot of people (not just
sysadmins, developers too - you know the one, in the corner who is old and
surly, been there forever, and hasnt been promoted in about as long).

The deck is stacked against sysadmins - developers are the people who deliver
business value to leaders, and sysadmins are the ones who cost money and say
'no'. When you are on a tight timetable, the person who says "sure" is the one
who gets the high fives... assuming things work out, of course.

~~~
Goladus
>sysadmin/ops group was the #1 obstacle to getting things done.

This happens when the company discovers that when the system goes down they
stop making money. If you've managed to design and build a system to be very
resilient and reliable on the chosen platform and has proved to need minimal
maintenance, then chances are good you have a very small ops team who doesn't
get in the way much.

------
jwatte
It's not clear to me what you think sysadmins do.

Who will make sure your provisioned configurations have proper redundancy?

Who will move your domain when an availability zone has an outage?

Who will work with upstream networks when you're getting DDoS'd?

Who will wake up in the middle of the night when a service you're integrating
with is not working?

Maybe you've never run a large system at scale, or maybe ou just had bad
experiences with "SysAdmins" that didn't, but the role of hand-holding a
complex, integrated service is not going away. The fact that you may not need
screwdrivers for certain systems is almost beside the point.

Nevermind that, when you're at sufficient scale, you can run your own data
centers cheaper than Amazon. Break-even happens well before a thousand nodes.

(I'm not a sysadmin; i'm an engineer who is very glad we have good sysadmins
around :-)

------
poutine
If he'll probably never hire another pure sysadmin then he'll probably never
build anything of scale that people will pay for.

Sysadmins do a lot of stuff, and this is only somewhat reduced by cloud
services. Or is he expecting to get the developers to maintain OS versions,
security patches, log monitoring, proxy servers, database replication,
backups, firewall rules, failover testing and so on across dozens or hundreds
of systems? Right.

Absurd.

------
imroot
I'm not completely sure that I agree. While this might be great for small
startups, once you get past the growth of needing more than a few dozen
servers, your cloud and other computing costs are going to be exponentially
higher than what they would if you had actually purchased the hardware from
your favorite hardware vendor, hired a competent sysadmin, and managed the
hardware (and storage/network) in-house.

Currently, at $DAYJOB, our database is growing by about 1GB/day, and our
storage is growing about 4GB/day -- in a downward housing market. If/when the
housing market kicks up, I expect to see those double, if not triple. Having
someone in house who can keep the right people informed of the risk (of
running out of space, of buying the wrong equipment) who is focused on keeping
of the needs of the company balanced with the needs of the software is
critical -- do you honestly think your cloud provider cares if your customers
can't access their data during an outage? If you're a small fish...probably
not...

EDIT: Spelling/grammar change.

~~~
TomOfTTB
I agree with you but I'll go even further and say this guy has no idea what
he's talking about. As someone whose been in charge of a fairly small cloud
outlay (8 virtual servers, 2 of which go off at night) I can tell you the
cloud introduces its own set of challenges.

So there's less hardware replacement and more DNS configuration but none the
less there's still a job for system administration.

In fact, I'd say the role of administration becomes even more significant
because the cloud industry is so new. Hardware has, in many ways, become
pretty rock solid in the last decade. We're to the point where a hard drive in
a RAID setup can go out and the users won't even notice.

The same can't be true for things like SQL Server Federation and caching
servers which used to be the exclusive domain of large web sites but are
increasingly being used by smaller companies who don't have a whole staff to
dedicate to their upkeep.

(One Caveat: If cloud models like Microsoft's Azure take over this could
change. By that I mean clouds that are literally frameworks in themselves and
not virtual machines)

~~~
coenhyde
You have to make sure you use the right tools. If you re-invent the wheel a
dozen times, managing cloud deployments will be a nightmare. Tools such as
Rightscale or Scalr.net are well with the money.

I manage 2 dozen Ec2 instances of various types as a side task to my regular
development job. I script everything. When things are scripted they are
consistently reproducible and fast. Humans make mistakes (including me).

I would prefer there to be a dedicated SysAdmin but there just wouldn't be a
full time job for him to fill.

------
amcintyre
_"Now, the watchword to the development team is: it's not done until I can
one-click deploy it."_

I have apparently been spoiled somewhere along the line, because I _really_
don't like having clumsy manual processes that involve software/computers. I
was recently tasked with figuring out how some researcher's code worked (with
an eye towards automating the use of their tools), and it took 8 pages to get
most of the steps documented well enough that I could follow them. Steps like,
"edit the path in line 977 to point to your input files, recompile, run, click
button A..." -- i.e., things that should never have been done manually to
start with.

How on earth people whose primary job involves writing code to do things for
them can fail to automate tedious (and frequently error-prone) parts of their
job is beyond me.

That said, many companies are still going to need to hire somebody to write
the one-click deployment. Somebody will have to know how to interact with the
cloud services, and how to get around all the little minefields and gaps left
in the cloud service offerings, and odds are you aren't going to want to pay
that person as much as you pay your developers that do the "real work." That's
probably where the sysadmins will go.

~~~
gridspy
It's simple : The people you are complaining about set these things up once on
their computer and then forget about it. They don't realise how many things
they have set up once along the years.

It is not until you appear that it's time for the app to be deployed
elsewhere.

~~~
amcintyre
Actually, no: in this case, the tools were used fairly frequently.

------
dgabriel
In 1999, everyone was telling me they'd never need to hire another Java
programmer now that Rational Rose would soon be able to spit out complete code
based on the input of a team of architects. Just sayin'.

~~~
zalew
yeah, it's like saying you don't need programmers because there's django and
rails

------
a2tech
That might work for him running a small business. How do you think those
images you 'just spin up' at a cloud provider are made? Who makes sure the
boxes that run those images you've just brought up have power and run
smoothly?

~~~
jwatte
His point is that Amazon hires sysadmins, not him. However, that seems based
on a very narrow definition or understanding of what a "sysadmin" is or does
for the business. I identify "sysadmin" with "operations" but perhaps others
see it differently.

------
rythie
People seem to think puppet and chef are some sort of revolution, as if we
didn't automate all this stuff before with bash, rsync, kickstart and
cfengine.

~~~
gaius
That is my experience too. The people here are going crazy for func - and none
of them have even heard of Expect!

------
darrikmazey
As is typical with this line of thinking, it only scales so far and only
applies as long as you are "on the map" of what is standard. Stray too far and
your devops personnel will spend more and more time "administrating" and less
time coding. Being primarily a coder with considerable sysadmin skills, this
has happened to me in every job I've worked in the last 5 years. I start out
devops while everything is standard and slowly do more and more admin until I
eventually quit (because I'm primarily a coder). The job I'm quitting this
month is because I'm now 95% admin. Like all resources, you need what you
need. Devops is sufficient is specific circumstances, but if you stray from
that scenario eventually you need an admin. They're the glue when things don't
quite fit seamlessly together.

------
rythie
What's a 'pure sysadmin'? If they DBA are they unpure? or code? or do biz dev?
or support? These all seem pretty normal for a sysadmin nowadays.

Reminds me of the 'No true Scotsman' thing:
<http://en.wikipedia.org/wiki/No_true_Scotsman>

------
philwelch
Summary: You don't need to hire sysadmins because you can outsource your
systems to a cloud provider.

This is true, but it's also a lot like saying you don't need to hire
developers because you can outsource your development to Lithuania. Really
what it means is that it's possible to outsource systems. If you're the little
guy and your needs manage to mesh well with what you can get off the shelf
from a cloud provider, this is true. Otherwise, this isn't true. If you're the
little guy and your needs don't exactly fit what you can get off the shelf,
it's going to be trickier. If you're the big guy, you can afford to cut out
the middleman.

------
inopinatus
A development team had a problem. They were in need of a manageable production
environment. "I know," the Chief Developer said, "We'll use Chef!". Now they
had two problems.

However, I agree with the basic contention - that the role of a pure, old-
school, risk-averse sysadmin is obsolete. In a world of Agile developers
practising continuous delivery (or hoping to), the system engineer is now
another cog in the development machine; not part of a separate machine.

The resulting closeness does wonders for understanding and quality of the
result. Unfortunately, at the time of writing, system engineers who speak
Developer are relatively few & far between.

------
migrantgeek
Operations shouldn't develop and Development shouldn't do operations. That
isn't to say that Ops can't program. I write a lot of Python to get things
done but that's not Development.

I'm a sysadmin in a 100% AWS shop. Before I showed up, Dev was running things.
It was horrible.

Versions differed greatly, they didn't use init scripts for anything,
deployment was pretty manual, and their release cycle was really slow because
they spent too much time in operations where they were clueless.

My job is to keep production rock solid and provide Dev with the resources
they need to write code.

If Devs aren't writing code, I failed.

------
ridruejo
I think you need to distinguish here different kind of systems.

a) Deployment of 'standard' applications like Drupal, Joomla!, Wordpress, etc.
It makes sense to automate it once and let people "self service" In this case
either the sysadmin is responsible for packaging those apps or they can get a
third party "catalog" that you can deploy to your virtualized/cloud
infrastructure (that's exactly what we are aiming for at BitNami, you can
check it out <http://bitnami.org/stacks/>)

b) Complex multi-tier, in-house applications. Those are much harder to
automate because they tend to be "one off" systems and will require
'traditional' system administration for a while.

c) Web companies that run a logical application across multiple servers (like
Zynga, Netflix, though obviously most people will do so at a smaller scale).
Here you need to use something like Chef or Puppet, there's simply no other
way of doing so.

In other words, before the advent of cloud it was 'optional' for system admins
to automate their work. Now it is expected. In that regard, I think automation
is the real benefit that cloud computing has brought us (rather than the
capex/opex and autoscaling that people often mention)

------
latchkey
If you look at it from the point of view of creating a new company that is
based online, chances are that you can design the system from the ground up to
be entirely hosted in the cloud. Even better than AWS is if you can host your
site on a PaaS solution such as Google AppEngine. GAE negates the need for a
DBA, netadmin and a sysadmin.

I'm building a web based company right now with a good friend of mine and
we've gladly chosen GAE because of this single attribute. Both of us are 15+
year sysadmins as well as software engineers. That said, instead of worrying
about how we are going to scale and run everything, we are able to focus
entirely on the development of features and building the company. As a result,
we hope to never have to hire any IT positions.

In the end, I don't think the roles will go away since someone still needs to
run Google. ;-) I just think that the roles will eventually morph a bit for
the companies that can take advantage of PaaS solutions.

------
jmspring
It is an interesting line of thinking. Way too many startups I've been
involved with/talked to have over looked the role of admins/an ops team. The
mentality seems to be that developers should also focus as admins/ops. Surely
nothing will go wrong on a weekend after a developer has already put in an 80
hour week, right?

From my experience, sys admins are important. If you are doing anything of
scale, knowledge on how to setup, tweak, and debug systems is critical. It is
helpful if the admins can stretch a bit into the app layer, just as it is
helpful for developers to be able to stretch a bit into the systems layer,
problems come up and knowledge of the parts helps.

As an engineer that can also setup systems for scale, I know my limits and
really appreciate help from and learning things from good sysadmins.

------
gersh
Somebody has to write the deployment scripts, choose a cloud platform, and
update the platform. If a cheaper cloud platform comes out, somebody has to
figure out which platform is cheaper. You may have automated error logging,
but someone has to go through the logging and see what went wrong. Somebody
should also look over how reliable the platform is.

If your business only requires a standard Rails site, there may standard
tools. However, when you develop new technology, someone has to deal with
deployment and scalability issues. This may be more of a developer job, but
there are significant sysadmin issues.

------
yxhuvud
This is just silly. You will need dedicated sysadmins when your organization
gets large enough to support 5-10 nontechies.

------
gcv
I think even "pure" sysadmins will have jobs until the cloud providers figure
out how to solve their IO bottlenecks. I keep hearing horror stories about
database performance and how people are forced to go back to good ol'
dedicated hardware to solve it.

------
mathattack
I think it'll be both faster and slower than he says.

Faster: Cloud adoption will be huge.

Slower: Cloud technology will increase the amount of apps and data used. Low
end jobs will exist helping firms get on the cloud. High end jobs (fewer
though) will exist at the cloud providers.

------
samarudge
I work at a company currently in the process of deploying an Oracle 7 cluster.
Regardless of technological advances in startups, enterprise will always need
sysadmins, if for nothing other than filling in the correct check boxes on the
audit forms.

~~~
chrisbolt
Today's startups are tomorrow's enterprises.

~~~
samarudge
It depends. If it's still around in 10 years, I'd be willing to bet not a
single line of code currently used by Facebook will still be running. Some
startups will become "enterprises", because of what they do, but I'm talking
about financial services, banks, government departments, public services etc.
Their technology doesn't move forwards, because it can't. I'm sure little to
no startups are using Oracle IAS (Internet application server), yet it's one
of the most used technologies in enterprise applications, and will be for some
time. If you're a sysadmin with a deep understanding of IAS, or Oracle DB, or
HP-UX, or any of the countless other "old" technologies used in enterprise,
you'll have a job for life.

------
pbreit
I'd really like to see more activity around creating, publishing and refining
Puppet, Chef, Fabric, etc scripts that are usable in various projects. For
example, publicly available scripts that approximate what is provided by
Heroku and DotCloud.

------
dfc
Who runs the systems at the cloud provider / github or you outsourced
DNS/SMTP+IMAP?

------
zobzu
Who's going to setup puppet? Right. End of story.

