
System Admins R.I.P.? - danielle17
http://blogs.mulesoft.org/system-admins-r-i-p/
======
cagenut
As a sysadmin these kinds of articles have a sort of duality to them.

On the one hand they're insulting. The author is clearly wholesale ignorant of
the vast majority of sysadmin roles and responsibilities (and why companies
need them) and yet feels like they know what they're talking about enough to
declare the role dead.

On the other hand, developers with no clue whats entailed in sysadminning are
the number one source of our job security, so its a backhandedly good thing.

Please, by all means host your startup's database in a "aaS" solution with
~500 IOPS (except sometimes! at random!) across a 50 - 100ms link. It will
make me look like a wizard when I clean up your rookie moves.

~~~
pavel_lishin
I spent two days trying to figure out how to compile a 32 bit library on a 64
bit system.

A sysadmin would have figured it out in 5 minutes, and lazily dropped by my
desk to show me where I went wrong.

In my experience, sysadmins aren't just people who keep things running during
the course of Business As Usual, you guys are also a terrifying repository of
arcane knowledge.

------
shykes
Yes and no. System admins aren't disappearing - they're specializing into 2
distinct professions:

* Those working for infrastructure _providers_ (IBM, Amazon, Rackspace) focus on the bottom half of the stack: everything from the datacenter's floor plan to switching and VM allocation.

* Those working for infrastructure _consumers_ (everybody else) focus on the upper half: the business's software stack, how to glue it together, and where to run it.

Conclusion: system admin, as a profession, is gradually disappearing. But it's
being replaced by 2 _better_ professions: more challenging work, higher
perceived value within the organization, and higher demand.

~~~
nailer
Shykes comment is better than the article. since about mid 2000s sys admins,
regardless of what their job titles are, have become infrastructure engineers.
learn the vmware Api, the red hat satellite api, some databases, write some
socket apps for monitoring, learn some charting. I'm actually a developer now,
but the demands of infrastructure engineering is how I got there.

------
iigs
Maybe, but I don't think so.

Not many years ago people got paid to, among other things, pick out the
appropriate operating system for any given application. Clearly there's not
much of a future in that.

These days that same person is responsible for vetting cloud hosting
providers, email service providers, and has to be capable of comparing the
"old" in-house solutions and bring things inward if there's benefit.

The technology changes, but the role more or less stays the same. The system
admin is generally a bridge between business operations and the technology
stack, and there's always going to be glue there, particularly for businesses
of at least 10 people.

Disclosure: System Admin / Engineer by trade. Weigh my opinions appropriately.

~~~
TomOfTTB
A couple things here...

1\. I think there's a distinction between Sys Admin and a more management
position like Director of IT or CIO. In some companies there's only one person
who fills both roles but they are two distinct roles and I think this article
refers to the actual System Administration role rather than the management
role that selects software and vendors.

2\. While I don't think the System Administrator role will disappear
completely I do think the focus will switch from one that is more hardware
driven to one that's more software/programming driven. I think businesses will
expect System Administrators to handle integrating their cloud based services
into a comprehensive whole in the future rather than just making sure there's
a network to carry data. That's important because it will mean a dramatic
shift in the required skill set.

~~~
jrussbowman
My experience is that my job has never really been very hardware driven. My
experience with hardware is keeping our standard server spec up to date with
the vendor. Installing OS and applications. I also manage keeping track of
warranty status and handling hardware failures, and of course backups.

That's a lot of words to describe work that isn't a lot of my time annually.
Especially on a team where the load is distributed. Take into account that a
few hundred "servers" are virtual machines running on tens of physical servers
and there you go.

My skill set already requires me to manage multiple operating systems and
applications. My biggest value to the company is problem solving, whether it's
engineering a solution to meet new business requirements to diagnosing and
fixing a problem that can happen anywhere from the hardware to application
layer. Sysadmins don't just sit and look at blinky lights all day waiting for
one to go orange.

------
rnemo
Anecdote from a SysAdmin: I remember during the early 2000s there was a trend
for a little while to scale down the IT department, and rely on corporate
support, or use outside consulting firms, or other methods that involved
keeping the payroll mostly free of such "backroom" types. I've heard a lot of
horror stories from people who would go into a company to fix a small problem
or perform an upgrade, and would find virus ridden computers barely running,
people doing ridiculous things like burning cds to move a couple of files
(remember, early 2000s here), people losing their product license or support
information, or other terrible things, simply because there was no on-site
sysadmin to keep all of their computers in good functional order.

This article presents much the same scenario, but updated for 2011. And make
no mistake, it will probably work for small, well managed, web-based
companies, full of people that already know plenty about computers and
technology. For pretty much every other type of major company though, there
needs to be at least one person who's responsibility is making sure that the
technology in the workplace can be worked with and not just worked around, and
who can be responsible for dealing with the new technologies that come around,
and until the cloud can provide such a service, the sysadmin will be alive and
well.

------
zdw
Hey, instead of paying a person to keep our data on our own systems, we can
pay a bunch of companies to keep it on their systems.

They'll never fail, get obsolete, lose our data, get bought out by someone who
ruins them, have availability problems, or lock us into using just their
service, right? Guys?

~~~
patio11
Paying a sysadmin is, sadly, not a guarantee that one will have-future proof
redundant systems with seven nines of availability that are meticulously
documented enough to seemlessly hand over to anybody with a sufficiently
wizened countenance.

OK, dueling strawmen out of the way, let me try to add some value to this
discussion. Just like the lower rungs of the value chain are getting eaten up
in programming, the lower runs of the value chain in sysadminning are also
getting eaten away. Back when I was in middle school I was, I kid you not,
routinely told that the comparables to my labor at HTML editing all charged
$100+ an hour for making websites in Notepad. (Complete with under
construction signs.)

Thankfully, the state of the art has improved such that there are a variety of
options for e.g. a church school to have a web page listing upcoming Christmas
pageants without having to pay $10k a month on an ongoing basis for
maintenance.

You know the old saw about competent sysadmen trying to write themselves out
of jobs? Well, don't look now, but in a lot of areas _they are winning_. My
old day job called me a Systems Engineer, which includes basic sysadminning,
and over my very short career standard deployment practices for e.g. Rails
apps have improved radically in a direction which requires me to spend far,
far less work at this aspect of m job. There are "real companies" on similar
stacks who have server-to-admin ratios which just 10 years would have been
unthinkable. (100 to 1? No problem. It's really just 3-to-1 with heavy
automation in play, anyhow.)

Sysadmen in the audience worried about their career prospects should probably
try pushing themselves up the value chain. There was once lots of value in
babysitting cranky programs which blew up frequently in unpredictable ways.
That looks like it is changing. Look at Exchange versus I-can't-believe-its-
not-Gmail from the perspective of a firm with less than a thousand employees:
heads you have someone whose full time job is to fight outages, tails you do
not. Don't be the John Henry in that scenario: remember, he dies at the end of
the story.

~~~
itgoon
> the old saw about competent sysadmen trying to write themselves out of jobs

When I preach this, I modify it to "always eliminate your role". Your job
won't go anywhere, but what you're doing for that job should always be
changing.

------
jcdreads
Banking and health care data, for example, cannot for the most part live on
AWS or some other generic cloud provider. There are financial and legal
penalties for leaking a database full of medical histories and Social Security
numbers that preclude storing hospital records on slicehost or something. For
the time being, and, given the conservative bent of these industries, probably
well into the future, there will still be local sysadmins in the classic sense
managing physical hardware under physical security.

That said, there's a screaming business opportunity for a group of
enterprising sysadmins who want to set up HIPAA-compliant PaaS offerings for
medical information shops, or something similar for banks that won't make the
SEC or FDIC freak out.

------
Semiapies
As with virtually every headline-as-a-question, _no_.

Too much existing infrastructure. Too many systems that aren't web-UI apps
with cloud backends.

Maybe in ten years, this will be more plausible.

~~~
pavel_lishin
Will existing infrastructure disappear in 10 years?

~~~
Semiapies
No, but many systems will be that much more likely to be replaced by something
that could possibly run in the cloud. It'll become something actually on the
horizon in ten years, as opposed to something unreasonably far off.

------
chadr
High quality sysadmins are evolving into what is called the devops role.
Trouble shooting, scaling, architecting, and automating production systems are
just a few areas where devops people shine. The cloud just provides them
another set of tools to work with. It also frees them from dealing with the
annoying/repetitive tasks (spinning a CD to install the OS, plugging in the
network cables, etc) and allows them to focus on improving the application. A
number of devops people I know can easily transition into developer roles when
required. Summary: a great sysadmin should know how to code and does so in
order to improve the app.

~~~
neworbit
That still feels like "sysadmins are being replaced by developers" to me

~~~
njharman
Only developers think sysadmins arent developers.

~~~
knieveltech
Conversely only sysadmins think sysadmins can code. A classic example is a
close personal friend of mine who works as an sysadmin for an insurance
company.

On two occasions I've had the unfortunate displeasure of working with him on
web-related projects (we share similar hobbies and our local communities need
web services of various kinds) his approach has been vile, Rube Goldbergesque
concretions of Perl & shell scripts.

This is absolutely THE GUY I'd call if I needed some advanced logfile parsing
in a hurry, but when it comes to actually developing web stuff the guy's
totally in the dark. What's worse is his only metric is "It works and it makes
sense to me" so he's content to make a complete hash of a development project.

During a recent conversation he admitted to having no real grasp on basic HTML
and hadn't heard of CSS. The thing that kills me is he's convinced he's
qualified to run websites for our local groups and absolutely refuses to
accept input on the subject.

Oh well, the guy's also an ace whitewater kayaker and one of my favorite
paddling buddies. Considering this guy will probably save me from drowning one
day I can look past his failings as a "developer".

------
strlen
Yes, cloud will remove the need for systems administrators/operations much
like the power grid has removed the need for electrical engineers.

~~~
euroclydon
Actually, they're called "power engineers", and the maturity of the U.S. grid
has largely shut down American university production of such engineers.

~~~
strlen
Power engineering is one of type of electrical engineering. Demand has shifted
away from that type.

In the same way, the demands shifts from one sort of systems administrator --
the traditional sys admin who racks and stacks machine and manually edits
config files on a UNIX system, to another: one who approaches the problem from
a computer science and engineering point of view (using software such as
chef/puppet, developing custom tooling, working closely with application
developers, etc...).

All of this is not news to me: I've been doing this kind of work in 2006 at
Yahoo before it had a buzz word ("devops") attached to it (these days I've
shifted to being "pure" software developer). Broadly speaking, we had three
kinds of operations personnel: those working at the physical layer, the
network layer and those working between the operating system and the
application layer.

Majority of the upper layer work was either writing code or working very
closely with application developers. Thanks to these tools we could deploy and
re-deploy machines within the matter of minutes (with no virtualization
involved) and handle servers arriving by the truckloads.

I'm glad that Amazon and others have made the lower layers a publicly
available service, so that the people preferring to work at the upper layers
_but would prefer to be in a startup_ (rather than a corporate behemoth) could
concentrate on the work they find fulfilling.

------
wccrawford
You can have my system admin when you pry him from my cold, dead hands.

Wait, that doesn't sound right.

The points stands, though: I really don't want to do that stuff and he's
welcome to it. And I seriously doubt the whole world is going to use someone
else's servers... There will always be companies that don't trust 'the cloud'.

------
syaz1
This reminds me of a sentence I heard somewhere before... Paraphrased: You can
make _any_ statements -- so long as you end it with a question mark.

I think it was The Daily Show, when John commented on media headlines.

~~~
qntm
[http://www.thedailyshow.com/watch/wed-
september-13-2006/the-...](http://www.thedailyshow.com/watch/wed-
september-13-2006/the-question-mark)

~~~
syaz1
Nice find! WTF is up with Arnold at the end of the clip?!

------
krobertson
All of those systems introduce convenience at the cost of lockin and they are
no longer suitable once you reach a certain scale. Then the problem becomes
either the lack of visibility or the cost with the metered services.

For me, a nice service like PubNub would run us nearly $300/day. Compare that
to a simple EC2 m1.large at $240/month.

Visibility comes most of all with DBs where you want to manage lower level
settings. Things like MySQL configs, disk volumes and their configuration,
etc. Good luck tracking your IOPS on a hosted service, or since they're likely
cloud based, getting good throughput/latency levels.

Overall though, sysadmins are changing into devops. They're the ones who
connect the dots between what the app is doing, what scale it needs to run at,
and the systems need to support those. As things grow, need management of all
the moving pieces. Then comes monitoring minute aspects of the environment to
ensure performance and stability, paging when something goes out of bounds,
etc. And as you grow, minor changes or upgrades can have a huge impact or
require a lot of roll out, so need to test and benchmark several aspects.

Sysadmins in the small scale may be less important, but they're becoming even
more critical as you grow. The nice part is the tools they can leverage are
growing so you can do more with fewer hands.

------
jacques_chester
" Compared to system administration, being cursed forever is a step up." --
Paul Tomko.

------
zppx
Maybe it will be end of the BOFH or the eternal deployer (the guy that does
not everyday, just deploy software, I saw some of them in my life). Small web
companies maybe does not need a sysadmin, since they are focused on one
application generally this will not be a major problem for them if they know
some best practices.

The majority of sysadmins that I know works in the financial market,
telecommunications industry and ISPs, mainly in data centers, helping
developers who does not know about the infrastructure or as network operators.

------
heresy
Probably best career decision I ever made, was in 1999, 1 year into my career,
to switch from system administration to programming as my focus.

~~~
blueben
Because?

~~~
heresy
(Sorry, I missed this reply).

For me, it was the best decision, as I enjoy programming much more than
sysadmin work. I get my kicks out of creating new things.

Valuable experience though.

------
quarck
Sysadmins will not disappear, however, their role will change. The old
hardware lugger/patchjockey role is on the way out. However, managing the
sensitive data of the organization will be a role forever: data has to be
transported, converted, stored, retrieved, archived, etc.

Especially in organisations which handle data of a sensitive nature the sys-
admin (or whatever his/her role will be called) will have an important role in
the organisation (think especially of organisations which have to conform to
HIPAA, PCI, or other data protection regulations).

The success of an organization will in these Internet times be determined by
the value of the (customer) data it holds and the ability to extract business
value from that data. A data admin is therefore an essential task in any
organisation. The future role of Sysadmins will therefore move in that
direction in my opionion.

Disclaimer: the writer is an IT Architect at a large DataCenter Services
Provider.

------
knieveltech
It's interesting to see all these cloud-based services being used in the real
world but I hardly think this is the death knell of systems administration as
a profession.

It would take a pretty major shift in attitudes for medium and large companies
to start trusting vendors with the kind of data we're talking about here.

~~~
jacques_chester
I don't see why it's totally impossible. Outsourcing of archive management has
been going on for decades at this point.

~~~
knieveltech
Depends on your archive. There is literally zero chance a clinical research
organization (for example) would outsource any aspect of data collection,
storage or retrieval. Regardless of what systems where used there's simply no
way you could walk into a boardroom of client pharama execs and say with a
straight face you had the level of control over the process required to meet
their needs.

~~~
danenania
"literally zero chance"? Come on now...

I've only worked on ~10 paid development gigs and one of the first ones was a
pharma research company who hired my team to create an interface for a
truckload of sensitive data after nothing more than a phone call and a quick
NDA. This pattern has continued in a few subsequent projects, often with no
NDA or contract whatsoever.

A great many companies stop caring about privacy the second they can make or
save a few bucks, questionable legality or ethics notwithstanding one bit.
Businesses in general are fortunate that MOST software engineers seem to be
honest and good people, because they regularly hand over the keys to the
castle, often without even realizing it or considering the potential
implications at all, and to be honest I'm actually quite glad it works this
way. For whatever reason, we as a profession have apparently been deemed
highly trustworthy, and I much prefer this situation with all its risks to a
world of bulletproof firewalls and bureaucratic red tape.

~~~
knieveltech
I'm serious. Between government data retention regulations and hyper-paranoid
client pharmaco company execs the odds of a CRO putting their data out on the
cloud are functionally nil. This may change one day but that's the reality
today. We're talking the kinds of shops that are still signing million dollar
deals with IBM reps to provide in-house server infrastructure.

There's a world of difference between hiring contractors to come in and
develop in-house systems and betting the farm on a 3rd party vendor.

~~~
danenania
Point taken, but you shouldn't be so quick with the 'never gonna happen'
proclamations. Technology moves fast. There are many undisputed benefits to
the cloud model. Yes, there are serious drawbacks too, especially for certain
industries, but where you see insurmountable obstacles, others will see
business opportunities.

------
wccrawford
PubNub:

"Cloud-Hosted Service for Real-Time Messaging"

"There is no guarantee that all messages published will be received in the
same order that they were sent."

... So not really real-time then. Disappointing, since I was already trying to
think of ways to use it.

------
johngalt
It seems to me that every generation has to learn about IT the hard way. Not
just devs but also MBAs. The problems that your IT guy solves are not going
away.

1\. How to glue disparate systems together.

2\. Determine what services to standardize on.

3\. Prevent business types from choosing tech sizzle over steak.

4\. Translate technology to human readible form.

So long as control and management of information is important to businesses
there will be sysadmins.

------
tjarratt
So we've progressed from a small subset of the population as sys admins to ...
everyone who wants to deploy an application/service being a sys admin? This
seems like a step backwards, in some ways.

I suppose this is a win for people that just want email, or documents, or
calendars, but for anything else, you're stuck maintaining your own data in
the cloud.

------
bergie
From a couple of years ago: We're all ops people now [http://osspack.com/open-
source/edd-dumbill-were-all-ops-peop...](http://osspack.com/open-source/edd-
dumbill-were-all-ops-people-now/)

------
sunny_s
Sysadmins are being replaced, but guess by who? The fancier term "System
Integrator". Now the sysadmins design 'solutions' for businesses. Case in
point: Look at all the telecom solutions being deployed.

------
protomyth
I'll expect to see all system admins gone, when CASE tools actually work and
all those programmers disappear. Who ends up employing the system admin is up
for debate.

------
sunny_s
Was it just me who thought that this headline meant that some 'sysadmin' had
written yet another tool to automate stuff? ;-)

------
puredemo
This article couldn't be more wrong-headed.

------
jrussbowman
Are businesses going to start putting all their proprietary information in the
cloud? If so, I could see a short term where getting a job might be difficult,
until enough companies get burned and realized they should have never let
their information off their network.

Back in the late 90's I had a choice. I could go be a programmer or I could go
be a sysadmin. I really love programming. I in fact spend my "free time" (as
much as a father of an infant and toddler has) programming. However at that
time I noticed all the noise about how all the programming jobs were being
outsourced.

I made the conclusion then that they could outsource the programmer, but they
still need someone to run the infrastructure. I think that will always be the
case. I've worked for business focused, rather than technology focused,
companies my entire career. OK, 13 years in I've only worked for 2 companies
so I can't say I have a ton of exposure, but still... My experience is that
the business user wants things done a certain way and a lot of my time is
spent engineering solutions to meet those requirements, which often change
several times during the project phase. Cloud solutions currently are a pretty
well defined box, one that I don't see a lot of business users being
comfortable with the restrictions offered.

I do see some things going to cloud. Email for example... please take it. I
could get more work done not having to do restores because someone decided to
make a POP connection to their mailbox with their phone and deleted all their
email on the server. However most of the other solutions I'd see going to the
cloud are going to have to go onto basically virtual servers like what
Rackspace and Amazon EC2 instances offer. And who's going to know how to
manage, configure, and keep those security updates going? Oh yea, the guys
who've done it on physical infrastructure for the past few decades. So
basically if a company goes that way then I'll have less warranty replacements
to worry about.

I think perhaps the cloud is going to change the nature of a sysadmin's job,
but the profession isn't going to go away. Heck, I don't even know that I've
ever been a pure sysadmin. Seems I've always had to do some programming,
engineering and dabble in network the whole time I've been doing this job.

Lastly, I support my company. When there's an emergency, I am available to
handle it. I'm not busy managing several other companies disasters as the same
time. My business can also dictate my downtime. Oh, you're having a conference
in New Zealand during the time we have our normally scheduled email
maintenance, well let me just move that window for you so you can get your job
done. Don't forget, we're a service based industry and you're going to get a
lot better service from someone you pay to be dedicated to you rather than
from a company that supports a few thousand customers. You don't have to wait
in line for this car wash.

------
tastybites
The whole SaaS model just shifts the sysadmins around to where they are more
efficiently utilized. They're not disappearing.

~~~
mediaman
But part of being more "efficiently utilized" is improving labor productivity,
which all else being equal reduces the demand for sysadmin labor.

Although it might make for a more interesting sysadmin job for those that
remain.

~~~
ciaogiorno
It's not a zero sum game. There's no point in keeping around jobs are
unnecessary.

