
Software Developers Should Have Sysadmin Experience - beekums
http://blog.professorbeekums.com/2017/01/software-developers-should-have.html
======
cs02rm0
This might be controversial, but I don't think you get to be a half decent
developer without being a reasonable sysadmin.

Maybe my experience is unusual, but I've never worked anywhere that the
sysadmins knew more than the developers about how best to run their code in
production. And when things go wrong with it how best to find the cause of the
issue.

And I've never thrown code over a wall without having tested it in a
representative environment.

The worst sysadmins get in the way of developers. Ones that scale down your CI
server to the cheapest, throttled, one the hosting company has, leaving
$800/day contract developers waiting for builds that run in 20 seconds on
their laptops take nearly an hour. And then try and argue the toss about
whether the CI server is cost effective and every few months keep switching it
down despite the CTO saying it needs to be left alone.

When a sysadmin sees an issue in "their" environment that they understand
there's a tendency for some of them to just see that issue as the only thing
the developer has had to deal with that month. In all likelihood, in a
productive company, it's the most trivial issue the developer has had to
resolve that day.

Often this stuff goes more smoothly where the developers (I mean, it's not as
though if you're going to drop one of the two groups of people it's going to
be them going) manage production and there aren't people with separate job
titles and the resulting friction between them.

Sorry. There must be great sysadmins out there struggling with terrible
developers, I'm sure of it. I just haven't seen it.

~~~
pjc50
I've done the dual sysadmin/developer thing for a small company, and the
problem I experienced there was completely incompatible working modes.

Sysadmins must deal with interrupts (requests, crises, things driven by
external schedules etc) and then in the rest of their time build systems to
manage or reduce the interrupts. Developers are expected to produce work on a
predictable schedule. This is disrupted by interrupts and obliterates the
schedule for proactive work unless your management is very good at making it a
priority.

The "prevention of information services" problem is certainly real though.
Perhaps it could be addressed by embedding the sysadmins in the dev teams
rather than having a department of their own, but then you have to fight org
hierarchy.

~~~
krylon
_AMEN!!!_ In my current job, I am wearing both hats, and while I like that
there is a certain variety in my work, users calling for help is highly
disruptive when programming or doing some other stuff that requires deep
focus.

The upside that in a three-person IT department there is very little
bureaucracy to fight, just the odd "organically grown" legacy system.

~~~
kefka
As an aside, did you know that the word "Amen" actually is a acronym in the
Jewish language that means "El melekh neʾeman" (or AMN) which translates to
"God, trustworthy King". (source:
[https://en.wikipedia.org/wiki/Amen](https://en.wikipedia.org/wiki/Amen))

I figured the etymology of that word was rather interesting. But yeah, I get
the whole SysAd/Dev dual job. They're tough to balance and do effectively.
SysAds are firefighters. When the nag(ios) alarm rings, we come a-callin.

~~~
grzm
From that Wikipedia same article, it says

 _The Talmud teaches homiletically that the word amen is an acronym_

The etymology section shows the word has much more prosaic roots. The Talmudic
acronym seems to rather be an interesting backronym.

------
eviln1
Hi, I'm a Sysadmin, and I've been a grumpy one through a larger part of my 15
years experience. My main issue was that Developers were acting like Users:
they don't care about what you have to deal with, they want things to 'just
work'. In return, I've treated them like children, in some instances yelled at
them when they did dumb stuff. I've tried to educate them when possible, and
was angry when the education didn't stick. At the time i was the 'King of the
Hill' type of sysadmin - natural leader of a very small and tight team, kind
of irreplacable, and with enough years in the company behind me to consider
myself as a demi-god.

When I switched companies, I came across better developers. Some had decent
sysadmin skills, but the main difference was that they actually took interest
in how things worked past the 'git push', and when I asked / required them to
make some changes that would make my life easier, they listened, discussed and
adopted when appropriate. With those same guys, I took interest in what they
were doing, what their actual job was and came up with ideas that would make
things easyier and run smoothly on both ends. After a while I figured out that
they weren't actually better developers - they were better people. (Also, I
figured out that being grumpy was not the best approach and that patience,
kindness and gratitude could get people to do more than snark, humiliation and
flame-throwers.)

I guess my point is: you don't really NEED to have sysadmin skills to be a
decent developer; what you really need is to care about what sysadmins do - be
curious, talk with them and trust them when they say that your brilliant idea
won't work in production.

~~~
morbidhawk
I think there are definitely developers out there that give little to no
respect to systems administrators.

I've seen this ignorance even in college professors. In my first programming
class in college I took a CS class that had both CS and IT students in it
since it was required for both kinds of students. The (CS) professor kept
trying to convince students how much better CS was and gave some good
arguments (ie: salary) but the most arrogant thing he said is that IT is a
subset of CS and that by doing a CS degree you would understand everything it
takes to be in IT. He also mentioned how in IT you would be constantly fixing
other people's computer problems but as a software engineer you wouldn't need
IT's help since you can fix it yourself. The funny part is part-way through my
degree I realized that college didn't even offer a real CS degree it was
called "CIT with Computer Science Emphasis" which none of my advisers nor
professors mentioned would cause issues getting jobs outside of Utah, the best
thing I did was leave that school and finish my CS degree elsewhere which
caused me to lose a lot of unnecessary credits and almost felt like I was
starting over. I feel like I got scammed but that's beside the point I am yet
to work for a company where a software engineer gets to manage his own
computer without following IT guidelines like my CS prof had described.

------
bandrami
As a grumpy evil sysadmin, I think the good Professor misses where the real
disconnect is, at least nowadays: stack management.

Why do things like Docker exist? Because developers got tired of sysadmins
saying "sorry, you can't upgrade Ruby in the middle of this project". Why does
virtualenv exist? A similar reason.

Containerized ecosystems (which is to say basically all of them now) are
really a sign of those of us on the sysadmin side of the aisle capitulating
and saying that developers can't be stopped from having the newest version of
things, and I think that's a bad idea.

15 years ago, when a project would kick-off, as a sysadmin I'd be invited in
and the developers and I would hash out what versions of each language and
library involved the project would use. This worked well with Perl; once the
stacks started gravitating to Ruby and Python it was a dismal failure.

Why? Because those two ecosystems release like hummingbirds off of their
ritalin. Take the release history for pip[1] (and I'm not calling pip out as
particularly bad; I'm calling pip out as particularly average, which is the
problem): in the year 2015, pip went from version 1.5.6 to 8.1.1 (!) through
24 version bumps, introducing thirteen (documented) backwards
incompatibilities. Furthermore, there were more regression fixes from previous
bumps than feature additions. You'll also notice that none of these releases
are tagged "-rc1", etc., though the fact that regressions were fixed in a new
bump the next day means they _were_ release candidates rather than releases.
Ruby is just as bad; the famous (and I've experienced this) example is that an
in-depth tutorial can be obsoleted in the two weeks it takes you to work
through it.

Devs are chasing a moving target, and devs who haven't been sysadmins may have
trouble seeing why that's a bad idea.

[1]:
[https://pip.pypa.io/en/stable/news/](https://pip.pypa.io/en/stable/news/)

~~~
AlexB138
As a Sys Admin turned Automation/Tools Engineer, I think you're missing part
of the point. You've got the beginning of it right in saying that Sys Admins
used to be involved in pinning down versions, and even in why that was
necessary, but I believe you're incorrect in saying that the containerization
technologies are bad for removing that.

Those technologies don't exist so Developers can get around Sys Admins and
ignore your helpful advice. They exist to solve that problem that makes the
Sys Admins role there necessary. It removes the underlying need for a Sys
Admin to worry about the versions. Admins should see this as a good thing, but
in my experience many dislike it because it takes them out of their Gatekeeper
role. We shouldn't WANT to stop the Developers from from having the newest
version of things. They aren't kids playing with toys that we need to nanny
over, they're doing work that creates values and the fewer things we do to get
in the way of that, the better.

If something breaks due to version changes, their testing should catch it. If
things are breaking in production, we ought to get involved because there's
some other problem, but before that we, as a profession, need to learn to get
out of the way and let people work by letting technology handle the problems.
The "Gatekeeper" mentality needs to die as quickly as it possibly can.

~~~
jstimpfle
> They aren't kids playing with toys that we need to nanny over

In my experience (university), yes they are, and they should do that at home.

Why do you need the latest bleeding versions in the first place?

In my sysadmin experience, people believe software gets bad and deprecated as
soon as the glory next breaking version appears. I don't think I need to argue
why this is an illogical stance.

With my developers hat on, bumping to the next version mid-process reliably
introduces more friction than is worth it. People think the next version
solves that one weird issue but ignore that it introduces two new ones and
that the software must be changed to fix five new incompatibilities.

But the solution reliably is to just not use that weird feature that caused
the bug in the first place, and think what a clean solution would have been.
And guess what, the result is a cleaner and more compatible code base. It's a
tip that works for me again and again: If there is friction, think - before
spending the next hours with an update that will soon lead to new problems.

It's great that you can for example compile Linux without too much friction.
It's great that arcane shell scripts can run on any system. Stability (in a
compatibility sense) is not a nice-to-have, it's basic sanity.

~~~
treehau5
My comment to you two is -- why not both?

Stability, sanity, all that is amazing, and a must have.

But also bug-fixes, security improvements, and performance improvements are
wonderful too, which tends to come with using up-to-date dependencies.

The problem with the latter, as you mentioned, is when it introduces breaking
API changes and is wholly not backwards compatible. This is not a "kids
playing with toys wanting to experiment problem" this is a bad software
problem, which is why I like Go, and why I liked Java when I was doing it full
time. If the language you use has backwards compatibility as a first-class
citizen, most likely the package authors will act that way too, and then the
maintainers, and eventually the developers. Limit your software choices to
those who care about not breaking everyone's shit every 2 weeks. Heck even
when I write my own API's now that I know only my company is going to use
internally I am thinking about this.

~~~
brightball
Backwards compatibility is seriously under appreciated. When I tell developers
to ensure that their changes are backwards compatible, they tend to look at me
like I'm green.

I do not understand the disconnect that developers have with understanding all
of the benefits that it brings. Yes, you have some extra code in your code
base so it's less clean. You also have a stable environment as a result. The
first affects only your personal preference. The latter affects all of your
developers and users.

Unless you have a situation where it's impossible to maintain, not insisting
on it is pure self interest.

~~~
gaius
_I do not understand the disconnect that developers have with understanding
all of the benefits that it brings._

Because they've never worked on an old codebase, because front-end
technologies change so often and everything just gets re-written anyway. It's
a waste of time worrying about this when the code won't make it to is first
birthday.

If you were speaking to seasoned C and DB developers about stability in the
tools and the platform, you'd be preaching to the choir.

~~~
bandrami
This gets to the complaint that so much of the open source ecosystem gets to
version 0.8.6 (whether it's named that or not) and then completely rewritten
"this time the right way". That's not actually a good thing.

~~~
prodigal_erik
As jwz put it,

> It hardly seems worth even _having_ a bug system if the frequency of from-
> scratch rewrites always outstrips the pace of bug fixing. Why not be honest
> and resign yourself to the fact that version 0.8 is followed by version 0.8,
> which is then followed by version 0.8?

------
nokya
I'd say there is too much effort in reasoning on the wrong problem. What
worries me the most is the 'why': why do (too) many software developers don't
know about sysadmin?

I have been involved as a consultant in large software projects in the last
two years and a vast majority of money lost in delays and bugs was caused by
devs not understanding: 1) the difference between virtual memory and physical
memory 2) the difference between costs of data storage per storage medium 3)
the concepts of network round-trips 4) and hardware bandwidths 5) how to
install and configure a web server on a workstation 6) how DNS works 7) how AD
authentication works 8) what ORM frameworks do 9) how to write a raw database
query (not necessarily sql) 10) the difference between navigating through
database records on a database server vs. an application server vs. a client,
11) HOW TO INSTALL THEIR OWN WORKSTATION AND TROUBLESHOOT IT!!! N) etc. and
those are just the topics that I can immediately remember.

As I see it, it's not about "they should". For me it's about understanding how
many devs deal with such a level of ignorance on the systems they interact
with, on a daily basis. This situation hurt my feelings everytime it happened
and I struggled to accept it. I am not a sysadmin nor a developer but my daily
work is insanely improved by my (even basic) understanding of how my
workstation works and how to manage it.

~~~
Aloha
I've worked with all kinds - from windows devs who can't figure out how to
install visual studio - to people who understand Windows, Linux, and macOS -
as well as basic system administration for each platform. The people who are
most successful at rapidly developing good high quality software are more in
the later group.

Would you trust a RF engineer who couldn't troubleshoot his own radio designs?
why would you troubleshoot a software engineer who can't troubleshoot his own
software as deployed in a real world environment?

~~~
0xC0DECAFE2020
I recently questioned someone about this very subject. They wanted to hire a
"CSS expert" because the "developer" didn't have a grasp of css after having
developed the project in JS/HTML. I was so confused as to how that's possible.

~~~
aegnor
There's a large gap between basic understanding of CSS and actually creating
good CSS. Personally I avoid touching CSS as much as possible.

------
pjmorris
Should sysadmins have software development experience (e.g. DevOps)? For what
values should 'X have Y experience?' Should we go as far as...

"A human being should be able to change a diaper, plan an invasion, butcher a
hog, conn a ship, design a building, write a sonnet, balance accounts, build a
wall, set a bone, comfort the dying, take orders, give orders, cooperate, act
alone, solve equations, analyze a new problem, pitch manure, program a
computer, cook a tasty meal, fight efficiently, die gallantly. Specialization
is for insects." — Robert Heinlein, Time Enough for Love

~~~
afarrell
I strenuously disagree with some of these.

> Plan an invasion

This is actually a massive undertaking. An undergraduate at MIT taking a
semester-long course on this will barely scratch the surface of it.
Furthermore, you're never going to suddenly and unexpectedly need to know
this. Any situation where you plan an invasion is going to be preceded by
spending a long time getting into the position where people trust you with
their lives and the fates of their nation.

> die gallantly

You're only ever going to be in this situation once, and probably not even
that. Why does it matter how gallant your heart attack is?

These skills make a bit more sense in a world where most of us need to march
off to war. Happily, we don't live in that world.

You should prepare for the situations you are only mildly unlikely to be in
and where your skill matters.

~~~
pjc50
It's literature, and SF, so it would be a mistake to take the quotation too
literally. But since we're here; the thing to remember about Heinlein is that
he was a romantic and a futurist at the same time. Hence his militarism wasn't
so much about the enemy as about providing an opportunity for romantic
heroism. Similarly with the other items in the list. It's not grounded in
practical necessity but in a "renaissance man" / hero of a novel approach to
being able to not just handle situations but show off in them.

------
blowski
Sysadmin knowledge definitely helps, but so does an MBA, knowledge of writing,
public speaking, design, user experience, networking. Oh, and the domain of
the problem itself.

The skills I require of my developers depend on the rest of the team and the
project.

~~~
wpietri
The difference with this is that server-side software has direct operational
consequences.

A server-side developer who never deals with ops is like a chef who never
tastes the food. It's in theory possible to get right, but in practice the
results tend to be poor.

~~~
danenania
That's true for backend, but frontend developers don't necessarily need a lot
of ops knowledge if the team has a good separation of concerns. If frontend
devs have to worry much about sys admin issues, I'd say that likely points to
a flaw in the way things are being done.

Of course, the more someone knows, the better. Knowledge and experience in any
area of technology can improve understanding of all the others. But nobody has
time to learn everything, and there's way too much to learn. Time given to
learning more about sys admin issues is time lost to other potentially
valuable knowledge.

It's a good area to learn about, and it is essential to being a strong backend
developer, but through good architecture and management practices, there are
still plenty of ways to make very high value contributions for a developer who
hasn't spent much time focusing on sys admin.

~~~
eropple
I think your definition of "sysadmin issues" is probably a little more limited
than mine or 'wpietri's. I've had frontend developers insist that they could
bake configuration settings into their webpacked artifact...which needed to be
deployed into multiple environments because of _course_ we weren't rebuilding
something that had already been okayed in QA when we wanted to send it to
prod.

You might say that's not a "sysadmin issue," but I have seen it happen three
or four times now and in each case it was the "sysadmin" (read: devops
engineer) who caught the problem and explained it to the offenders in
question. (Maybe it's a "build engineer issue"...but at most companies I've
seen, he or she is probably the "sysadmin", too.)

~~~
danenania
I would consider that sort of thing to be part of the required basic knowledge
to be a competent frontend dev, so if you want to call that sys admin, then
sure, there's a bit of it involved. If it's on your side of the fence then
yes, it's your responsibility. I'm not sure how many actual sys admins would
consider configuring webpack to be 'sys admin knowledge', but there is a
spectrum and it's true that there are some ABCs that everyone needs to know,
particularly when there are security implications. Still, it's a pretty far
cry from unix, web servers, and databases.

~~~
eropple
You might consider it basic. I would have, insofar as I considered it pretty
obvious even when I hadn't written a line of frontend code in five years, and
webpack etc. weren't even on my radar then. ;) But my experience has led me to
believe that it's not.

I think that's kind of the point of devops, though, is that something like a
build system _is_ a cross-cutting concern, that architectural decisions for an
application need to involve people across the stack. Classifying somebody as
"a sysadmin" is the problem in the first place, which is why I caveated my
post as heavily as I did. My experience is that your "devops" or "sysadmin"
people functionally become the "junk drawer programmer" who are relied upon
for all sorts of weird stuff. I've been at jobs (and at clients) where I had
to teach senior backend devs how to use VisualVM or what the implications of
using Kafka and CQRS are. I've been at gigs like the aforementioned where I
had to explain the ramifications of webpack to people knowledgeable and
capable enough to make React dance. And so my definition is probably
necessarily more broad...but it's also stuff I've had to do in practice, so,
enh.

And, 'cause fair is fair, I think a "sysadmin" who couldn't step in and write
production-quality code (allowing for a little ramp-up) is probably an
endangered species over the next ten years, too.

------
kabdib
I used to work on a product where the three major teams were Client, Server
and Ops. Tens of millions of customers used our stuff.

Client and Server folks were on separate floors of the same building. We
didn't interact much at all, except by trading bugs back and forth. The
management chains met at a VP. Three or four times a year we tried to
coordinate a release, and it always took at least a month. Getting a feature
out the door might take a year, with all the paperwork and pipelining of
release schedules.

Ops was in another building. The only things that both Client and Server teams
were sure of was (a) they did a bunch of customization of our stuff so that it
would work, and (b) they hated us.

Support was in another state. We were not _allowed_ to talk to customers.
Maybe once a year Support would fly in to talk to the teams about pain points.
I think we did an okay job addressing these, but it took a _long_ time, and
customers suffered a lot.

I won't talk about the disaster that ensued when Scrum was thrust upon us, or
the splinter projects that spun off to try to fix things (but wound up being
lots worse).

You _have_ to be close to the customer or you won't know if you're succeeding,
or even on the same page. You _have_ to know what your software is doing in
production or you're just sitting in an ivory tower pontificating about
angels-on-the-head-of-a-pin nonsense. You _have_ to spend time in the trenches
measuring and fixing stuff or you're hatching an unmanageable disaster. The
good news is that most of this is actually kind of fun. The bad news is that,
when managed badly, this can turn into a horrible and soulless grind of pager
duty and making legacy code even more legacy to fix wee-hours downtime.

I still get a rush when I get feedback from a real, live customer, and I think
that isolating your teams from customers is one of the worst things you can do
to a team and to a product. Getting teams to work on ops and support aren't
bad ways to improve this.

------
curun1r
We took it one step further on my team. The best way to ensure that the
pager(duty) doesn't go off at 2am in the morning is to put developers in the
first on-call group. Not only did it work, but the developers got a much more
nuanced view of how systems operate. When we first started, I'd see developers
using ping to determine whether a nameservice entry was correct. After a few
months of handling almost all of our own ops, developers knew how each piece
of the puzzle worked and how it all fit together rather than the hazy, largely
abstracted view they had before.

But we learned that we needed it to go the opposite direction too. We had ops
people who, when given a corporate-wide mandate to apply a security patch or
some such task, would log into every machine and apply the patch, despite the
fact that we'd been practicing immutable infrastructure with zero-downtime
deployments. We hadn't given them the necessary exposure to the dev side to
understand that you had to apply those fixes to a base image and trigger a
redeploy. There was a lot of finger pointing a couple of weeks later after it
was discovered that the fixes were overwritten by an application deploy.

~~~
adrianN
I hope you increased their salaries together with the new 2am pager duty.

------
gedrap
That's what I did at my tiny engineering team (3 folks) and results have been
great.

If you have a small organization (say <10 engineers), it's crucial that every
developer writing server side applications can also do at least some sysadmin
work. As the article says, it leads to deeper understanding and it really
helps when thinking about scalability, fixing certain kinds of issues, etc. It
also often shortens the feedback cycle and requires less throwing over the
wall. As a bonus, you increase the bus factor.

Automation is the key though. If everyone connects to the boxes and does
random things manually over ssh, nothing good will come out of it.

Still, you need to have a person or two who are responsible for the vision of
the architecture/systems and who make sure that things don't go off the rails.

------
kqr2
Somewhat related, I feel that mechanical engineers who design cars should have
some experience servicing vehicles.

Sometimes engineers will not leave enough space, use weird fasteners, etc.
that make a simple job much more complex.

~~~
userbinator
It could be a deliberate form of obfuscation/discouraging of repair, or just
the common trend of making things more complex than they really need to be.

~~~
slavik81
That's possible, but lack of insight into what's easy or hard in manufacturing
or construction is a pretty common problem. I saw it a number of times when
working for a few different manufacturing companies, though none of them built
cars.

A failure to understand construction concerns also played a role in the Hyatt
Regency walkway collapse. The original design was poor, but redesign to
address construction difficulties accidentally weakened the walkways further.

> Havens Steel Company, the contractor responsible for manufacturing the rods,
> objected to the original plan, since it required the whole of the rod below
> the fourth floor to be screw threaded in order to screw on the nuts to hold
> the fourth floor walkway in place. These threads would probably have been
> damaged and rendered unusable as the structure for the fourth floor was
> hoisted into position with the rods in place. Havens therefore proposed an
> alternate plan in which two separate sets of tie rods would be used: one
> connecting the fourth floor walkway to the ceiling, and the other connecting
> the second floor walkway to the fourth floor walkway.

> This design change proved fatal. In the original design, the beams of the
> fourth floor walkway had to support only the weight of the fourth floor
> walkway, with the weight of the second floor walkway supported completely by
> the rods. In the revised design, however, the fourth floor beams were
> required to support both the fourth floor walkway and the second floor
> walkway hanging from it.

[https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse...](https://en.wikipedia.org/wiki/Hyatt_Regency_walkway_collapse#Investigation)

~~~
wiredfool
There was another problem -- the beams were spec'd as box section, but what
was on the shop drawings was 2 channel sections with the flanges welded
together. (like this: []). The attachment point that supported twice the
design load was also compromised by the weld and less competent section.
Whoever was checking the shop drawings didn't pick up on the importance of the
change.

IIRC, either change on it's own would have been marginally ok, the two
together weren't. (by marginally, I mean probably wouldn't have killed people
but wouldn't be to code)

When I was going through civil engineering, there was a big push to use a
statistical basis for loads and resistances, rather than using a blanket
factor of safety. Loads vary, strengths vary, potentially normally, probably
not. But they're described by statistics at any rate. Blunders of this sort
aren't, at least on a per project basis.

------
xenadu02
I don't think "should have" is the right statement; more like having at least
one person on the team with sysadmin experience is extremely helpful. I know
it has been for me.

Of course in HS/college I ran a website that was a frequent target of hate in
the late 90s/early 2000s and it taught me about XSS and CSRF before people
invented fancy terms to describe them. It also taught me about HTML/JS
escaping, DoS/DDoS, SQL injection, and how all your defenses are useless if
someone social-engineers their way into root and nukes everything. I have the
assumption that everything is compromised and user input is toxic waste burned
into my subconscious.

------
PaulRobinson
I worked for an ISP in the late 90s/early 2000s for about a year before my
final year at Uni that when I started had 2,000 customers and by the time I
left we'd got to 750k customers.

It was the best training I could have for writing software for the Internet.
Just a couple of months ago we noticed one of our core apps was not scaling no
matter how many docker containers we had spun up in Mesos. Spent a week
breaking it apart using that experience and being able to make changes
directly to the code base and being able to talk to devops in a language they
understood, and in under 5 days we managed to identify and fix 7-8 different
issues.

I would value a dev with sysadmin experience _far_ higher than one without in
a tech business with a headcount under 500: it's going to lead to fewer
problems and issues in the short and medium term.

------
just2n
Anecdotal as it is, I started my career working as a sysadmin while studying
CS. When I was younger I was quite interested in security (e.g. I followed
defcon and CVE lists and read a lot of manuals and source code). I built
server software, broke applications, and did a lot of reverse engineering. In
that time I was forced effectively to learn about how the OS worked and how to
utilize it, both Linux and Windows, from C APIs to scripting, package
management, and everything else needed to effectively work in those
environments, primarily around reverse engineering and vulnerability research.

That experience naturally lead to me becoming a sysadmin during my study at
university. It was a fairly straightforward application of what I already had
learned with a much larger scale of management. The primary thing I gained out
of it was a drive to automate everything. When I started that job most of the
sysadmin work was manual, but a few of us spent a huge amount of time focusing
heavily on automation and when I left most of the work was automated and we
were just doing meaningful firefighting and supporting development.

As an engineer, the main benefits have been understanding how my software is
going to run in an actual software/hardware stack, easily jumping into a
production environment and debugging complicated issues, being able to quickly
have my OS do what I want, and that drive to automate everything. A lot of
that informs how I build software and in general it feels like it makes me a
lot more productive.

~~~
AnonymousPlanet
I have a very similar story as a background. To add to what you said: while
maintaining and debugging software installs, I learned a lot about how and
when things break. Especially, how important it is to keep things simple. This
turned out to be invaluable when I began working as a software developer after
graduating.

------
tyingq
I would change it from "should have sysadmin experience" to "should have
operating system knowledge, or ask the right questions".

The example in the article isn't what I tend to see in real life. What I do
see are things like:

\- Not knowing about various limits (number of open sockets, or listen queue
depth for example), how to know you've hit one, how to deal with it.

\- Not handling various error situations correctly (can't open file due to
permissions for example)

\- Security issues. ( socket listening on 0.0.0.0 when localhost would work,
for example)

\- Making assumptions about things like "current working directory" or
"certain environment variables will be already set for me"

For many of these, including a sysadmin or system knowledgeable architect in
the right discussions would suffice.

------
arca_vorago
This is a short little article that just barely touches on a much deeper,
often hidden issue; The state of system administration in business is abysmal.

It's not the developers fault though, at least not as much as devops types
would like you to believe. I think the author has a good point, in that its
good to get Devs thinking about real world environments on deploy, but the
real world is much more complex than concurrency of servers.

All that being said though, very rarely have I as a sysadmin of 10+ years seen
problems so easily attributable to devs. Of course I haven't lived in hn/sv
startup land either, so take that into account, but failures in systems I have
seen have almost always been a failure of management, up to and beyond C
level.

I could go into detail, but I'll save it for another time. Suffice it to say,
what businesses need to be doing is getting better CTOs and CIOs who can
bridge the gaping chasm between sysadmins and managment.

Devs, you keep being Devs, and let the sysadmins be sysadmins. Cross train and
communicate when you can, but don't fool yourselves, it is management that
bears the responsibility and burden of you both. Management just doesn't like
to admit that to themselves or anyone else, so don't play into this Devs vs
sysadmins dialectic too much, lest ye find yourself the scapegoat next go
round.

~~~
eropple
I agree that the dialectic shouldn't be played...but not that it's "cross-
training." Rather, it's the same skillsets being applied in slightly different
ways. There was a day in which "sysadmin" very often meant "shit-hot Perl
slinger." It was before my day, but I know some of the graybeards who can
still lay claim to it.

The sysadmin who can't write good, maintainable code is going to rapidly see
their positions reduced to sinecures in large and slow companies. That may be
enough to finish out a career, but I wouldn't bet on it if I was under 50 (and
I don't bet on it; I have always, as said elsewhere in this thread, framed
myself as "a developer whose output is configured systems" rather than "a
sysadmin" for this reason). Similarly, in an age where infrastructure-as-code
is becoming the norm, you had _better_ be able to work with it or, as a
developer, you are very limited in what you can do without being blessed by
somebody else--and the set of environments where that's gonna fly is
shrinking, too.

This is emphatically not to take any weight off of the shoulders of
management, to be sure. But rather that I feel very strongly that what divide
existed between these "disciplines" no longer exists, and both developers
_and_ traditional sysadmins need to move to catch up.

~~~
arca_vorago
> The sysadmin who can't write good, maintainable code is going to rapidly see
> their positions reduced to sinecures in large and slow companies.

I completely disagree, but respectfully. What I think we are lacking in this
discussion is a differentiation between what we mean by sysadmin as a product
and as a job description. Systems Administration is a aggregate management of
the technical infrastructure. (for $reasons).

In super small technical infrastructure, such as webdev startups, there is
very little active sysadmining to do if done properly, but as complexities
grow, you need multiple people to perform all the various duties needed to
maintain a system. In the performance of these duties, there are various
descriptions with varying levels of requirements.

What I postulate is that traditional businesses in seeing the agility of the
dotcom and startup culture have been attempting to cut costs in the technical
infrastructure, but for the most part because the managers of the technical
infrastructure (the sysadmins) haven't had anyone to advocate on their behalf,
this has resulted in poor business performance bottlenecks.

I would be curious to hear what other have to say, but in the "shit-hot Perl
slinger." days, usually the senior sysadmin was the slinger, and he reported
directly to the CFO and CEO, if not ownership.

What I argue is that systems have gained in complexity from a computing
infrastructure standpoint such that this old structure no longer worked well,
hence the creation of the CTO/CIO class(very similar). The problem is, they
aren't doing a good job in my opinion. Therefore, in this current business
climate, my argument is that the main problem is we already have good code
slinging sysadmins on hand (or can train them), but what businesses need/the
industry is lacking, is business/political game aware senior sysadmins to make
up for that failing of the C's. Hence I disagree the days of non-code slinging
sysadmins are going anywhere. Indeed, some of the best senior sysadmins I know
live in meetings, but if they are performing well in fulfilling that role, and
not slinging code, but instead making big picture, high-level overview
decisions and then monitoring progress, I don't think there is anything wrong
with that. Now, if we got CIO/CTOs back on the right track, the
political/business game demand could fall and those sysadmins could get back
into hard work, (which just happens to sometimes involve code-slinging to
solve problems.) Devs are an entirely seperate entity, but still part of the
process, in anything other than a pure web startup type businesses with little
to no real infrastructure (eg, workfrom home contractors).

Of course I want to qualify this in that this is ancedotal and I admit I
haven't seen every environment, but I have seen many environments (as a
contractor, seeing more insides than the guy going for retirement), from
fortune 500s to 2 man lawyer shops.

Any business that can see this issue coming and address it head on will be far
ahead of the game. Those that don't, will one day have very rude wakeup calls
as the complexity exponentially increases and they don't have the structures
in place the handle the demand, mostly due to lack of foresight/vision.

~~~
eropple
To me, you're just describing another layer of management. Which is fine--
management is important!--but it's not the same thing as the actual
implementor class that "sysadmin" usually refers to. And those people are
inexorably going to be doing their administration via code. There isn't enough
time in the day to waste with "pet" servers. They're cattle. Infrastructure as
code is here, and it's just going to grow more extensive. If you can't write
code effectively--both in terms of stuff like "hey, this code can be read by
other people" and "this code can be automatically tested like any other
code"\--you're in _trouble_. One of my clients is a top-10 commercial bank in
the United States that has not only moved entirely into the cloud (save for
some legacy mainframe stuff that they're working on) and every team is not
only required to provision and operate strictly with automated tools (both
instance and cloud-level provisioners), but their non-database servers are
killed automatically every few months to ensure that they're rotating
successfully and without human involvement.

I'm pretty confident that I've seen where we're going with this. The
management piece of the puzzle is totally important...but the _practice_ is
going to continue converging with every other bit of software development.

------
gbuk2013
Software developers should also have customer-facing support experience
(supporting other people's code) so that they realise, the hard way, the
importance of having extensive logging and debugging capabilities accessible
to those unfamiliar with the codebase.

------
jedberg
Having someone who is cross trained is always better than not, but they will
be harder to find and more expensive.

A frontend UI person who can write their own backends is great. A backend
developer who knows javascript and can build their own frontend is great.

A coder who manages their own cluster is great too, as is a sysadmin who who
can write code.

And a developer who can do customer service, and therefore can fix a problem
for every customer at once through modification of the application, is better
than just someone who can answer the phone and make the customer happy.

All this is to say that someone cross trained will always add more value and
will also cost a commensurate amount.

~~~
eropple
_> All this is to say that someone cross trained will always add more value
and will also cost a commensurate amount._

Man...I _wish_ the latter was the case. I ended up going into consulting
precisely because of the way that salaried employees are valued in tech right
now. I literally _am_ all of those things, I am comfortable and have delivered
at a high level mobile, web, backend, and infra projects--but the "market"
loves valuing those roles individually, and not the synergistic capability of
being able to do all of them.

Consulting is fun, but finding a place where those talents actually are valued
would be rad. (Anybody out there: have an opening for somebody who can deliver
value at literally every level of your engineering organization while _always_
being game to help bring your other developers up? Email's in my profile. ;) )

~~~
jedberg
If you claim to be an expert at more than two, then chances are you're either
1) not really that good at any of them or 2) would make a great founder!

If you're good in multiple areas the only way to really get paid what you're
"worth" is to start a company and use your skills to create an amazing
product.

Because you're right: the market doesn't value a true polygot.

~~~
eropple
I don't claim to be an _expert_ at anything so wide as a full category of
software. Hell, I don't even claim to be an expert at a _language_. (I learned
yesterday about Ruby flip-flops, ferchrissake.) Merely (well, "merely") that I
can deliver a solid product at any of those levels. =)

As it happens that's the other side benefit of consulting--being able to build
a product with downtime. And I am! But it's still a lottery, and risk
mitigation is a thing. So I like hailing for interesting offers to come my
way.

------
rsingla
While I agree with the general sentiment, the examples feel fairly contrived.

What I think is more practical is understanding when certain problems fall
outside of your own scope or capabilities and when to engage someone else for
their advice.

------
icameron
Sysadmin or devops is responsible for uptime directly, being ultimately
responsible for the services to remain performant. It's a mindset and
temperament that may help developers see solutions or foresee pitfalls while
designing and implementing code. Zero-downtime attitude. Making and testing
deployments, upgrades and migrations to be least impactful as a core function
is probably the biggest effect of putting a developer in a sysadmins shoes.

------
fauigerzigerk
What he's talking about is systems architecture not sysadmin experiece.

Of course developers have to write code for a logical model of a machine that
isn't necessarily the same as the laptop they're working on. But I don't think
anyone would ever dispute that.

Storing pictures on a local disk when the software is supposed to run on a
cluster has absolutely nothing to do with sysadmin experience.

~~~
jessety
I came here to say this.

If a developer is building an app for 100K users the same way they'd build an
app for 100 people, the systems architect dropped the ball- not the developer.

------
pmontra
The post has a valid point even if the example is more about system
architecture than system administration.

In my experience a developer doing one year in an Operations department gains
some insight in what is going to be important in the longest phase of the
software life cycle, that is when it's exposed to customers and the company
has to react quickly to problems. Proper logging, anything that can help
pinpointing the cause of problems and even hot patch them at 3 AM.

The usual lean startup might have little thoughts for that (is it even going
to have customers?) but established businesses do.

If it's 6 months to develop and it runs for 10 years (with maintenance and new
features), which phase is more important to a company, development or
production? I'm a developer which did a couple of years of operations and I've
little doubts about it. It's where the company makes the money from.

------
bthornbury
With the rise of devops (docker, puppet, ansible, etc...) , and cloud VMs, how
often are sysadmins different from software devs?

~~~
peterwwillis
That question doesn't make any sense at all. That's exactly the same as
saying: with the rise of do it yourself home repair, how often are plumbers
different from carpenters?

~~~
bthornbury
Not the same at all. Let me rephrase the question.

Reading the article, the sysadmin is described as the person who constructs
and maintains the environment the code runs in, including the physical
machines, but also things like dependency management (I infer).

Given that just about everything except the physical machine can be handled
through devops automation (from experience), and those things don't require
"special" skills other than docker/ansible knowledge, why are sysadmins still
considered separate from software devs?

On a more opinionated note, I think that using people, and not code to setup
the operating environment of their application is pretty inefficient nowadays.

~~~
Intermernet
I started as a dev many years ago, then moved into sysadmin / devops stuff for
over a decade, and then moved back to more hands on dev stuff again a few
years ago. I think your statement that "those things don't require "special"
skills other than docker/ansible knowledge" is over-simplifying things a bit.

Writing a production quality Dockerfile or Ansible Playbook requires more than
just a bunch of "apt-get install" statements. You need a bunch of knowledge
about how each bit of software should be configured and tuned for each
individual use-case.

Once it's been written, then the devs can happily automate all the things as
much as they like. Writing it in the first place is where a lot of what was
traditionally sysadmin knowledge comes into play.

Anyone with a bit of patience and practice can follow a recipe in a cook-book.
Not everyone can _write_ one.

~~~
bthornbury
I'll agree that I definitely oversimplified it. I think what I was getting at
was that it was a similar set of skills to other types of software
development.

------
mybrid
UC Berkeley's approach to teaching computer science I believe is the correct
approach.

1\. In CS150 one has to build a hardware computer. 2\. In another class one
has to build an OS. 3\. In another class one has to build a Database. 4\. In
another class one has to build a Network.

Education is key. Understanding the TLB inside the CPU gives one an
appreciation for context switching between the only 2 users the CPU has
hardwired into it: kernel and user.

Being a sysadmin is insufficient experience as education. Every developer
should know how to build a hardware computer from the ground up.

Computer science at Berkeley doesn't have a "Java" class. The language one
programs in changes depending on the Professor and topic. That way one doesn't
get too attached to a language.

Most developers do not understand the hardware implementation of a thread, or
now-a-days a docker. As a result both get used wildly incorrectly.

Threads are useful blocking on IO to keep the CPU from starving. However,
switching threads simply to swap CPU bound jobs is inefficient. Just adding a
thread doesn't guarantee equal service to users just like adding processes
does not.

Developers are using dockers today who have no clue why they are doing
so...it's just that everyone is doing it and they do not want to appear
stupid.

Experience as a sysadmin is no substitution for education.

------
Steeeve
The unspoken elephant in this thread is that sysadmins and developers have
broken relationships surrounded by organizational dysfunction.

Personally, I feel that system administration experience of some sort is
essential to development skills, and development experience of some sort is
essential to system administration.

Communication and leadership skills are essential to both. It _used_ to be OK
if you were the jerk at the office if you were good enough at your job
(sysadmin or developer). That's no longer the case because we've progressed
well past the point where a single developer or sysadmin can be enough of an
asset to look past their ability to work well with others.

Operations needs stability. Developers need to hit moving targets. Sysadmins
need to keep things manageable, compatible, and secure. These aren't
incompatible goals, and usually when you get the techies talking they are
sympathetic towards each other's challenges and helpful to one another. With
as many project managers, engineering managers, manager managers, directors,
and executives in the mix I wonder if the problems in all these environments
isn't more of a historic issue with leadership failings.

------
jcadam
You do not know pain until you've had to deal with government sysadmins as a
contracted developer. And the govt has a strong preference for bringing
sysadmins into the civil service rather than contracting that work out (like
they used to) these days.

Which means the ones with actual, marketable skills aren't working for the
government.

Spending _months_ waiting for a sysadmin to perform a task you _know_ takes
about 3 minutes is great fun.

------
vinceguidry
I think the problem here is that decent admin jobs are getting harder to come
by these days. Used to be any reasonable organization needed an admin just to
keep systems straight, now they all rely on less-skilled help desk types and
make it up with outsourcing companies, which in turn are shedding personnel
and relying on big vendor contracts rather than individual skills.

I predict sysadmin culture and skills will be the next Fortran.

------
AliAdams
The underpinning logic seems false.

To extend the analogy of the house on a slope, if you don't tell the developer
you need a house designed to be built on a slope, that is a problem with the
spec provided to the developer.

Although it can at times be more efficient for a dev to have sysadmin
experience, making blanket statements _requiring_ all developers to come with
such a background is just sensationalism.

~~~
sah2ed
Wholeheartedly agree.

I think the lesson the author is really trying to convey is the importance of
learning about _distributed computing_ [1] concepts and not necessarily
_system administration_.

[1]
[https://en.wikipedia.org/wiki/Distributed_computing](https://en.wikipedia.org/wiki/Distributed_computing)

------
b212
I'm a front-end developer with very little knowledge of sysadmin stuff, but I
can't agree more, the best developers I've ever seen had massive sysadmin
experience.

Can you point some good resources about sysadmin I could use to improve my
knowledge on the subject? Note I'm a noob when it comes to all this server-
side magic.

~~~
swozey
When I was getting into ops 10 years ago the book "The Practice of System and
Network Administration" by Thomas Limoncelli was a great overview (very high
level and not specific) that we all read.

The Phoenix Project by Gene Kim is kind of a more modern uptake on it, but
both can be read easily by people outside of the field and I doubt the first
one will ever not be relevant.

Both are more about establishing proper ops methodologies. If you're looking
for something more low level, like a Programmers guide to sysadmin I can't
think of anything off the top of my head.

~~~
schnable
Limoncelli's newer book _The Practice of Cloud System Administration_ is
awesome. It's really more about how to build distributed systems to performant
and operational.

~~~
eropple
I started writing a book about cloud system administration before I read this
one, FWIW. It's worth your time.

------
ohstopitu
I've had to work at start ups where I was one of the initial developers (and
the most experienced in the team) so I've been forced to learn a lot about
Sysadmin and design (and a lot of surrounding dev tools for instance) but all
in all - knowing more about sysadmin has made my dev experience change - I
think a lot more about how it's deployed, how it works in prod, how to get
data to improve my code and so on.

I do feel all devs at one point should have some sysadmin experience - if not
professional, then informally.

That said, sysadmin is strongly divided into 2 types of tools - configuration
and code.

I love tools that can be coded (as an example - Gulp), over configuration
(like Grunt) - as they generally have a lot less undocumented "gotchas" and
getting started is generally a lot more simple. (I'm looking at you
Webpack!!!)

------
kweinber
A great reason to have cross-experience at the management level is to
understand the different motivations of the two roles: * devs are incented
(paid) to introduce changes to the system...mostly in the form of new
features. * sysadmins are incented to stabilize the system and make it scale.

In organizations with uncoordinated or siloed management, this leads to
infighting... sysadmins try to stop "troublesome" devs from making changes at
all and actively slow down changes in the name of stability and developers at
the other extreme try to get changes in without caring about stability and
blame sysadmins for that.

Great management with experience in both will find ways to incent the teams to
introduce regular change while maintaining stability.

------
raarts
Wherever I went, I have seen the war between developers and sysadmins being
fought. In enterprise the sysadmins have the upper hand, in frontend, mobile,
SMB the developers.

Developers don't have the burden of maintaining and upgrading multiple
applications on the same server, with conflicting dependencies. 24/7\.
Sysadmins don't feel the pressure by end-users to deliver new features
yesterday.

At the moment developers are winning because Docker. But when they grab that
responsibility they will be the ones called when their 50000 containers are
being hacked because vulnerabilities. And they will be under fire for the
system being down and that costing the company a lot of money, so everybody
breathing down their necks.

------
krosaen
I think it's more accurate to say that good developers need to have some level
of systems expertise which probably means OS & networking fundamentals +
practical Linux experience. This is because you need to have some
understanding of how your program will execute on (or across) machine(s), and
some know-how to get your program up and running on a real machine available
to users on a real network. Whether you need to be an expert in the latest
best practices for managing a fleet of servers compliant with whatever
regulations, that's more for the sys admin (though it never hurts to learn
more, there's just so many things we developers could spend our time
learning).

------
fnj
It's hard for me to "get into" developers who don't know how the system works.
I don't mean they have to be full-fledged sysadmins, but it just seems natural
to me that they should have a decent understanding of the electronics, how to
assemble the hardware, and what's involved in tuning it and keeping it all
running.

It may have something to do with my education being EE, but that's not the
heart of it. I never practiced EE. I taught myself (more or less in order)
building circuits, programming, building PCs from pieces, and elementary
system administration. I know there are lots who never "get" all three, but
the best I knew did.

------
saosebastiao
I get the sentiment. Personally, I could say the same thing about control
theory, systems theory, economics, operations research, and statistics. And I
could also share dozens of anecdotes of bugs and suboptimal software that was
produced as a result of this lack of experience.

But at some point, you have to acknowledge that developers can't know
everything. Go deep or go broad, but accept that both paths have their
limitations and benefits. Maybe in some cases you'll run into a bug you
created due to your lack of sysadmin experience, but don't feel bad about
it...your experience has led you to other types of expertise. Fix it, learn
from it, and move on.

------
al2o3cr
In the case described by the author, it seems like most of the trouble would
be resolved by the developer understanding the concept of "distributed file
storage is hard" and coordinating with ops early - the dev doesn't necessarily
need to know the details of setting that up, but they need enough
understanding of the domain to realize there's work to be done on the issue.

For that matter, I suspect most sysadmin / ops teams would be happier if
developers could only strip "just" out of their vocabulary (and vice versa) -
nothing more annoying than "could you JUST do {thing asker doesn't understand
and thinks is trivial}"

------
protomyth
Actually, I would rather language and tool designers have a go at a true
System Admin job. There is a reason PHP gets installed by default and Ruby on
Rails does not.

Frankly, at the end of the day, most businesses don't want something that was
"working" breaking. All the conflict comes from that one desire. There is a
reason System Admins have to do the "Patch Tuesday" dance and despise any
software that just updates by itself. Getting yelled at because Chrome updated
to a version not supported by your ISV or your local developers is an
amazingly fun experience.

Yep, System Admins have to be the company nanny. It sucks, but the apps need
to keep working.

~~~
ihattendorf
True for the most part, however PHP is a language while Ruby on Rails is a
framework.

~~~
protomyth
PHP comes with all the stuff to make a website. Ruby needs a framework, and
Ruby on Rails is the most popular. I think you proved my point for me.

------
jordache
Just like how you typically distinguish front and back end developers, went
wouldn't you distinguish dev from devops?

Sure, it helps if an individual is not entirely myopic, and would be great if
that person can wear multiple hats with skillset spanning all those
disciplines.

However those different labels exist primarily they align to unique mindsets
and solutions to a problem in a mutually exclusive manner.

Do you want a jack of all, but master of non?

Seriously, how many individual out there do you think have the capacity to
develop their career so they have deep capabilities for all of those
disciplines?

------
joshwcomeau
I think that there's good advice in this article, but a couple counter-points:

\- Plenty of software developers work for small- to mid-size companies where
the Node or Rails server running on your machine is nearly identical to the
one running in production.

\- Plenty of software developers work exclusively on the front-end, which
means that the computer your code runs on in production is very very similar
to the one you develop on (although then you have the added concern of testing
on lower-end mobile devices, other browsers, etc)

------
seanwilson
I think everyone would benefit if they learned just a small amount about what
their team members have to deal with. For example, project managers should
learn a little about what programming involves while managing programmers and
graphics+UX designers should learn a little about coding to make their designs
more practical (i.e. identifying cases where a slightly better design requires
a massive amount more coding effort and probably isn't worth it).

------
teekert
I think many professions should have sysadmin knowledge. In my work (biology)
there is a lot of unnecessary clicking, not only in Excel (where Python/R
would be much more efficient) but also in web interfaces to transfer data
daily. The latter is often completely automatable using cron and rsync over
ssh. Many repetitive actions are a shell script or command line pipe away. But
people don't know.

~~~
mmirate
> automatable using cron and rsync over ssh ... shell script or command line
> pipe ...

Where I come from, that's called "knowing how to use a computer".

Or at least it _was_ , until Windows et al arrived.

(Interesting how the user-interface technology advance called the GUI,
_decreases_ the efficiency of the human/computer interaction.)

Except ... for those who still regularly use a Unix-like OS, that knowledge
still is quite basic and, thus, surprisingly difficult to trumpet on a resume.

------
kapilkale
I understand the point here; it can be useful to understand the abstractions
on top of which you are working.

But given most engineering projects are crud apps without scaling problems,
and PaaS companies like Heroku totally abstract away the system adminstration
piece for those projects... I certainly wouldn't recommend a new engineer
spend any time learning system administration.

------
vickychijwani
I'm moving from dev to devops for a while, because of similar reasons. Excited
to learn all these new things and peer beneath all the tools and abstractions
I rely on! I have no doubt in my mind that it will make me a better developer
- most abstractions have a tendency to leak, eventually.

------
pantulis
This seems obvious to me. You'll never see your code again with the same
attitude after being on call.

------
mrskeltal
As a developer with practically no sysadmin skills, how would I go about
improving these skills?

~~~
FLUX-YOU
Small/micro AWS/Azure instances. Set up your own CI/CD pipeline. Read sysadmin
books. Get comfortable with the environment shell (Powershell/bash/etc.) and
write some scripts.

Bonus points if you make an actual product that solves a problem and also
practice all of these skills.

------
0xC0DECAFE2020
I have been at this since the early nineties and can't comprehend that someone
can be a developer and not have any experience with How to set up and
administer the environment. I guess it's a normal thing now?

------
golergka
Just a small nitpick with the article: it probably meant "backend software
developers". Because a graphics developer who mostly writes shader code would
surely benefit more from artist experience.

~~~
TillE
It's incredible how apparently 99% of HN are web developers who don't even
bother considering that other things exist. It's not at all representative of
the larger industry.

~~~
arca_vorago
Very much this. I don't pretend to be a dev, but I listen and work with them
to accomplish business goals. Sysadmin work is very much more business
management oriented in the real world, but to see so many Devs say "I know how
to admin my software in prod therefore I am sysadmin too" just makes me
realize just how little the Hn/sv web startup dev knows about real world
business systems administration for established business.

Once again, like I said in my other comment, even in web startup land, this is
a failure of management, in particular CIOs and CTOs, and the management that
selects them.

The problem is that companies can usually manage to survive for quite a while
ignoring these problems. There is little short-term feedback mechanism for
duct taped systems... Until that day when shit starts dying and no one can fix
it. Or a cryptovariant hits the file server. Etc.

------
mmartinson
Careful with that should, or you might take someone's eye out.

------
partycoder
As a developer the way we resolve this is by including the sysops department
as a stakeholder that produces requirements for us to negotiate and implement.

------
adamdonahue
The general trend in software development appears to be toward generalization
over specialization. I'm not sure this is a good thing.

------
nanodano
Funny, I always see this from the other side. I always find myself thinking "I
wish sysadmins had more programming background."

------
EliRivers
Yes, sysadmin experience will give me vital insight into my work writing
embedded code for dishwashers.

Snark aside, if the post was headed "Software developers need to understand
the environment where their code will be running or they may not build it
properly", which is the point being badly made by conflating "environment"
with "some kind of commodity x86 with a standard OS", it would be a lot more
applicable.

------
dpaluy
Software developers should have experience in the things they need to get
their job done. This is not a revelation.

------
youdontknowtho
sysadmin really means "technical debt manager" most of the time these days.

I really spend the majority of my time trying to slot something into place in
a way that no one will notice that doesn't disturb existing stuff...often that
means duplicating bad behavior because its become a dependency.

------
imjustsaying
Maybe it's only because I've only ever worked with myself, but it's hard to
imagine a software developer who isn't also a sysadmin by necessity. How do
you debug things otherwise?

In what city or company is the barrier to entry for software developers so
low?

I just can't imagine paying above median income for a software developer who
can't install, for an example given in this thread, Visual Studio.

------
ultrasaurus
One thing I like about DevOps is never having to hear "it works on my
machine!"

~~~
yoz-y
I always take the "it works on my machine" as a synonym for "you need to
explain your problem better". In my experience most problems that "work for
me" are caused by somebody installing an unstable build and patching it with
random crap they have thought of.

------
tete
I think the reverse is much more true, but maybe nobody argues about that
anyway.

------
nier
This needs to be mandatory for developers working on printer software.

------
mrmrcoleman
Shameless plug perhaps, but this is free and will help here:
[https://www.cncf.io/event/webinar-
cloudnativenetworking](https://www.cncf.io/event/webinar-
cloudnativenetworking)

------
Diederich
I worked in the home office of WalMart Stores, the retailer, from the mid 90s
until 2009 as a hybrid network engineer/developer. That is, my team worked in
Network Engineering, but we wrote tons of code, because when there are
millions of different IP addressable nodes on a centrally managed network,
there will be code to manage it. (:

In that era, I think WalMart handled this apparent developer/sysadmin
confliact quite well. Believe it or not, even though we (the whole company)
had exceptional uptime, we were also extremely agile. LOTS of code was written
and rolled out across thousands of sites, multiple times a day.

There were a number of keys to this, and I'll enumerate a few.

1\. Most new hires (those that had minimal previous experience) to development
positions had to spend their first six months working at one of the four or so
major help desks. Note: they were paid their target salary, but in an hourly
fashion. 2\. The various operations teams had complete and final say so over
what went out, and how incidents were handled. A couple of the big operational
areas were Unix Operations, Network Operations, Windows Operations and
Mainframe Operations. I called them 'teams' but each had a number of teams,
and their own help desk. 3\. Any time there was an impacting problem
associated with a program developed by team X, a war room was called by the
relevant operational area. A member of team X (a developer) had to stay in
that war room (switching off between team members if it ran very long) until
the production problem was not only fixed, but also deemed unlikely to recur,
except in cases where that dept of a fix would take a long time. 4\. Every
development team had a pager rotation, with rigorous expectations about
responding to such pages. This was primarily to support the previous point.
5\. Because of the enormous operational scale, all of the major operational
areas had dedicated teams focused on automation. My team was that team for the
networking area. Furthermore, most of the rest of the operational folk
read/wrote code to some extent.

In short, incentives were aligned. Teams that wrote externally facing code
felt pain if the stuff they wrote and released caused problems. Operational
folks wrote/managed/interacted with tons of their own code in order to manage
the enormous infrastructure. Also, ops folk were far more willing to let
things move with velocity knowing that the people who actually wrote the code
would be required to support it, globally, any time of the day or night.

Another, perhaps even more important reason we were so successful during those
years (and the years before) was a strong and vibrant esprit de corps. The
entirety of Information Systems was, at the time, around 2000 people, and we
were facilitating double digit year over year growth over a 150 billion dollar
company. We had over 5000 remote sites in 15+ countries, with a diversity of
software and infrastructure that was honestly pretty astounding. Each of those
sites had quite a surprising amount of infrastructure.

We worked hard, and we produced huge velocity with fantastic uptime. For
example, the network achieved six 9s of availability a couple of quarters.

In end, while things were sometimes contentious, we trusted each other, and
only minority of teams were forced to work bad hours for any length of time.

------
timkofu
Agreed.

------
andrewclunn
Yes, yes. We get it, we should know all things, keep up with all the latest
rends, work 80+ hours every week, and do our company's IT support on the side
for free while we code their software. Or companies could compensate us for
our time and hire more god damn people if they need it. You know, just saying.

------
kahrkunne
On top of the 20 years of experience we already require of junior developers?

~~~
bdcravens
Probably preferable to calling someone a developer after a 12 week bootcamp.

~~~
ab1243
Just make sure they're 22, 23 tops. Can't have any of those 30+ year olds with
their early onset dementia.

------
madihaawan
As software developers, this should notice and kept in mind that all these
experiences are very much important. Thanks for sharing an important article.

------
A_Crazy_Idea
Managers should have management experience and less nepotism experience. One
step at a time, guys.

------
necklace
Ironically I can't see the images at the bottom of that page.

------
ovdoll
That’s interesting.

------
whenwillitstop
Sysadmin is getting automated. There is not reason to know it.

------
tzakrajs
I have ten years of experience in systems and have written tools software in
Python and Javascript for Netflix, Stanford University and Google (current). I
have never been a software engineer officially, but I want to be. I am a PC
gamer, anime lover, and a proud SJW. Any takers?

~~~
divanvisagie
What are you on about?

~~~
tzakrajs
I am interested in finding positions. Sorry if my post was ambiguous, I know
it is a bit out of place here.

------
WhitneyLand
No. Developers do not necessarily need any experience as a sysadmin and I
would advise against moving productive developers into this role for the
purpose of misguided training.

The argument about understanding how code scales beyond one computer is
fallacious. In fact it's quite easy to learn and practice all kinds of
distributed high scale architectures without getting up from a desk.

I will agree it's important to understand the people and work connected to
your job. It's probably a good idea to eat lunch with the sysadmin, or QA, or
UX person.

Would also agree its important to have respect, empathy, and collaboration
others, but it doesn't mean you need to switch jobs.

~~~
eropple
I _strongly_ disagree with this, from experience. There _is no difference_
between a "developer" and a "sysadmin" in a healthy shop in 2016. It's all the
same job. Until you actually put something into practice in a hostile
environment you have no idea what rakes are going to be in that yard. I would
be a bad developer if I did not understand how the systems my code runs on
worked. I would be a bad sysadmin if I couldn't marshal my systems through
code and fully and completely understand the software that would be run on
them. (As it happens, I'm not too shabby at either, and it's additive.)

The mindset that you are describing is why that "devops engineer" that gets
hired so very often ends up being burnt at both ends being the savior of the
team because, unlike the developers, he or she _does_ understand both sides of
the equation and is able to solve the problems that the incurious developer
cannot. I would strongly urge that you reconsider.

And, for your own sake, please excise the word "fallacious" from your
vocabulary. It will help you, I promise.

~~~
WhitneyLand
>There is no difference between a "developer" and a "sysadmin"

Ridiculous. Many developers have specialties that take years to gain
proficiency. If there were no difference we could just replace a math PhD
doing computer graphics with a sysadmin. Sometimes it's possible, but the
blanket statement does not hold up.

>I would be a bad developer if I did not understand how the systems my code
runs on work

I agree, but so what? Becoming a sysadmin is not the only way to do that. You
shouldn't confuse what you've seen work, with being the only way something can
work.

>[your] mindset is why "devops engineers" get hired and very often end up
being burnt at both ends

You’re reading way too much into this. I never said a sysadmin rotation
couldn't be helpful for some. I said devs don't _necessarily_ have to do it to
be great at their jobs. I wouldn't pull a dev who was, in the zone so to
speak, being highly productive into a sysadmin role misguidedly thinking it
would have no impact on delivery.

>[sysadmins are] able to solve the problems that the incurious developer
cannot

I never implied I wanted “incurious” devs who couldn't solve problems to the
extent they needed a “savior” as you put it. Maybe you are projecting your
past experiences onto others based on overzealous inference? I didn't say what
you suggested, and I don't believe it.

> please excise the word "fallacious" from your vocabulary

What are you talking about? The definition is “based on a mistaken belief”. I
claim the argument that devs cannot understand scale out architecture without
being a sysadmin is based on a mistaken belief. Why are you so against that
word?

~~~
eropple
_> Why are you so against that word?_

Because, to be frank: it makes you sound like a tool. That's what 'greglindahl
was referring to in his post, 'cause I said exactly that before I edited it to
assume good faith and be nice about it. But leading off your reply with
"ridiculous" confirmed that you're okay with that. I'm not gonna play this
game with you; I'm having way more interesting and way better conversations in
this thread with people who don't wanna dance that dance. Haveaniceday.

~~~
WhitneyLand
That's unfortunate it came across that way. I did not intend to do anything
other than make a good faith (and maybe passionate) argument.

------
cekvenich3
Nope. SD should not be using DB. They should use API to make REST |
Microservices, ex FireBase. SD should not be using servers. They should go
serverless, deploy static HTML to CDN>

etc. If you are still using docker, puppet or worried about scale/security...
you are living in the past. (one ex: [https://github.com/struts3/what-
demos](https://github.com/struts3/what-demos) )

~~~
petra
You're right.But like any big change it will take time.Say 5-10 years to most
new stuff being serverless.

