

Ask HN: Is this true? - glen

We were in the process of hiring a CTO. One of our programmers suggested that server guys are not normally good coders; and programmers are not normally good server guys. He suggested that, in general, the fields have bifurcated and it is uncommon to find people who excel in both domains. Would you say this is true? mostly true? or BS?
======
elptacek
Likely this sounds petty and glib, but I have yet to meet a programmer who had
the kind of administrative sense that makes a 'good server guy.' The politics
surrounding change management and policy enforcement require technology
administrators to have at least one foot firmly planted in reality. This
doesn't keep server people from being proficient coders, but it suggests
there's an upper limit. And, from personal experience, since I switched from
full time admin to (mostly) full time programming, most days I couldn't tell
you what day it is. It really feels like a personality shift due to less
demand for administrivia and more demand for creativity. Four years ago, I
knew the location of nearly every object in our home, all the bills were paid
on time, all of my socks were matched up and we had a meal plan. Suffice to
say, I've started wearing mismatched socks and the bills get paid when someone
calls. Much to my delight, the world has not come to an end. Maybe I'll live
longer, too.

------
tptacek
I spent the first part of my career as a "server guy" (along with networks). I
ran tech at a popular ISP in Chicago called EnterAct.

I spent the next part of my career as a developer, first at a security
company, then at a streaming multicast company I founded, then at a network
managment company.

I spent the next part of my career as an almost full-time security researcher
for hire, with a pronounced focus on high-end enterprise technology (SAN, WAN
compression, replication, database, storage).

Looking at it from all three angles, I'd say: there's definitely something to
this. Being a programmer gave me insight into how systems worked. Being a
security researcher gave me _even more_ insight (I had projects that were
literally months spent reverse engineering equipment firmware and building
protocol test suites).

I'm confident that after all this "insight", I knew more about how, say, iSCSI
worked than _any SAN administrator in Chicago_.

So, how would I have done as a manager of a SAN array? Terribly!

The things that actually matter in administration --- understanding the
processes that need to be in place to make changes, understanding what kinds
of changes will occur, understanding what's a typical kind of failure that
will just take a couple hours to diagnose versus what kinds of problems mean
that your vendor is putting engineers on an airplane --- those things you
learn by being in the operational/engineering role.

To excel in the operational/engineering role, you need to dedicate yourself to
the things that matter for that role, and not allow yourself to get distracted
by shiny things. Developers and researchers: prone to distraction by shiny
things!

What I don't think is true is that strong tech people are predisposed to one
of these roles or the other. Could I be a strong ops/engineering person again?
I think so. But I know I can't do it while I'm doing this job.

------
snom370
At a certain point you're talking about SANs, FibreChannel routers, HVAC
systems, UPS, virtualization, etc etc, and keeping up to date on all of this
knowledge is a job in itself. So a guy spending most of his time doing "big
IT" sysadmin stuff might not have the time to become a really great
programmer.

That being said, I know plenty sysadmins that have gone on to become great
programmers. And truly great programmers tend to love computers, both software
and hardware.

Larry and Sergey were apparently also server guys:
<http://www.flickr.com/photos/bsmif/3465464623/>

~~~
nostrademons
Nah, Larry and Sergey were entrepreneurs, which meant they did whatever they
had to to keep their idea alive. Which often included hardware design,
sysadmin, algorithms, infrastructure, graphic design, accounting, investor
relations and marketing. They sucked at most of it, but they were "good
enough" to turn their idea into a multi-hundred-billion-$ company.

Those corkboard servers you show, BTW, were removed from service in a hurry
once the fire marshal found out they were building servers out of _corkboard_.

~~~
snom370
Sure, but the point is that in the beginning, they were arguably great at
developing software _and_ building a server farm that worked. My main point
was that until you hit the "enterprise" server stuff, you have the capacity to
be both a good server guy and programmer.

Above a certain point, there are so many technologies you need to know that
people need to specialize.

~~~
wdewind
In 99% of cases going with an ex-enterprise sysadmin for your CTO for a early-
medium stage startup is a horrible idea. Sysadmin needs are significantly
smaller than programming/engineering needs until you get to a really big
scale. So, while either is probably OK if they are overall smart and good at
tech management, my advice is go for someone with experience managing
programmers vs. someone with experience managing sys admins if you have to
choose. Ideally you get someone who has been CTO at a startup before and has
experience bridging this particular gap.

The third aspect of this is management. Someone who has been in management for
10 years is not going to be a spectacular programmer or sysadmin because they
simply wont be up to date, but they've got a solid tech background and great
managerial skills. These tend to be better later hires, but could be right as
well depending on your situation.

------
jk4930
Mostly true. Tautologically because programmers specialize in programming and
server guys specialize in running servers. Different problem domains with
different ways of thinking, tools, problems. And we are talking about real
programmers and real sysadmins and not some young hobbyists, I guess.

~~~
Locke1689
I think this is a pretty good characterization of my friend -- he's a great
coder, but I think he will always be a better sysadmin. Not because of some
innate sysadmin trait, but because he likes it better. You always put more
effort into what you like more.

------
jeromec
True. It's about specialization. You could also ask if programmers make good
database administrators. Highly capable/experienced programmers that develop
for the Web (like many on HN) will have some level of knowledge of all facets
of the network computing experience, including how routers and browsers work.
However, their primary focus and highest area of knowledge should be
programming, hence their title.

An analogy might be basketball. Growing up you learn to just get the ball in
the hoop, but those that turn professional specialize in a certain position.
Very few can play all five positions quite well (like Magic Johnson).

------
paraschopra
No offence but you why are you calling the position CTO if he is expected to
code and do sysadmin work. Why not simply call him/her an engineer?

Just curios on the choice of title.

~~~
lsc
do you seriously think a non-coder can effectively choose good coders?

do you really think a non-sysadmin can effectively choose good SysAdmins?

I mean, there is enough overlap between the fields that a good sysadmin can
usually tell if a programmer is completely BSing, and a good progrogrammer
(well, a C systems programmer) can usually tell if a SysAdmin is completely
worthless, but I wouldn't trust anyone who was 'pure management' to detect
either.

------
ihodes
I think that's BS. While I'm not a professional as either, I do both and enjoy
both – I didn't have the resources to hire my own sysadmin, but now I've found
that I enjoy it. I think to enjoy doing either requires many of the same
qualities: problem solving, patience, innovation, curiosity and vigilance.

Then again, there are jaded "specialists" who don't dein to dabble in the
"other" field, I'm sure.

~~~
glen
Good point about them drawing on the same qualities. Makes sense.

~~~
doron
You can enjoy both, but sometimes, and mainly due to lack of extra hours on
the clock. The focus needs to be maintained on often divergent set of
problems.

Not all programmers are very familiar with hardware issues, or dependent on
their area of programming expertise on network design.

I also found that many programmers are oriented towards creating new features,
while administrators focus on running established tools at maximum efficiency.

While drawing on the same set of qualities, the interpretation of them can
diverge.

Some people posses both sensibilities and the mind space to treat both
disciplines well in environments and projects that scale. They are not common.

~~~
lsc
maximum efficiency, bah! a good sysadmin is focused on keeping shit from
breaking. If that means putting up an active-active redundant system where you
really only need the capacity of one system, so be it.

Those of us who get paged when the shit hits the fan tend to be pretty careful
of breaking things; and quite often, we work on systems where relatively short
downtimes can lead to losses greater than our yearly salaries.

That said, all the best SysAdmins have at least some background in C and _NIX
systems programming, and nearly all_ NIX systems programmers are also pretty
good SysAdmins.

------
viraptor
When you don't expect much at the start, one person can do programming or
sysadmin job first - if you're just hosting your own page without any special
features. If you get to the situation where you have many servers to take care
of (5+ let's say), then sysadmin simply becomes a full-time position that
needs someone experienced with this environment. Someone who's mainly a
programmer may not have enough real-world experience to deal with the
potential problems, or might not know where to look for solutions. If you
require quality, you'll need specialists at some stage.

Splitting the roles is also practical. At some point you may have a project
deadline coming soon, when suddenly (for example) network fails - do you
really want to send experienced programmers on a trip to the datacenter in
that case?

Really, it's a question of how big you are and how much time does your
infrastructure need...

~~~
lsc
I pretty much agree, though really, a programmer who cares enough can carry
you well past 5 servers. A good SysAdmin can handle a ridiculous number of
servers; I've worked places with north of 10K servers per SysAdmin. (of
course, here the hardware monkey and the SysAdmin job was split; you need more
hardware people than that, unless you are outsourcing your hardware to amazon
or the like. )

------
delano
That certainly used to be true for the most part. The lines have started to
blur:

[http://jedi.be/blog/2010/02/12/what-is-this-devops-thing-
any...](http://jedi.be/blog/2010/02/12/what-is-this-devops-thing-anyway/)

------
thwarted
As someone who self-identifies as both a systems guy and a programmer, I'm
continuously surprised by how many sysadmins don't read any C code and by how
many programmers can't effectively configure Apache, to give two examples. I
really don't think I'd be as effective at my job as I am if I didn't have both
backgrounds, didn't do both regularly, and haven't held positions where I'm
expected to do both. Often times, it becomes institutionalized to have the
developers "throw their finished products over the wall" to the systems team.
I've found this doesn't result in a good environment, rarely do the
programmers know about the deployment environment, and systems people don't
get a chance to provide input on issues of scalability and maintenance.

One guy who was an early mentor of mine in systems work actually hates doing
it, and prefers to program exclusively. I didn't know this when I was under
him, I credited him with helping me get to where I am today but he'll hear
none of it.

I do think it's true that is is rare to find people who do work effectively in
both domains, or even like to. There is a sort of closed mindedness on each
side toward the other -- come on guys, we're all engineers! I've had technical
and non-technical people alike be surprised at where I can contribute (a
memorable one is someone in the accounting department at an old job who was
forced to admin a machine back in the day being surprised that I could code,
and more recent one is someone being surprised that I was able to successfully
contribute to dealing with IE layout issues -- in both cases it's been like
"but you're a systems guy, you can't do X", because they've only ever known me
to be in that role "officially").

------
orev
True. While many programmers seem to think they can also do server stuff, as a
15 year veteran "server guy", I can say that I've rarely, if ever, seen a
programmer who makes a good server guy. The things you need to focus on are
just too different.

Programmers tend to focus on the very small details and seem to only look at
the short term question of "if something is possible to accomplish". A server
guy needs to think about the big-picture, with a focus on not only if
something is possible, but the impact of a particular approach on the rest of
the infrastructure and the long-term maintenance cost of any given action.

~~~
xilun666
The contrast that you describe between the state of spirit of sysadmins vs.
programmers seems to mean that programmers do not need to consider impact on
the rest of the architecture and the long-term maintenance cost of any given
code change.

The programmer you are describing are a kind of programmer who makes lousy
programs. I've read enough of those programs to say: they are not pretty --
they do not work well in large scale -- and they cost a LOT.

------
jpcx01
I mostly agree with that. Probably because I consider myself a good programmer
and am perfectly fine to delegate all sysadmin stuff.

I think the point is they are distinctly different skillsets. Programming is
very different from setting up and maintaining systems. There's a tiny bit of
crossover, but not all that much.

Sysadmins do need a basic sense for coding, however they tend use different
technologies than the one people generally use to write web applications
(though maybe they'll release Bash on Rails sometime soon)

------
kogir
I've found that the best sys-admins know enough programming to automate their
jobs and debug application issues. They don't have to be the greatest
programmers, but they're fantastic debuggers.

The best programmers know all the system settings, services, and behaviors
that can affect their apps and thus have to know the underlying system.

I have yet to meet a truly great programmer who didn't thoroughly understand
the system he was running, or a fantastic sys-admin who wasn't a pretty good
programmer.

------
Hoff
While a good server admin or a good programmer with (relevant) generalist
skills and people skills can succeed as a CTO, you need a CTO that knows your
market and your goals and your organization and your customers, and that can
quickly determine what works and what doesn't.

While a CTO is solving today's problems with today's shiny and with shiny duct
tape, you also need a CTO that can identify and can hold both your short term
and your broader and longer-term organizational goals in sharp focus. You need
a CTO that establishes clear goals and measurements and trends, and a CTO that
can then push, pull, poke, suggest, prod, cattle-prod or ego-stroke toward
those organizational goals. And you need a CTO that adapts.

If a server admin or a programmer has the requisite adaptability and people
skills and technology skills and presentation skills and has an indefatigable
focus on making your market, hire him. Or her. But you're not selecting for
and not hiring for server admin or programming skills here.

You want somebody with the technological skills of and the focus of the Borg.

------
thaumaturgy
It depends on what you mean by "good" programmers or "good" server guys. Up to
a certain point, yes, there are plenty of people that can do both well enough
to get most jobs done.

However: unless the guy you're hiring happens to be an active developer for
the operating system that you're expecting to him to admin, then there's a
pretty low chance that he could be considered an _expert_ in both fields. Each
field is just too deep and broad, and requires too much time to maintain
current knowledge in.

A programmer can probably figure out a particular sysadmin problem, and a good
sysadmin can probably figure out a decent approach to a programming problem,
but while they're figuring it out, you're paying them and potentially also
paying for downtime or a slower development process.

------
xelfer
I'm a Linux Syadmin. I build large student computer laboratories, build,
configure, and maintain servers, with bits of desktop support thrown in there.
I'm not a strong programmer. Sure, I've made a few iPhone applications and do
a little bit of bash scripting, but I don't have the time in my job to
maintain the skills that an excellent programmer would have. When I've spent a
few weeks programming I get better, but when we don't have time to maintain
that skill, I can lose it pretty quickly.

I've worked with some amazing python programmers and without using it day in
and out there's no way I could have the in depth knowledge of all the python
libraries like they do, so compared to them I'm pretty useless.

------
gaius
Are you hiring a CTO or a sysadmin?

~~~
glen
Well, we ended up hiring a person already. I was just intrigued by the thought
that server guys are usually not good programmers and vice versa. What do you
think?

~~~
gaius
It's a matter of temperament rather than technical skill. Some people like to
really get their teeth stuck into a problem and work on it for years at a
time. Some people like to execute one discrete task and move onto the next.
Both could be equally good at writing code and configuring operating systems.
But one will be happier as a programmer and one will be happier as a sysadmin.
They might even produce the same number of lines of code a year, one a single
10,000 line program and the other a hundred hundred-line scripts.

------
wanderr
Programming is about problem solving. A good portion of system administration
is also problem solving. I know several programmers who are excellent system
administrators, and a couple of sysadmins who are OK programmers (but they
don't really enjoy it). Personally I'm more interested in programming but I've
definitely jumped into the sysadmin role on occasion to help solve problems
the regular sysadmins were struggling with.

------
malbiniak
In general? No. No to tue, no to mostly true, no to BS. Programmers are not
necessarily good coders, and sysadmins aren't necessarily good administrators.
Instead of worry about that validity of the generalization, what do you want
out of your CTO? What unfilled role in the team are you trying to fill?

------
madair
I think it's worth realizing that it's tribal behavior as much as anything
else. Corporate fiefdoms develop. Staff protect their turf. In my opinion many
people are unaware of the likeliness that this is the result of evolutionary
biology.

That said, the value of some level of specialization clear. We have to commit
a lot of energy to the things we will become really good at. However that
doesn't come without problems. In the scientific community there is concern
that it's becoming harder to do cross-discipline research.

While we must accept specialization for practical reasons, and realize that
there are also social and biological factors, there is value in encouraging
cross-disciplinary experience, _especially in leadership positions_.

The programmers may like to critique the potential CTO as not as good a
programmer as them, and the sysadmins may do the same. But I sorta think you
want a CTO who can see the forest for the trees.

(Cue criticisms of dumb CTOs....Now! ;-)

------
rssreader
I didn't read your question. Just came here to say that if you want me to read
your post, you need to write something a little more descriptive in your
submission title than "is this true", or "look at this", or any other such
non-content.

------
anigbrowl
True IMHO. Get a server guy, that's where the rubber meets the road.
Ultimately your business succeeds or fails on whether your service is secure
abd available.

------
jimfl
Programmers should be required to have been "server guys" because they would
have a better appreciation for building in logging, management, and other
troubleshooting aids. The sysadmin doesn't have the benefit of the debugger
when something goes wrong in production. The more information you make
available (on demand or as a matter of course) the fast production issues will
be resolved.

------
ww520
BS

