

The danger of having system programmers around - preek
http://utcc.utoronto.ca/~cks/space/blog/sysadmin/SystemProgrammerDanger

======
mechanical_fish
This, needless to say, is what those people who talk about the virtues of
"diversity" are trying to say. And the virtues of a diverse team of
specialists is, in turn, the theme of most caper films (e.g.: _Sneakers_ ).

Especially in emergencies, it can be really great to have three or four people
attack a problem from different angles. What you need is someone reading the
docs _while_ the system programmer is running strace() _and_ the tuning guru
is reallocating processes and memory _and_ the customer liaison is on the
phone trying to figure out if they really need to be running this particular
enormous task on every page load.

The alternative, if you're by yourself, is to try to multiplex the roles, but
it can be extremely difficult to pull yourself out of systems engineering mode
while you are in there.

------
heyrhett
This reminds me of what my electrical engineering friend once did when his
washing mashine stopped working. He got out his multimeter, opened up the back
panel, and finally traced everything to an open circuit: the lid was open.

~~~
julius_geezer
Ah, yes. I spent five or ten minutes this year examining a washing machine
problem my wife reported. The problem: her husband had turned off the hot
water feed while he tightened the connection, then had failed to run it back
on. I believe she forgave him his negligence ...

------
maayank
This is a general programming issue, not a system programming issue. Not so
long ago at work, I had to implement a regex search over a Lucene document
store. Now, the thing is that we used the latest version (at the time) of
Lucene, 3.0. All the examples I found online and most of the documentation
referred to version 2. Naturally they didn't mentioned they were using version
2, I just had to scratch my head a few times while I tried to realize why the
regex API depicted there is not only different, but not even merely existing.
Turns out one of the design goals of Lucene 3 was to separate stuff from the
core Lucene to external libraries. So, after finally searching through their
SVN I found the source files for the regex libraries. I ran into some problems
with the way Lucene parses regex queries and relates it to terms and again, as
in the article, after hours and hours of searching through documentation I
resorted to just reading through Lucene's code, which indeed turned to be
fruitful for less the time I invested in finding and reading documentation...

I really was not sure what to learn from that episode. The logical conclusion
seems to be to try to read your way through the target source code, if you
have it, before investing the time to find and skim through documentation. On
the other hand, it feels like a really bad lesson. Time will tell if this
moral holds up.

~~~
pmjordan
I've had this go both ways before, too. I think the point to take away from
this sort of experience is that the best you can do is apply your best
judgment and re-evaluate as you go along. Often opaque documentation makes
sense after you gain some context by reading the source; it works the other
way, too. With experience, you gain a sense for what path to go down;
sometimes that (subconscious) heuristic is totally wrong, which is why it's
important to consciously re-evaluate on top of that. The current approach
isn't making you any practical headway, even though you're learning a lot
about the system in question? Take a step back and try looking at the bigger
picture; what are you _really_ missing? If you can, sleep on it for a bit.

Most of all, if you're getting it right more than 50% of the time (and I'm
finding it hard to imagine a way you'd do worse than that) there really is no
point beating yourself up about it.

------
jt2190
My friend and colleague lost his job over this type of behavior... He'd simply
go of the rails for days trying to figure out a problem. When he got his next
job, I advised him to always think about how the company makes money from his
labor. Sometimes it makes sense to "go deep", but often it's cheaper to let it
go. If you're dead set on figuring it out, do it on your own time, not the
company's.

~~~
Dilpil
The real trick is finding a company and industry where it always makes sense
to go deep.

~~~
evo_9
"that's what she said."

------
presidentender
Despite the "incomplete and frequently wrong" nature of documentation, I still
think a careful reading of that documentation is probably the best place to
start.

If it doesn't hold the answer, fine, we can look elsewhere. But it does hold
the answer often enough that reading that funny manual first all the time will
save time over the course of a career.

~~~
berntb
>>I still think a careful reading of that documentation is probably the best
place to start.

I used to work like that in the mid-1990s.

When I had a problem, first I thought a bit, then I looked at the
documentation -- and as a last option I searched (at the time, mostly Usenet).

A year later I'd become realistic (or lazy), I searched before reading the
documentation.

And after another year, I searched before even thinking about the problem...
:-)

------
Xurinos
First, there is no such thing as a "systems programmer". This is a useless,
career-limiting, HR-invented categorization. When I made the mistake of
signing on with a few recruiters on a job hunt a while back, they looked my
recent job experience and labeled me a "systems programmer". This was
differentiated from things like web design and GUI. It is wrong wrong wrong.
We are software developers. It should not matter what layer of architecture we
are coding in. Every layer has so-called "business rules"; everything has MVC.
Label yourself, and you limit your thinking and your evolution.

This article does have a valuable lesson, in spite of the miscoloring. What we
see in this article is an example of overthinking the problem. With
experience, your debugging/problem-solving skills will get better, and this
kind of mistake will not be often repeated. It took me many years to get out
of the habit of speculating and to get closer to actually debugging. I had
learned that it was faster to look at what was really happening than to guess
at it. It helps even more to come to a detailed understanding of your system,
but if you do not have time for that... Never start at the bottom first: start
with the symptom.

~~~
rxin
For sure there is. I'd call the group of engineers "systems"
hacker/engineer/programmer if they are working on work on OS, database,
networking, and distributed infrastructure kernels.

These type of programming requires a distinct skill set and usually more
experiences than for example web frontend development (PHP, JSP, etc).

~~~
Locke1689
Anecdotally, I can recall exactly how to manoeuvre the TSS register in real
mode to successfully boot a VMM guest but I couldn't for the life of me recall
how to make a speckled background in CSS.

Asymptotically there's no such thing as a systems programmer but that's not
really how the real world works.

------
angusgr
The other critical factor is that "digging straight in" is _fun_, at least for
people with this mindset.

Reading documentation, on the other hand...

------
diN0bot
unfortunately it's not always easy to recognize when a programmer is doing the
right thing, because then things are running smoothly. it can seem like the
programmer isn't special because hey, it's easy to read the manual, etc,
whereas going deep to figure something, whoa, good thing we had that guy on
the team! same thing with simple and elegant versus complicated designs.

------
sedachv
Funny, my experience with systems programmers has been that the real dangerous
part is the unmaintainable and unportable code they write.

~~~
sedachv
Wait, I left out the punchline: _for projects that don't involve systems
programming_

You think I'm trolling or trying to be funny, but it's actually a real trend
I've noticed.

------
berntb
I've seen some advice for chemists formulated like:

"With a few weeks of hard work, creativity and dedication in the lab you can
often save _hours_ in the library."

