
The Myth of the Super Programmer - jsonmez
http://simpleprogrammer.com/2012/08/12/the-myth-of-the-super-programmer/
======
btilly
Nice thesis.

He's wrong.

The super-programmer that I'll hold up is Jeff Dean (see
<http://research.google.com/people/jeff/> for some of what he has done) and
the task that I'll hold up is coming up with new standards and abstractions
for creating scaleable software.

Sure, it is possible to break any one of the things he has done down into
simple components. As a random example look at any of the many blog posts that
discuss how MapReduce works and how to implement it yourself. (Heck, he had to
do so when he turned his ideas into running software.) But could you have come
up with those ways to think about problems? I am sure that I could not.

~~~
seanmcdirmid
MapReduce is a simple realization that what we were doing in functional
programming 20+ years before was useful in systems programming. Actually,
"simple" isn't "easy" here, and MapReduce great realization so they deserve
all the praise they get. But there was a lot of groundwork in place already.

~~~
btilly
A great number of advances in math and computer science can be recognized as
someone taking techniques developed in one area and recognizing how they can
apply to different area of programming. MapReduce is clearly one of these, and
they were quite clear where they pulled inspiration from.

However Jeff Dean was the source of a number of insights which were utterly
essential to Google's managing to scale. Only some of these have been widely
disseminated outside of Google.

Once lightning has struck enough times in one place, you can rule out
coincidence. It is clear that intellectual lightning keeps striking at a place
named Jeff Dean. And the fact that it does is part of what makes him such a
great programmer.

~~~
seanmcdirmid
I would give Sanjay some credit also...and probably many others. Google has a
great formula, but let's not get carried away; there are a lot of smart guys
in systems who are very good at what they do. But super programmers is just a
bit too condescending, I don't think any of them would appreciate that label.

~~~
btilly
Jeff Dean is famously modest, and undoubtably would resist the label "super
programmer" in the same way that Terry Tao resists the label "math prodigy".

That doesn't mean that the label does not fit.

And no, he doesn't work in a vacuum. On the other hand, I think that he
deserves the praise that I just heaped on him.

------
pmb
It's like math. You don't believe the difference between the best and the rest
until you meet the best. It is apropos that someone linked to a Terry Tao math
post in this discussion, because right now, in math, Terry Tao is definitely
in the "best" category. Unless we have chased the people like him away from
programming, they must certainly exist in our field as well. However, our
works are largely anonymous, which means that it is much harder to tell who is
the best.

But Rob Pike probably is up there.

~~~
tluyben2
Indeed; I thought I was very good at math until I landed in a course where
there was this guy who just instantly grasped everything and it's consequences
without having ever seen the subject we were taught. It was great
talking/working with him but it deterred me from pursuing math as major; I
reasoned that if there is someone so incredibly brilliant that he can replace
this entire auditorium by himself, what am I doing here? I was very young at
the time and I now know that brilliance by itself is not enough. He is not
working in math now but is a programmer somewhere; so him replacing the entire
auditorium for math didn't happen.

For super programmers (definitely there are plenty of them like Carmack) this
of course does not fly; one super programmer can maybe do the work of the
entire auditorium, however the rest of the auditorium is needed as well as
there is a shortage. Which is different from math, physics etc.

------
simonsarris
> But I do have a challenge for you. Surely you know a “super programmer.” You
> might even be one. If so, let’s hear from you or them. Tell us that complex
> problem that is too difficult for the rest of us to possibly understand.

"There are only two hard problems in Computer Science: cache invalidation and
naming things." -Phil Karlton

I'm a little hesitant but inclined to agree.

I hesitate because like most discussions on these matters, the terms are
defined differently for everyone I talk to and that makes agreeing a little
difficult without qualifying a lot of statements.

I'm not sure I've ever come across something that was simply too difficult to
understand, _per se_ , just things that take different amounts of time and
different amounts of diligence to understand or implement.[1]

A lot of content in programming (and math) is what I like to call "NP-
understanding complete". Problems that are difficult to find solutions for but
once shown a solution are very easy to comprehend. Coming up with novel
solutions to these problems in the first place is admirable, and I would be
inclined to call those who do "super", but I don't know if there are very many
"super" people who consistently do this. It may well be that finding novel
solutions is merely a product of spending a lot of time at the frontier of
something, and in fact its not only not a gift, but difficult to see them
coming (the solutions) until you have them!

[1] When looking at superhuman accomplishments or worrying that I might not
understand some subject (something I was completely frightened of as a child)
I'm always reminded of The Ladder of Saint Augustine by Longfellow.

    
    
        The mighty pyramids of stone
          That wedge-like cleave the desert airs,
        When nearer seen, and better known,
          Are but gigantic flights of stairs.
    
        The distant mountains, that uprear
          Their solid bastions to the skies,
        Are crossed by pathways, that appear
          As we to higher levels rise.
    

(The full text: <http://www.poetryfoundation.org/poem/173902>)

------
pcestrada
Sure you might be able to understand something given enough time and guidance.
But to me a super programmer is someone who is able to make that leap and
create something novel, useful, and valuable. For example, John Carmack, who
many would describe as a super programmer, showed the rest of the game
industry what could be done with 2.5D in Doom, and later commodity 3D hardware
with Quake. You can certainly review his code since it's open source and
understand it, but how many people could have independently done what he did
when he did it? Not many I would wager.

~~~
edwinnathaniel
How many people?

The answer is one: the guy who spent most of his time honing his skill, and
mastering his domain.

It's that simple.

No one can solve a problem unless he/she knows the problem inside out and has
the pieces to solve the problem (a.k.a previous experience: success or
failure).

You can't put someone who is not familiar with the problem domain and expect
him to do superb job given a deadline. If he can solve it under-pressure and
on-time, chances that he has previous experiences or building blocks that can
help him greatly.

The problem with most people is that they're focusing on the wrong thing
because of many reasons: distraction from family, distraction from the things
they read, tend to spend too much time viewing the problem from the same
perspective instead of taking some time off to think about it from a different
angle.

------
dsymonds
This is nonsense. His key argument is that hard tasks are reducible to simple
tasks, and therefore they are actually not all that hard.

Ask anyone who's implemented, say, Paxos, and ask them whether because the
individual steps were easy the whole system is therefore easy. Or indeed any
other distributed system. Just because something can eventually be decomposed
into simpler things does not mean that it is a fundamentally simple construct,
nor does it mean that a programmer who can manage the simpler constituents is
competent to handle the overall system.

------
crazygringo
The author blows his own argument out of the water when he says:

> I would venture to say that really good programmers are about 10 to 20 times
> as effective as just average ones.

Aren't those the super ones? Or at least on that track?

But disregarding that, I'd personally say the super programmers are the ones
capable of inventing git, or bitcoin. I don't care what you say, 99.9% of
programmers simply couldn't do that. Saying super programmers don't exist is
like saying Einstein's theory of general relativity is decomposable into easy
parts.

Sorry, I don't think so.

~~~
jsonmez
You do not really understand something unless you can explain it to your
grandmother.

Albert Einstein (1879 - 1955)

~~~
spacemanaki
Just because you understand something deeply and possess the pedagogical
skills to explain it to an elderly layperson doesn't mean it's suddenly easy
after you've explained it, and doesn't mean it was easy to put in the hard
work required get to that level of understanding. Breaking something
complicated down and explaining it to a layperson is a different thing than
actually _doing the work yourself_.

Are you suggesting that if one is able to explain a complicated problem to
their grandmother, somehow the original problem was actually easy, or is made
easy by this explanation? I haven't read Einstein's book on relativity linked
in your sibling comment, but I don't think the existence of this book makes
relativity an easy thing to have worked on.

------
noonespecial
"Superstars" seem different to me not because they can or can't decompose
problems into steps, but because they seem to have a 6th sense for what is
possible.

Everything is easier once someone has done it before. You don't even have to
know how it was done, just knowing its possible is enough. Superstars can
substitute some sort of magic pixie dust _(1)_ for this foreknowledge and
proceed like they already know its been done and they're just rediscovering
the minutia of making it happen.

Its humbling to me when I meet them because they think thoughts and have
insights that I know I would not have had even if I worked on the same problem
for 100 years.

 _(1) I strongly suspect that its made almost, but not quite, entirely of
crazy._

~~~
einhverfr
Here's what I think.

Sometimes I am in superstar mode and more often I am not. When I am in
superstar mode, I am able to latently process problems ahead of time and
decompose them. That way when I get to the problems they all seem simple.

The rest of the time I have to run up against a problem first, let it sit and
see if it decomposes for me, and if not come back and actively try to
decompose it. Once I am actually ready to sit down and start coding, all the
hard part is over. The question is how much of the hard part can be done
latently before I get to it.

------
TamDenholm
I think this kinda goes along with the four stages of competence, when you are
at a level of unconscious incompetence in any subject matter, it all seems
like black magic to you, you've no idea how it works and it seems like the
most difficult thing in the world, but just a little insight into how
something works can open your mind into understanding it enough to see that
like most things, it just takes time and effort to learn just about anything.

------
tytso
Taking complex things and making them simple is not the mark of a "super
programmer". Or, at least, it's not enough. A highly skilled programmer needs
to be able to deeply understand multiple layers of abstraction at the same
time. He or she needs to be able to know what an abstraction layer promises,
and what its performance characteristics are --- and also be able to grok the
layers above and below that abstraction layer to a similar deep level.

I call it "abstraction surfing", and it's not something that all people can do
well. I believe having this skill tends to correlate very well with high
productivity programmers --- and it's not a skill that all people have.

------
papaver
its not really about programming.

its either you are a superstar problem solver or a simple problem solver. the
programming is just a means to an end. a superstar problem solver has the
ability to take most problems and break them down and find the means to solve
any problem, weather related to programming or math or most anything else.

superstar problem solvers can jump into most fields of software and learn how
to cope in their new environments. every job i've had has forced me to learn a
different language and work on a completely different problem. the problems
may change, but how one attacks and breaks them down them rarely does.

------
Arelius
> But I do have a challenge for you. Surely you know a “super programmer.” You
> might even be one. If so, let’s hear from you or them. Tell us that complex
> problem that is too difficult for the rest of us to possibly understand.

I generally tend to agree, and in the majority of computation I think this is
generally true. Compilers for instance I think are one thing that the majority
of programmers imagine to be simply above them. While in reality they are much
like the rest of the software we write.

However as a counter example, the hardest things I've ever written included a
rather complicated lock-free data structure, specifically optimized for 3D
effects rendering and simulation. The amount of mental capacity that took was
pretty astounding, and what I took from the experience is that I'd try to
never have to do something of such complexity again. I am also, however unsure
if I could complete a similar task again. It did give me an appreciation that
some things are just simply harder.

------
spacemanaki
I think this post has good intentions. There's some truth to it and there's
definitely value in plumping yourself up from time to time. No one should be
worrying about whether they're a "Real Programmer" or not. I agree that
there's no value in beating yourself up or beating up your work.

That said, I think the post takes it too far, and veers off into self-
congratulatory back-patting that I think is _not_ valuable, at all.

I also think the author is wrong about hard problems, or at least managed to
come up with a tautology that's not very interesting. Sure, many (and perhaps
all) hard problems can be decomposed into smaller, simpler problems. But
that's often not trivial, and the decomposition step often requires a lot of
hard work. Even if hard problems can be broken down into smaller pieces, that
doesn't always make the problems easier, even if it makes them more
manageable. They still might require tremendous effort.

The example given isn't a very good one; that sum just isn't that advanced
mathematics. Now that's not to say it's not Greek to some programmers, and not
to pooh-pooh people who have a hard time with math; I have struggled with it a
lot. But there's a lot of math that really is hard to grasp, and isn't just a
question of learning the notation, which is what the sum example is really
about.

The challenge sets up a straw man, IMHO. "Tell us that complex problem that is
too difficult for the rest of us to possibly understand." It's asking the
reader to be an arrogant ass.

"I’ve just never seen a problem that couldn’t be broken down into simple to
understand components, yet."

Again, I think this just isn't a useful criteria. Broken down into simple to
understand components doesn't make things _easy_. Here's an example, from my
own dabblings: I am absolutely fascinated by programming language theory and
compilers. Recently, I've been trying to get at the heart of two sub topics:
parsers and types. I've been struggling to find presentations of parsers and
parsing that really peels back the magic and shows how everything works,
without just punting and letting Lex and Yacc do the dirty work (btw if you
know of a compiler text that doesn't do this, let me know!) Yes, you can break
parsing into a lexing step built up from regular expressions. Yes, you can
translate regular expressions into NFAs and convert those into DFAs... and so
on. But putting this together into a lexer generator is non-trivial. It's not
_easy_ , even if the components are _simple_. It's the same with type theory.
You can write simple syntax directed type checkers and they might fit in your
hand, but if you really want to understand type theory, it turns out there's a
lot of math, and proofs to get through. Sure, you can decompose that into
topics from set theory and so on, but this doesn't make it _easier_.

More and more, I think his core argument is just really flawed. There's some
connection here with Rich Hickey's "Simple made easy" talk, where he argues
that simplicity can be hard to come by, and how _easy_ and _simple_ are
different... but I've written too much and will leave that as an exercise :)

~~~
btilly
_I've been struggling to find presentations of parsers and parsing that really
peels back the magic and shows how everything works, without just punting and
letting Lex and Yacc do the dirty work (btw if you know of a compiler text
that doesn't do this, let me know!)_

Here you go: <http://www.cs.rochester.edu/~scott/pragmatics/>

(Expect to spend several months reading it.)

~~~
aaronblohowiak
<http://www.bayfronttechnologies.com/mc_tutorial.html>

~~~
btilly
That explains how to write a top-down (or recursive-descent) parser. Humans
generally find it easier to understand and write top-down parsers.

Tools like YACC exist to make it easier for humans to specify bottom-up (or
shift-reduce) parsers. They have a number of benefits over top-down parsers
including performance.

If you want to understand the magic of YACC, without the whole "it's magic"
bit, you need to understand how to write shift-reduce parsers.

~~~
seanmcdirmid
Recursive descent is nice if you want to finesse your error recovery, which is
much more important these days than raw parsing performance. Times are a
changing...

------
o1iver
A lot of people have commented that they do not agree because most people
could not come up with Git, invent (from other ideas) modern-day MapReduce,
etc.

I don't agree that the article is wrong for those reasons. Those people who
invent those things are able to do so because they have spent years learning
the underlying principles. And this does not only apply to programming; it
applies to any scientific field. Somebody mentioned Terry Tao; well I would
say that Terry Tao is a "superstar" because he has combined talent and more
importantly because he has studied maths for years.

Talent + (a lot of) practice = superstar!

Look at the story of Andrew Wiles's proof of Fermats last theorem. He is
undoubtedly a talented person, but was able to complete that work because he
had studied math for a long time and combined the insights of many fields into
his proof. Practice and talent makes a superstar!

------
kenster07
This discussion enters into the realm of "myth" so much more often than fact,
because in these discussion, no one even tries to define what "average" is. Is
a student studying Java in high school for the first time a programmer? By
average, do you really mean the level of programming skill that you meet with
the highest frequency or the actual mathematical average? How do you even
measure "productivity" or "effectiveness?" I bet one can define parameters
where the best programmers are only 3x as "productive" as the average
programmer.

And I'm not saying such things can't be measured. I'm just saying that
discussions about these myths are often devoid of such measurements.

In my view, discussions like this are much more about the human tendency to
deify than an objective search for truth.

------
substack
Actually that code snippet that computes a summation can be done without any
loop or recursion at all:

    
    
        function sum (m, n) {
            return m * (m + 1) / 2 - n * (n - 1) / 2
        }
    

I only know this because I remember from math competitions in high school
learning that you can compute the sum of the digits from 1 to n quickly from
`n * (n + 1) / 2` which naturally follows from the equations regarding
binomial sums.

This example somewhat agrees with the author's main point that we are more a
product of our experiences than endowed with certain innate capabilities.

------
mathgladiator
I think this could be tied to Terry Tao's recent piece:

[http://terrytao.wordpress.com/career-advice/does-one-have-
to...](http://terrytao.wordpress.com/career-advice/does-one-have-to-be-a-
genius-to-do-maths/)

<http://news.ycombinator.com/item?id=4370338>

\- - -

The problem that I have with "programmer" is that there are a lot of clueless
people that call themselves "programmer", and I think there is a clear
boundary. Given the population and size of clueless programmers, I'd define
the non-clueless programmers as super any day.

------
dman
The super programmer does not necessarily solve a different category of
problems. She sees maintanable and performant abstractions when she encounters
new problems. She sees order where others see chaos.

------
cletus
Here's my view on this:

1\. There is a law of diminishing returns when it comes to effort spent vs
reward. I think we can take that as _axiomatic_ ;

2\. Different skills are naturally harder or easier for different people. Call
this _affinity_ ;

3\. Given the effort vs reward curve and the different affinity levels
different people have you will find that with any given skill people will tend
to plateau at a particular level.

This isn't simply a question of intelligence or innate ability. More than
anything it comes down to motivation. Priorities vary. Perhaps someone is
particularly keen on their social lives, families, hobbies or whatever and
whatever skill from which they derive their livelihood just needs to be
sufficiently good for them to get by.

For example, I was an OK chess player. Not great. Not even particularly good
but I did reach a point where I recognized that if I wanted to get better it
was going to be a serious amount of effort. Nowadays I tend to play Euro
boardgames simply because they have a nice effort vs reward curve and aren't
(potentially) demanding mistresses like, say, Chess, Go, etc;

4\. The more time you spend in any particular field the more you will master
it and begin to see problems with it; and

5\. Certain environments will tend to make you more or less likely to succeed.
For example, if you want to get better at pretty much anything, hang around
with really good people who are really good at it.

Many people don't get that far because they either plateau or the effort vs
reward curve gets too flat and they move onto other things.

So what I tend to think of as "super programmers" combine the traits of:

\- some level of natural affinity;

\- the dedication to stick with something; and

\- they care enough not to simply to just enough to get by.

The last I think is particularly key. When I assess an engineer one of the
things I'm looking for more than anything is that they care about their craft.
They can assess what they see and what they do critically and recognize where
it falls short, what works and what can be done better.

It's part of the reason I'm a strict pragmatist and prefer other engineers who
are like-minded. Ideologues (IMHO) tend to have judgement clouded by dogma and
preconceived notions.

Woody Allen famously said "80% of success is just showing up". If you care
about your craft and keep plugging away at it good things will happen. Those
programmers we tend to think of as "super" or "rock stars" I think tend to
combine all these traits with the timing and luck of having that flash of
insight at the right time _and following through with it_.

Like they say, good ideas are cheap. It's execution that matters.

~~~
edwinnathaniel
Very thorough explanation that I definitely agree.

I love watching Soccer and Basketball. In those two fields, the best players
are those who dedicate themselves to the game.

Lionel Messi, Christiano Ronaldo, Michael Jordan, LeBron, Kobe. The list goes
on. They have the same trait: passion and dedication. Some may have born with
the skill (Ronaldo, LeBron). Some takes time to develop (Jordan, Messi).

Having said that, our field is a little bit strange because there's a divide
between being good at a particular set of skills and being jacks of all
trades.

For example: we tend to argue and discuss the topic of how learning new
programming language can change the way you think about a particular problem
and hence make a conclusion that those who put the effort and time to learn
niche language such as Erlang, LISP dialect, Ruby, Lua get this automatic
recognition as a "better programmer". (They may be a "better" programmer if we
compare it to John Doe Programmer, but are they truly a better programmer?)

On the other side of the coin, we have people like James Gosling, Bjarne
Stroustrup, John Carmack, Dennis Ritchie, Joshua Bloch, Guido von Rossum, Kent
Beck, Martin Fowler, etc. that are pretty much sticking with one or two
languages but truly focus on their problem domain (as oppose to the languages
that they use).

There seem to be a group of people who are good at plumbing different
technology and get something out of it and there are another set of group of
people who specialized in specific technology but they're just... awesome at
it.

Example: A typical blog article explaining the full-stack of technology => "We
use Rails as front-end, MongoDB as our back-end, memcached for our caching
layer, ActiveMQ for our queueing system" vs "Here's the architecture of
Mailinator: Java and Tomcat"
([http://paultyma.blogspot.ca/2006/12/architecture-of-
mailinat...](http://paultyma.blogspot.ca/2006/12/architecture-of-
mailinator.html))

PS: Kent Beck and Martin Fowler may not have the "proper" HN street cred but
they are definitely put their sweat, effort, and time to get better at what
they do (OOP, software design/development, software process).

~~~
excession
Out of curiosity why would Kent Beck and Martin Fowler have "proper" HN street
cred? I would have thought it safe to say that they had both have had
revolutionary insight into their areas of focus.

~~~
edwinnathaniel
Some people would argue that they are a "consultant" type like Ron Jeffries
and the whole sudoku Norvig debacle.

Plus OOP is taking some beating lately.

------
j45
One quality I find with above-average programmers is they type so much faster
than the average programmers.

100-140 WPM on average.

What's the importance? They code quicker, hit walls quicker, debug quicker,
fix quicker, ship quicker. They also have to be able to think quicker once the
steps ahead of them are clear.

Getting through this cycle quicker I think makes a programmer much more
effective at building lots more software, and through it, finding the lessons
from the experiences a lot quicker.

~~~
T-hawk
Input speed is totally a major part of being a productive programmer. Not
necessarily raw words-per-minute typing (it helps, but how much of programming
is words, compared to symbols?) But navigation speed around your machine and
environment makes a huge difference. Things like cutting a block of code and
jumping down two screens to paste it. Or memorizing all the keyboard shortcuts
for your debugger, so your mind stays on the problem rather than nosing around
toolbar buttons. Or operations as mundane as fixing the indenting of a block
of code, that a programmer could easily do a hundred times in a day. Save five
seconds on that and it seriously adds up.

I naturally zip around (Windows) environments very fast. I can typically
compile and alt-tab over to the output directory and copy and paste to a
deployment folder in under two seconds, often before the programmer watching
next to me registers that I've done it. This seriously matters! Like you say,
programmers with this practiced skill set can code quicker and debug quicker
and ship quicker. (There are downsides; I can't stand pairing with a slower
typist, drives me crazy.) Laptops are right out for serious productivity for
me, for this reason. My hands know all the (Windows) keyboard shortcuts in the
Ins-Del-Home-End block of keys, so my navigation speed on a laptop keyboard
without that usual layout drops to half or worse.

Actions per minute as a metric can go beyond Starcraft. It matters for
programming too.

I dearly wish to see the day when neural human-computer interfaces become
readily available and accurate. Imagine just _thinking_ 'click' at a toolbar
button and there it goes. Not just for typing and programming either, but how
about games? Imagine playing Starcraft at the speed of pure thought, with no
APM constraints to get in the way...

~~~
j45
I really like the idea of 'actions per minute'. It's a really nice way of
summing up what I was describing.

------
slurgfest
Any sufficiently advanced craft is, from the perspective of people who do not
have that craft, indistinguishable from magic.

Doesn't mean it is not actually ordinary and accessible, though...

I question the premise that there is some qualitative difference between the
"super" people and the rest of us dunderheads.

Rather, there is a hard line between what you understand and feel confident to
reproduce, and what seems magical to you.

------
kenmazy
From my experience with super programmers:

The super programmer converts complex ideas into code coherent to the normal
programmer.

The super programmer makes difficult decisions where the normal programmer
sees only one option.

------
eranki
But simplification is the hard part... Being able to understand a solution and
being able to come up with a solution are two very different things.

