
Designing a study to see if “the 10x programmer” is a real thing - chrismealy
http://andrewgelman.com/2014/12/04/designing-study-see-10x-programmer-real-thing
======
ghshephard
Depending on the field, it could be as much as 40x. We had a firmware
engineer, who in 3 months, single handedly wrote an entire FHSS MAC layer for
an RF application, with a high level of quality and hundreds of functions,
while simultaneously laying down the architecture for it's growth over the
next seven years. He was extraordinarily capable, intelligent, and had worked
on the identical project in the previous 5-6 years, and so was able to
anticipate all the blind alleys to avoid.

I'm reasonably confident that a low-performing engineer with little or no
experience in this space would have taken seven-ten years to accomplish the
same feat at the same level of quality and completeness. (Of course, it could
be argued that our engineer _had_ taken seven years to accomplish this task,
it's just that they had worked six of them at a prior company)

The _challenge_ in these types of tests, of course is (A) who defines/measures
quality, (B) How do you know which functions were essential? Programmer
performance is not a function of how fast you can complete coding challenges,
but how well you can architect, document, communicate, collaborate, and all
the other peripheral functions not associated with cranking out algorithms.

~~~
svisser
How does that qualify as 40x? It's not reasonable to compare an engineer who
has worked for 5-6 years on an identical project to an engineer that hasn't or
has less work experience.

For this particular task he may have been 40x more effective than anyone else
but only because he walked the path before.

~~~
geocar
Do you pay programmers with 5-6 years experience in a domain 40x junior
programmers?

~~~
alkonaut
Productivity as an inherent property of a developer has to be for "unseen"
tasks. If someone started working with me today I'd be more than 10x as
productive as them tomorrow and maybe all of next week. That is relevant for a
company, but not really interesting.

~~~
geocar
That only means instantaneous productivity isn't useful. Fortunately most
employers know this: The employer pays for a mean-productivity-over-time.

If your productivity falls below that value you will get fired, and if it
exceeds some other bound you may be bonus'd or promoted.

However _if_ the productivity exceeds this line by 40x, then the question
remains: Do you pay your programmers 40x?

------
joelthelion
A productivity factor is not a good way to model programming ability.

I consider myself a decent programmer, but I'm not that much more productive
in terms of features/week than other less skilled programmers.

The real difference is that I can do things that poor programmers simply
cannot do: architect a system correctly, optimize performance, debug a
difficult memory leak, write correct code, etc.

~~~
objectobject
This. Measuring productivity is useless if you are producing the wrong thing.
Even ignoring all the things programmers do other than writing code, this idea
still completely ignores the many external factors that interfere with
productivity.

Organisations are more than the sum of their parts. It's not like you can take
a bunch of 10x programmers from a successful project, put them on a failing
project and expect them to just get it done. Why is 10x programmers a thing
but not 10x management?

------
vorg
Programming shops are full of both 3x and -0.5x programmers. Many 3x
programmers end up looking after a group of say four -0.5x programmers, and so
after cleaning up after them they themselves come out looking like mere mortal
1x programmers (because 3 + 4 x -0.5 == 1 ).

------
sklogic
I suspect that all this 10x buzz is coming from the overstretched, sleep-
deprived people who accidentally managed to come to work once or twice well-
rested and realised: "omg! I'm 10x compared to my usual self!".

~~~
ZoFreX
I don't. I recently witnessed someone writing a 10 line function to test for a
particular condition in our codebase. I could have done it in less than one-
line, by re-using pre-existing code.

What took them 10 minutes would have taken me 1. My solution is also more in-
line with how it's done over the rest of the codebase, if you applied my
approach 100 times you would have a consistent and easy to maintain codebase,
apply their approach 100 times and it would be inconsistent and harder (more
time consuming, more expensive, more risky) to change.

That's where the 10x comes from.

Edit: Just in case they're reading this... I don't think this is because I'm a
better engineer or you're a worse one, I blame our lack of documentation and
training for this feature!

~~~
sklogic
That's an anecdote, not an evidence. Same as my own revelations after trying
to work while being unusually well-rested.

The next day the same person may do something 10x times faster than you,
because of the differences in your past experiences, problem solving
methodology, etc.

What really matters is amortised performance, and there is no evidence that
there exist a significant deviation in such among professionals of comparable
experience profiles.

Of course, someone who already did something ten times will then do the same
or a similar thing in no time, while a completely new problem will be faced
with blank eyes by anybody otherwise smart and experienced. It's such an
obvious thing that it cannot be a source of a myth on its own.

------
qwerta
How about making comparative test where one subject has suboptimal conditions?

\- no objective given, or change objective in middle of test

\- no IDE allowed, only notepad and outdated compiler

\- place next to garbage bin or turbofan engine

\- randomly generate some 'legacy code'. Result must contain 90% of this
code...

\- no internet connectivity allowed

\- no external libraries allowed...

\- code must be deployed within 1 week, it must past code-review before
deployment, reviewer is on holiday....

\- ask subject some trivial question every 10 minutes. Or thrown in 5 hours
meetings randomly.

I am sure the productivity will vary... :-)

~~~
sklogic
You forgot to add a Swordfish-like hacking scene to this list.

~~~
touristtam
Without mentioning the off work life challenges. I mean those have possibly an
impact that often see dismissed in those stories about 10x engineers.

------
thorin
Not sure why x10 is such a surprise. It [software] is a good field with decent
money and a low barrier for entry at a lot of places. Not everyone can be
Dennis Ritchie.

If songwriting had lots of well paid jobs open to virtually anyone with a
hobby interest or non-related degree do you think everyone would produce like
Lennon/McCartney?

You could give me a million years and I wouldn't write the Linux Kernel but I
can still produce useful software which improves someone's productivity.

~~~
jiggy2011
You couldn't write the 2014 Linux kernel because it is the product of
thousands of people and millions of dollars , you could probably write the
1991 Linux kernel given time and motivation. Linus did a lot of things wrong
in the early days.

------
ExpiredLink
Why does the "10x programmer" myth live on? Because all supporters assume that
they are one of the '10x' \- or at least aspiring to be one of those. It's a
self-energizing myth.

~~~
cpks
Because, for complex software, it's not a myth.

~~~
jbergens
I think it is partly a myth. It can probably only work for small
parts/systems. If a large system has to be written and it takes 3 good
developers 2 years to do it it will probably still won't take 3 "bad"
developers 20 years to do it. They will learn things when they work. If I can
write a few (like 5-10) classes to do something and it takes me an hour that
still don't mean someone can write it in 6 minutes.

~~~
geocar
I've always understood the origin of the "10x programmer" was about overall
performance, and not mere "time-to-type". That is: we're talking about the
reported value of the employee to the company.

In that way, the best performer is as much as 10x the worst; but the best
performer is probably only 2-3x the majority.

------
alexandros
A developer can have negative performance, either by wasting time on the wrong
thing, or by producing a solution that is flawed and needs to be rewritten (if
the project survives). On the upside, a developer can invent a whole new way
of doing things, and as such acheive things that an arbitrary amount of other
developers wouldn't be able to. As such, 10x may be a useful metaphor, but in
real terms is a gross understatement. It may make more sense in traditional
enterprise settings where most things are determined from above, but in
startup settings, a good developer can make or break the company.

------
lhnz
Do you want to test the productivity of programmers with FizzBuzz or writing
an OS from scratch?

The distribution of productivity amongst engineers is probably going to vary
depending on the complexity of the task.

Some tasks I would expect the top performers to be far better than 10x at.

And not all top performers are equally suited to each task.

Additionally this idea of measuring productivity purely by problem solving
ability is nonsense as engineers are hired also based on their ability to
write maintainable code and to work as a team, etc.

------
cpks
It's hard to study. The initial studies were coding time, but the productivity
difference comes down to a few things:

* Good architecture choices. These make a big difference in total development time.

* Communications. A team of 50 1x programmers has much more overhead than a team 5 10x programmers.

* What's possible. There are some things some people just can't do, even given infinite time.

* Process overhead. Bad people require much more process than good people.

Overall, the difference in productivity between small, good team of expensive
people and large, bad teams of mediocre people is much more than 10x for
complex systems. It's very small for large, simple systems (e.g. "business"
systems which handle inventory, HR, etc., which just have a very large number
of relatively independent simple bits).

That explains super-high-profit companies like Valve.

Once organizations get into the mediocre person game, it becomes almost
impossible to go back. Good people won't want to work there. If they do, the
amount of process will slow them down. You won't leverage them well -- you
still have the communications overhead. They'll always be told what won't
work. Etc.

------
jiggy2011
What task could these programmers perform that would not be biased towards
programmers who have recently solved similar tasks?

~~~
spacecowboy_lon
depth of exprinace is a factor eg yesterday I wrote a feature extraction tool
in 2/3 of a day (including geting up to speed with python) to strip out the
usable body copy for a large uk website.

I could do this so quickly becase I had 3 years previously spent several days
experimenting with new scientist's archive and allready knew what library
would work.

------
alkonaut
Like so many before, this post seems to avoid the fact that while there is
talk about a "10x variation" in productivity, that's not really the
controversial claim, if by variation we mean "range" or "span". To find a 10x
span just look for a 1/10 dev and you are done. _There is no lower bound in
productivity_ so the ratio of productivity between least and most productive
is infinite since there are likely some that produce zero value (I'm pretty
sure I've met those that deliver negative value).

So. Before one designs a test to find a unicorn one should define unicorn. Is
it a certain _variance_ in the sample? A variance such that at least one
sample point is 10x more productive than the mean, that is at least well
defined, but how large would the sample size have to be if you are looking for
a single unicorn?

~~~
SiVal
I think this is a good point. After all the claims about how the _majority_ of
self-described programmers can't write FizzBuzz, are those of us who _can_
write FizzBuzz already 10x programmers before we even get started on anything
harder?

------
geocar
This made me think about how Arthur's been passing around some cs107
challenges that he found online.

Here's one of them:
[http://web.stanford.edu/class/cs107/assign2.html](http://web.stanford.edu/class/cs107/assign2.html)

The advice page suggests the student hours spent on this project is:

    
    
        * < 5 hours 5%
        * 5-10 hours 14%
        * 10-15 hours 19%
        * 15-20 hours 38%
        * 20-25 hours 14%
        * more than 25 9%
    

Pierre did a solution in 1hr 30m; I was just under an hour.

I wonder how people do here?

------
qq66
Whether "10x" exists depends entirely on the problem domain and the way you
define productivity.

If the problem domain is building an HTML login page, and your measure of
productivity is time to complete, then a 10x programmer probably doesn't
exist. A whiz maybe writes it in 5 minutes and the average Web dude takes 20.
Typing speed may be the limiting factor.

If the problem domain is building a better way to search the Web, and the way
you define productivity is in long-term revenues generated, then Larry Page
and Sergey Brin are billion-x programmers for building Backrub/Google.

------
auganov
The point is some people will provide an order of magnitude more business
value in the long run. Most of that value will be outside of "implementation
skill". The 10x programmer isn't really just a programmer. The more decision
making power one has the higher the potential multiplier. Trying to find "the
10x programmer" in a large enterprise or a consulting company is like looking
for a lion in Sweden. Though even with simple tasks 5x is probably not that
rare.

------
ExpiredLink
What do they want to achieve? You certainly can find one woodchopper who is 10
times as productive as one other woodchopper. But what does this say about
woodchoppers?

~~~
DanBC
Once you find 10x woodchoppers exist you study the differences between them
and 1x woodchoppers.

Do they use different swing of the ax? Do they chop different wood? Are their
axes sharper or heavier or have longer handles?

You then take those things and give them to the 1x woodchoppers and see if it
makes any difference.

~~~
shurcooL
The reason you might see a 10x woodchopper is that you measured their
productivity for an hour, and the other guy was on his lunch break for 45
minutes.

Similarly, engineer productivity may be one level one week and a different
level another week.

What if you could measure lifetime productivity? Or over 10 year period.
That'd be more insightful.

------
vellum
_Peopleware_ was one of the sources for the "10x" concept. However, the book
only said the best:worst ratio was 10:1.

As for the rest:

 _Count on the best performer being about 2.5 times better than the median
performer._

 _Count on the half better-than-median performers outdoing the other half by
more than 2:1._

[http://javatroopers.com/Peopleware.html](http://javatroopers.com/Peopleware.html)

~~~
throwawayaway
mythical man month was the one wot done it

------
pawelkomarnicki
The "10x programmer" myth is a fallacy that boosts the rat-race mentality in
programmers, believing you need to be able to solve problems with quick hacks
in order to be considered "a good progreammer". But the problem is:
productivity is not a constant. It's more like a tide, when it comes, it comes
strong, but then it fades and you can't do anything about that. Unless you're
a robot. Also, engineering is that fancy field, that often gets overly excited
about short-term wins, loosing the track of the bigger picture — one
programmer can do something in 15 minutes, using lots of poorly designed
hacks, without any optimisation or deeper thought, which will come back to
bite very soon; and another programmer might take few hours, making a solution
that will be rock-solid even after some time. The short-term "productivity" is
in favour of the 1st one, but in reality the 2nd one is more productive in the
long run.

------
yetihehe
Speed will also depend on length of the task. If you use 1 hour long simple
task, you will have less variance. Significant results could show only with
1-week task or longer.

~~~
jbergens
See my previous answer. I don't think it works for longer tasks either since
developers learn while coding. And if it takes 10 months to build a system it
still might be impossible to do it in 1 months even if you're skilled.

------
verydapeng
from my own experience ... there are negative-value-created developers ... 1/1
is 1x, 1/0.1 is 10x, 1/0.01 is 100x ... 10/0 is infinity, then 10/-1 ??? not
very well defined

