
Visualizing How Developers Rate Their Own Programming Skills - q-_-p
http://minimaxir.com/2016/07/stack-overflow/
======
lacampbell
I have a degree in computer science, but at the end of the day it's just a
piece of paper - I learned programming mainly through the internet. I talked
to people who cared enough about programming enough to hang out on IRC, or
write technical blog posts, or write their own libraries, etc.

What has shocked me since entering the "real world" is how little most other
programmers I meet care about programming, their craft, code quality, or
learning new things. I was legitimately shocked at how good I was compared to
the real world population, as opposed to the "internet programmer" population
where I was mediocre at best.

This isn't a value judgment - not everyone is passionate about their day job.
But it might resonate with someone out there.

~~~
thegeomaster
One time, when leaving a company, I helped interview candidates for my old
job, since I could best tell what skills are needed and to what extent.

I reckoned at the time, from most interactions on the Internet, that I am
(globally) a mediocre, if not below-average, developer - in each discussion I
participated online, there were always people above my level, from the ones
having a slightly better understanding of the subject to the people whose
depth of knowledge appeared somewhat otherworldly.

This belief was particularly strengthened by the fact that I didn't have much
contact with real-world developers in real life, as my friends tended to come
from different backgrounds.

When the interviews came, I was astonished how many of them couldn't code
their way out of a paper bag. The ones who did, wrote code with messed up
indentation, inconsistent style, ugly and hackish approaches, and the like. I
couldn't imagine how someone could leave their code like that and say "there,
this is my finished product." But here they were, and it seemed like they
didn't really see anything particularly wrong, or just not important enough to
justify the added effort of cleaning the mess up. It was a very strange moment
for me.

I believe this to be an example of the classical selection bias: people
inclined to participate in online discussions, answer questions on
StackOverflow, write blogs and type out detailed technical comments on HN are
on average a lot more skilled than the average just-earning-my-living
developer fresh out of some half-assed "be a software engineer and make tons
of money!" three-month course.

~~~
leesalminen
Indentation! Not much in this world upsets me as much as inconsistent
indentation. Ruins my day any time I have to actually read 500 lines of it.

~~~
mahyarm
You should automate that issue and have an autoformatter on every commit.
Everything else is a waste of time

~~~
aninhumer
Honestly, I feel like we should have gone further and decoupled display and
serialisation formats of code a long time ago. (Although I understand why we
haven't.)

------
geebee
This is interesting. I do think there can be a problem around the concept of
"competent" vs "average". For instance, consider a poll like this: On a scale
of 1-10, where 1 is inexperienced, 2 is competent, 3 is above average, and
4-10 are various levels of good-brilliant, where do you rank yourself?

My guess is that even with these explicit instructions, people are reluctant
to rate themselves a 2. I can see a few reasons why higher scores would
actually indicate that people see themselves as average. For instance, some
people might view anything under 5 is incompetent. In this case, a 7.5 would
indicate the middle of the range, which is actually 5-10. Alternatively,
people might think of grades in US based schools, where 75% is a C,
technically "satisfactory", but essentially failing in an era of grade
inflation. In many PhD programs, an A- (91%), is a shot across the bow, and a
B+, (88%), is meant as a vote of low confidence. This could lead many people
to rate themselves as an 8 or 9 simply to indicate their averageness.

Lastly, programming is still a young field with a wide variance of talent, but
I think we need to be more of a "paper hat" distribution than a "pyramid". To
explain, I don't think we're a field that works well with a large, unskilled
base, that slowly narrows to a tiny elite at the top. I think the baseline to
be a competent developer is quite high. I know, I read about these "CRUD"
apps, and what can I say? The framework churn, the challenge in communicating
with clients who understand business but aren't familiar with the unforgiving
logic of a program, the need to deal with deadline pressures under
uncertainty, the challenges of data integrity and testing coverage...
honestly, simply reliably deploying a relatively simple web app is
considerable, in my opinion. This is why I go for the "paper hat" distribution
- devs are largely on a long, flat curve, with a small little mini-pyramid in
the middle, where the true innovators live. In short, maybe we are a field
represented by wide, short rectangle of 1's-3's, with a minuscule steep and
thin little triangle of 4-10s popping up at the top. However, the 1's really
are smart people who are learning, and 2-3 represents quite a valuable and
talented developer.

~~~
Bartweiss
This seems like a really significant issue. First, because "5 as median" isn't
a universal grading scheme, and 50% is often failing. Second, because this
poll didn't specify what the numbers mean.

The bounds of "programming ability" are basically undefined. Does 1, being the
lowest, mean "cannot program"? Does 10, pain-scale style, mean "best
programmer imaginable"?

Or do the numbers represent deciles of skill? If so, do they include everyone
who has written code? Only people who have completed a substantive project?
Only people who have been "programmers" in a non-hobby setting? Only in a non-
hobby, non-academic setting? Only people taking this quiz?

And worse, it's a meta question. We aren't guessing what the _survey_ meant
here, we're guessing what the _respondents thought it meant_ \- which means
that the skew may be totally accurate under their chosen boundaries! Keynesian
beauty contest and all that.

So this is an interesting result, but I can't really use it as data about
self-reported skill, only about interpretation of surveys.

------
ivanhoe
My own perceived rating significantly lowered with the rise of social
networks. As you start to follow more closely some really great programmers
out there, you stop comparing yourself with your peers and start comparing
against the best in the game... and that is self-confidence killer :)

~~~
collyw
Move jobs a few times, inherit some crappy application and look at the code
that others write, that has done the opposite for me.

Though a lot depends on what you define as being a good programmer. I used to
be a lot better at clever algorithmic stuff. Now my focus is on clean
maintainable code and getting the higher level architecture right, so that I
don't need clever algorithmic stuff.

~~~
edblarney
"I used to be a lot better at clever algorithmic stuff. Now my focus is on
clean maintainable code and getting the higher level architecture right, so
that I don't need clever algorithmic stuff."

This.

'Clever' is BS.

There's way too much 'clever' going on in high tech, and way too many people
interviewing for the wrong skills.

I used to be very clever as well, today, I don't think I could get a job at
Google (not that I care) - but I think those interview styles are upside down.

These days I look at Eng. more like construction or plumbing: most of it is
not rocket science, really. You don't want new or fancy anything unless you
have to.

Good code should be _boring_ and read like English. There should be nothing
special about it, unless the problem space really calls for it.

The paradox is - simple code is not psychologically impressive, at least to
some people.

I'm actually trying to develop an interview method that will allow me to
measure this.

Clever is expensive, dangerous, and usually not worth it.

In fact - if you're trying to do something clever, it might be a sign that
something is wrong. There is probably a library for that.

One caveat: it's good to have done a bunch of clever but not-so-useful things
in the past, because when I use a library for something, I know what's going
on - that's worth something.

Sometimes I feel that first 2 years in dev is basically 'training' \- and I
wonder if someone with < 2 years experience should even be writing production
code.

~~~
lacampbell
I must admit I am wary of this attitude. I've worked at places where lambdas
are clever, or generics are clever, or writing constructors that take
arguments is clever, using LINQ on collections is clever, refactoring copy-
pasted code into methods is super clever, etc etc.

One mans clever is another mans blindingly obvious. Should we always work to a
lowest common denominator?

~~~
rhizome
Does the choice have to be about the lowEST common denominator? Maybe a good
criteria is to code for a lowER common denominator, just to account for the
Bus Factor, maintainability, and so on. The lowest common denominator _that
makes sense_ , rather than playing denominator golf or disregarding the fact
that you're coding for the benefit of your colleagues, too.

~~~
lacampbell
I disagree. Rather than coding for the benefit of someone who doesn't know the
language very well - people should learn the language. If the company is
hiring maintenance programmers who don't know what lambdas are - again, their
problem.

I feel like we should have higher standards, and use the abstractions
available to us grow the code towards the problem. Instead I see people
writing long sprawling application that use the same low level library
routines everywhere.

Maybe it's a quantity vs quality thing. And quantity is much harder to hire
for, so my ideas lose out.

------
matt4077

        I normally dislike working with survey data since there 
        is a high possibility of selection bias among 
        the respondents.[..] For this reason, I will 
        show confidence intervals whenever possible to
        reflect the proportionate uncertainty for 
        groupings with insufficient data [..]
    

That... is not how statistics work(?). I mean – confidence intervals help with
small sample sizes, but they do nothing for systematic errors such as those
introduced by selection bias.

    
    
        [continued from above] and to also account for 
        possibility that a minority of respondents may 
        be dishonest and nudge their programming ability 
        a few points higher than the truth.
    

There's a surprising amount of assumptions that went into this sentence. I'd
question the assertions that:

\- people are "dishonest" (My intuition would point to a subconscious bias
more than actual dishonesty)

\- It's a minority (The second chart shows that >50% of respondents with one
year of experience or less rate themselves as better than average).

\- That subconscious biases only work in one direction

... and once again I have no idea how confidence intervals can help. A large
interval may indicate bad measurement. It may also indicate high variability
in the actual data.

    
    
        Also, keep in mind that these groupings alone 
        do not imply a causal relationship between 
        the two variables.
    

... someone paid attention in his middle-school statistics club...

    
    
        Employing traditional regression analysis to 
        build a model for predicting programming ability
        would be tricky: does having more experience 
        cause programming skill to improve, or does having
        strong innate technical skill cause developers to
        remain in the industry and grow?
    

... but failed statistics 101. A regression analysis doesn't care about
causality. If "Mac users are more likely to be college-educated" it doesn't
matter that "buying a Mac may not actually make you smarter". I can still make
the prediction "a given Mac user is more likely to have a college degree".

~~~
minimaxir
OP of the article here.

Microaggressions aside, these are fair counterpoints. I spent far less time
editing the body of the post than optimizing the visualizations/Jupyter
Notebook (especially in this particular post). I've taken more care in future
posts since.

~~~
tedsanders
If you are looking for feedback, here is another suggestion for future posts:
I found the use of violin plots for discrete data to be confusing. To be
honest, I still am not sure how to interpret the unlabeled Y axis. I think a
histogram would have been easier for me (and others) to interpret.

But suggestions aside, I found your article to be interesting. Thank you for
it.

~~~
joshuahhh
I second the criticism of using violin plots here. A violin plot is a kernel
density plot. It is designed to show a distribution at a scale much larger
than variations in the data. Its raison d'etre is to smooth out and aggregate
these small-scale variations.

But in this survey data, the values in the distribution are spaced far apart.
The discretization is so large that the violin plots show meaningless and
weirdly inconsistent curves between x-values which actually have data. A bar
plot would be much clearer.

OP, I think you have done a fine job of styling your plots tastefully. But I
recommend taking another look at the visual language you have chosen to
communicate the data.

~~~
minimaxir
Agreed. The violin plot was an experiment that did not work out in retrospect.

~~~
vadman
I learned that violin plots exist, so your experiment was worth it to me :)

------
partycoder
"Programming ability" is something very vague.

There are many domains of expertise. Dealing with networking, graphics, sound,
threads, filesystems, embedded platforms, compilers, cryptography, operating
systems (which is in itself another world), distributed systems, databases,
drivers, high performance computing, machine learning, etc.

Ask a senior network programmer to write you a shader and ask a graphics
programmer to write you a production-strength multithreaded socket server...

Ask a senior web designer to write you a compiler from scratch, and ask a
senior compiler engineer to write you a responsive website that renders
correctly on all browsers.

Then, some programming abilities are rarer. There are more web developers than
compiler engineers, the amount of jobs and the requirements for them are
different.

What I am trying to get at is that software development comes on many flavors
and it's really hard to touch all those surfaces in your career.

------
pascalmemories
Looks exactly like driving competency. Over 93% of US drivers self-rate their
driving as above average (even significantly above average). It's very
frightening to see that 36% rate themselves as above average drivers _whilst
sending text messages_.

There's even a name for it, Illusory Superiority
[https://en.wikipedia.org/wiki/Illusory_superiority](https://en.wikipedia.org/wiki/Illusory_superiority)

edit: added some of the latest figures from the Wikipedia article.

------
emodendroket
Lake Wobegon: Where all the women are strong, the men are good-looking, and
the children are above-average.

------
k__
I get this asked a lot and if I tell people "If Carmack and Torvalds are a 10
and someone who can write a hello world is a 1, then I would rate myself 5
max" and most just think I must be a bad programmer because of that rating.

~~~
aninhumer
Well even with those bounds, is that scale in terms of the population as a
whole, some measurable notion of productivity, or in terms of some abstract
concept of absolute skill? Depending on the definition, 5 could mean many
different things.

Not to mention that there are different aspects of skill. Is Torvalds a good
functional programmer? (I actually have no idea, so this might be a bad
example, but hopefully I make my point.)

------
irq-1
The consistent trend in unemployed / part time / full time is a little strange
considering that many programmers value time to work on their own projects,
and can gain new skills that are valuable to an employer. Seems like HR and
cultural attitudes about employment are large factors in programmers view of
skill.

------
gregmac
The number of people that picked "never check in or commit" or only do it
"once or twice a month" but that also rate themselves as a 7+ makes me a
combination of sad and irritated.

------
coldcode
Interesting analysis. I do wonder what a self rating for developers actually
means: how do you define the difference between a high rating and a low
rating? What is a great developer vs an average one?

------
sytelus
This is a great piece. Of course, the headline stat for me was:

 _As it turns out, there is no correlation between programming ability and and
the frequency of Stack Overflow visits_

------
callesgg
I always get a bit pissed of/jealous when i see how much Americans earn.

I wonder if Indians and people from other lower wage countries feel the same
about my own salary.

~~~
hood_syntax
Have you tried freelancing for companies in the US? I imagine that could open
up some opportunities for you if you haven't looked into it already.

~~~
callesgg
No, i have always assumed that it is hard to get permits and even get
interviews.

It is certainly hard to be picked from the crowd here in Sweden.

------
austincheney
The definition of competence wildly varies based upon who you ask. There are
many experienced developers in the work force, whom I am sure would rate
themselves high, who cannot do their jobs (at all) without their favorite
frameworks and abstractions. Epic fail. They might rate themselves with an 8,
but I would give them a 3.

~~~
tbrownaw
And perhaps those same developers would have a similar opinion of someone who
keeps reinventing the wheel?

~~~
austincheney
I am sure they likely do, which only suggests developers' opinions of
stylistic qualities and tool choice is an extraordinarily weak metric of
competence. A better and more objective metric is an evaluation of the
product: the speed of code writing, execution speed, fewer dependencies, few
lines of code, and so forth. From that you can guess an opinion that better
software is written by better developers. Asking developers to rate each
other, or themselves, is next to worthless.

------
blauditore
Interesting, though no big surprises.

Just the hick in the salary/ability plot at 160k-170k makes me wonder what's
going on there. It might just be inaccuracy due to small sample size, as this
range is probably an unusual salary. I guess people tend to gravitate more
towards round numbers like 150k and 180k when negotiating salaries.

~~~
skylark
165k is the standard offer for a fresh graduate at Google.

~~~
swolchok
salary or salary + bonus target + equity?

~~~
skylark
Salary + bonus + equity

------
sigstoat
i'd like to see the same group of people asked to pick a random integer
between 1 and 10 inclusive, and compare that distribution to what we see in
the first graph.

------
alexpetralia
Nice data analysis. Definitely a selection bias with respect to self-
assessment - I am sure many are comparing to 'programmers' they know in the
workplace who are not particularly strong and would seldom use a resource like
S.O. (and consequently not respond to a survey like this).

~~~
Bartweiss
This was the first thing that I thought of also - surveying Stack Overflow
survey-takers about their place in the larger order of programming threatens
sample bias in several ways (experience, community engagement, etc).

------
Terr_
> As it turns out, there is no correlation between programming ability and and
> the frequency of Stack Overflow visits, as the averages and distributions
> are virtually identical across all groups.

As an infrequent SO participator, this is somewhat comforting to hear.

~~~
rifung
> As an infrequent SO participator, this is somewhat comforting to hear.

I'd assume that people who visit infrequently are better on average than those
who visit often because it means you don't have to look things up. Why would
you assume the opposite?

~~~
Terr_
Uhm, I feel your reply is a little bit of a non-sequitur, or at least I'm
having trouble following your line of logic.

The linked article says that people's _self-reported_ skill doesn't seem to
depend on how often they "visit" SO, which includes answering questions and
commenting. Given that this data is from people who actually filled out a
survey on the site, that automatically excludes a big portion of the people
who are "just looking".

The reason I say it's comforting is because it weakly contradicts a worldview
(one I dislike) where "good programmers are visible programmers", and people
are often get judged by their blog-posts and their tweet-followers and their
SO answers.

------
nickthemagicman
Would love to compare this to word charts of their questions asked and see
what topucs they ask about. Sort of a dunning kruger test.

------
PunchTornado
what programmer gives himself a 10 out of 10?

probably one that is at level 4-5 and cannot grasp what he doesn't know.

~~~
sgdread
John Carmack?

