
Please be more careful when interpreting the Stack Overflow Developer Survey - SoReadyToHelp
https://meta.stackoverflow.com/q/382731
======
dahart
> We asked respondents to evaluate their own competence, for the specific work
> they do and years of experience they have, and almost 70% of respondents say
> they are above average while less than 10% think they are below average.
> This is statistically unlikely with a sample of over 70,000 developers who
> answered this question, to put it mildly.

I’m seeing a lot of argument about statistics, and very little about the
wording of this question.

It seems to me that by asking specifically for competence “for the specific
work you do” means that everyone gets to define who they’re comparing against.

What this means is that it’s entirely statistically possible for 70% of people
to be above average relative to whoever they choose their peers to be. I might
compare myself to you, and you don’t compare yourself to me, so we can both be
above average.

In order to argue about the math and statistics, the question needs to be well
defined, and the group to which we’re talking about averages needs to be the
same for everyone. This question, the way its worded, practically guarantees
that the group someone compares themselves to is different for every single
respondent.

~~~
dahart
It’s too late to edit, and maybe nobody will see this now, but I wanted to add
that the particular question of competence and the majority result strongly
reminds me about the story of the Air Force discovering that cockpits designed
for the average human size were causing crashes because literally nobody is
averaged sized.

Discussed on HN before
[https://news.ycombinator.com/item?id=11230287](https://news.ycombinator.com/item?id=11230287)

“Before he crunched his numbers, the consensus among his fellow air force
researchers was that the vast majority of pilots would be within the average
range on most dimensions. After all, these pilots had already been pre-
selected because they appeared to be average sized. (If you were, say, six
foot seven, you would never have been recruited in the first place.) The
scientists also expected that a sizable number of pilots would be within the
average range on all 10 dimensions. But even Daniels was stunned when he
tabulated the actual number.

“Zero.

“Out of 4,063 pilots, not a single airman fit within the average range on all
10 dimensions. One pilot might have a longer-than-average arm length, but a
shorter-than-average leg length. Another pilot might have a big chest but
small hips. Even more astonishing, Daniels discovered that if you picked out
just three of the ten dimensions of size — say, neck circumference, thigh
circumference and wrist circumference — less than 3.5 per cent of pilots would
be average sized on all three dimensions. Daniels’s findings were clear and
incontrovertible. There was no such thing as an average pilot. If you’ve
designed a cockpit to fit the average pilot, you’ve actually designed it to
fit no one.”

[https://www.thestar.com/news/insight/2016/01/16/when-us-
air-...](https://www.thestar.com/news/insight/2016/01/16/when-us-air-force-
discovered-the-flaw-of-averages.html)

So to go one step further and make a stronger statement, it’s not just
possible for 70% of devs to be above average, it’s statistically likely for
100% of devs to be above average with respect to _something_ they do, relative
to _some_ group. It’s so subjective that it’s meaningless, and we can’t really
talk about the statistics.

------
d0mine
> It is an error to use the sample size of a non-random sample to support the
> underlying comparison with the population of interest. Sample size can
> decrease random error, but not bias

------
_pastel
Next year, ask: "Compared to developers of similar position, experience, and
willingness to respond to Stack Overflow Developer Surveys, would you consider
yourself less competent than average, of the same competency as average, or
more competent than average?"

~~~
leshow
How can you compare another developers willingness to respond to SO surveys to
your own?

------
asaph
> any conclusions drawn by such studies must be taken with a not-insignificant
> sized grain of salt.

Please be more careful about drawing conclusions about the statistical
significance of salt grain size. ;)

------
superpermutat0r
Abuse of the Dunning Kruger effect, lead to a faulty conclusion that data
can't support.

------
shadowmint
> you cannot generalize from a non-random sample

So, honest question:

If any survey of any size can be ignored on the basis that the sample is not
random, then how is any survey meaningful?

Isn’t this a self defeating argue?

You can’t prove the sample is random, all you can do is show differences
between samples and suggest its not _consistent_... but how do we go away and
prove that some other survey we’re comparing it to is from a random sample?

ie. Isnt this just a convenient excuse to deny that a survey is meaningful?

Statistically, how do you _mathemtaically quantify_ the effect of selection
bias?

...because, it seems to me, unless you can actually do that, you’re just doing
some arm chairmhand waving because you don’t like the results youre seeing.

This has come up several times (eg. js survey about react vs angular), and no
one has ever given me a meaningful and mathematical response.

Its always just.. “it must be sample bias”, regardless of the 90000 people
they surveyed.

I don’t accept you can survey 90000 developers and cannot offer any
generalisation from those results _without quanatitively proving_ there is an
overwhelming sample bias, and _specifically_ quantifying the degree of that
bias.

Am I missing something here? Everyone seems thoughorly convinced that this is
perfectly normal.

(I’m not proud, I’ll take your down votes, but please answer and explain what
I’m missing)

~~~
prepend
The key is in how you randomly select the sample from the population.

This was the author’s point. Just because you have 90k SO respondents doesn’t
mean you can say anything about developers as a population. You can say lots
of stuff about SO users. Or maybe developers who use SO. But just because you
have lots of responses doesn’t mean you know what developers or jugglers or
farmers or whatever population interests you.

The confusion rests with SO’s statement that their survey should be
representative of developers in general (or CS graduates or whatever other
than only SO visitors).

~~~
balfirevic
It's not even a random sample of SO visitors, as the there is, at the very
least, self-selection bias.

~~~
prepend
I agree, although I think it’s more easy to correct for this bias to
generalize to all of SO than to all programmers.

