

Are Oklahoma Students Really This Dumb? Or Is Strategic Vision ... - riffer
http://www.fivethirtyeight.com/2009/09/are-oklahoma-students-really-this-dumb.html

======
btilly
From the answers to the questions I am inclined to believe in the "made it all
up" theory. But at a 99% confidence level I can reject the possibility that it
was made up with a simple simulation of the type described.

How am I so sure? Well we're given all of the probabilities for all of the
answers. We can therefore calculate what (assuming independence) the likely
odds are of different numbers of correct answers. We're also given the number
of students who got each number correct. Here is the result (I'll give the
program that I used at the end):

    
    
      0 right: 46 vs 32.56
      1 right: 158 vs 147.61
      2 right: 246 vs 273.73
      3 right: 265 vs 278.71
      4 right: 177 vs 174.46
      5 right: 80 vs 70.58
      6 right: 22 vs 18.74
      7 right: 6 vs 3.23
      8 right: 0 vs 0.35
      9 right: 0 vs 0.02
      10 right: 0 vs 0.00
    

The argument is that we should be suspicious because those two sets of numbers
are too similar. The tails are not heavy enough. Now I admit that graphically
it does look close. But if you look, the tails _are_ heavy. Is that difference
significant?

We can use a confidence testing to answer that. The null hypothesis is that
the distribution comes from a simple simulation and should be close to the
theoretical numbers that I just gave. The alternate hypothesis is that the
tails should be heavier and the middle lower.

Let's look at the middle because it is easier to calculate. Theoretically
0.55244 of the test population should get one of those two numbers correct. It
is claimed that 511/1000 did. How likely is that?

Well in the null hypothesis each student is an independent observation. Each
observation is a 1 or 0 with average 0.55244. The variance of an observation
is readily calculated and turns out to be 0.2472500464.

The number of people getting one of those two answers after 1000 observations
is therefore approximately normal with average 552.44 and variance 1000 *
0.2472500464 = 247.2500464. The standard deviation is the square root of that,
which is 15.724. That puts the observed value more than 2.63 standard
deviations out. at a 99% confidence we can conclude that the numbers were NOT
generated by a simple simulation of the type described.

For the curious, here is how I calculated the theoretical distribution of
answers:

    
    
      #! /usr/bin/perl -w
      use strict;
      use Math::Polynomial;
    
      my $ans = 1;
    
      # Number from the % of students who got each question right.
      for my $x (qw(.28 .26 .27 .1 .14 .61 .43 .11 .23 .29)) {
        $ans *= Math::Polynomial->new(1-$x, $x);
      }
      
      print "$ans\n";
      
      __END__
      
      (5.295902595984e-07 x^10 + 2.07154949414e-05 x^9 +
      0.000345419801607472 x^8 + 0.00322944050567802 x^7 +
      0.0187445906666514 x^6 + 0.0705826597827996 x^5 +
      0.174464060957845 x^4 + 0.278709766012616 x^3 +
      0.273731283943492 x^2 + 0.147609870020204 x +
      0.0325616632239042)

~~~
dagw
Assuming the people faking these sort of numbers aren't complete morons
they'll know that this sort of naive statistical test is the first thing
people will try. Thus they'll probably run their simulation and then manually
poke a few of the numbers to make sure that this sort of test fails.

Also If they're slightly smarter still they'll realize that the chance of
getting a question right or wrong is not independent. You're chance of someone
getting questions 5-10 right is probably quite closely correlated to whether
or not they got questions 1-4 right.

Anyway my basic point is that assuming the person in charge of writing a
simulation didn't sleep through all of statistics 101 it is quite trivial to
write a simple simulation that won't be detected by simple statistical
analysis.

~~~
btilly
You're overestimating them. The final digit analysis that broke this whole
story is something that only catches numbers being made up by hand. If they
rolled a die to randomize things, they wouldn't have left an obvious pattern
like that.

Given that, it was reasonable to investigate the obvious possibility first for
this case. Particularly since there was a suggestion that they had made up the
set of numbers with such a simulation.

~~~
dagw
Maybe I'm guilty of overestimating them, but I always find it hard to believe
that people who would go through the trouble of faking these sort of things
can be so monumentally stupid. These people work with polling and statistics,
they must know about all of these tests. It cannot be that hard to use the
knowledge to write a randomizer which produces reasonable results...maybe
that's a good idea for a startup.

~~~
camccann
_These people work with polling and statistics_

Well, current evidence seems to suggest that maybe they don't, actually, work
with polling and statistics...

------
CrazedGeek
An analysis of the post: [http://blog.smellthedata.com/2009/09/analysis-of-
pollster-fr...](http://blog.smellthedata.com/2009/09/analysis-of-pollster-
fraud-and-oklahoma.html)

~~~
yannis
Both posts are very well thought out. I always mistrust polls and any
statistics that don't publish details of their methodology and raw data.

Sometime back I spent some involuntary time in the army. We had to take a
number of tests. I answered most of them randomly.

On the IQ test, I really tried h a r d to get all the questions wrong, in the
vain that they will think I was an idiot and they let me go (they never did
and no-one ever gave me a strange look).

I sympathize with those Oklahoma kids and understand that there many ways to
tell a grown-up a 'fuck you' ;)

~~~
CrazedGeek
As an Oklahoman high-schooler, I think the "they made the whole thing up"
hypothesis is way more likely. Either that or they got a _really_
unrepresentative sample. I know that at least half of my 2000 student school
could have made at least a 6 on that quiz. Not too many people here actively
try to buck the system, sadly.

~~~
TetOn
I tend to agree, though my feelings fall more on getting student buy-in.
Without some real outcome that matters (a grade, or something similar) what's
to insure a reliable sample. Assuming things haven't changed since I was in
high school, I'm pretty sure we'd have gone out of our way to skew the test
purposely with outrageously wrong answers...

------
chrischen
"communist and republican" ... lol those kids got brainwashed!

~~~
andreyf
The question is a bit of an affront to one's intelligence, so it makes sense
to pick the silly answer ;)

~~~
dagw
I remember when I was in high school and they where doing some sort of poll
about the upcoming nation election at several school. Somehow word started to
spread around that we should all 'vote' for the Communists, so we did.

A few days later there was an article in the local newspaper about a
disturbing trend showing the over 20% of 16-18 year olds would have voted
communist and what this might mean for the next election when all these kids
have the right to vote. We found the whole thing highly amusing.

