
A Statistical Analysis of Coughing Patterns on ‘Who Wants to Be a Millionaire?’ - okasaki
https://medium.com/@liam.philip.shaw/a-statistical-analysis-of-coughing-patterns-on-who-wants-to-be-a-millionaire-187be5cc6af1
======
fanzhang
A p-value of 0.3% is not that low given the multiple hypothesis testing nature
of the show.

A quick Google result shows that 592 total episodes have been filmed of the
show. Under the null hypothesis that there was no cheating in any of the
shows, you would still expect more likely than the chance that _some_ show has
a strong p-value of 0.3% for cheating to be 1-.997^592 = 83%.

As a jury, I don't think I would convict with just a p-value of 0.3% alone.
You'd want something like a DNA test with a p-value expressed in 10^-6 or
something to reduce the problem of multiple hypothesis testing / setup
assumptions. Or you'd want to rely on other evidence like a confession or
wiretap.

~~~
hedora
Note that 10^-6 for DNA testing would be strong evidence if the suspect was
found via a traditional investigation.

If it was found by searching all of 23andMe it would be meaningless on its
own.

------
ikeboy
This neglects selection effects. There's a reason he's watching this episode
instead of other episodes - if the studio investigates every large win and
only makes noise about it when they have some level of evidence, as here, then
there's a strong selection effect.

~~~
gerdesj
From Section 5 (Summary): "This analysis has some obvious limitations. ...
Ideally, we would also analyse other episodes of WWTBAM and see if the
distribution of coughs is similar to this infamous episode. (I hope to do this
when I have a chance.)"

I'm looking forward to that analysis as I am sure you are too.

~~~
ikeboy
But coughs is not the only relevant variable. You'd have to analyze everything
that could plausibly be used as an accusation of cheating. Looking only at
coughs is already falling prey to selection bias.

~~~
celticninja
not really, if they say coughs were the method of cheating then you can just
look at coughs. if their claim was any noise counts,then yes you have to look
at sneezes or any other sound. however to be able to use this then you need to
know what you are listening for, so a range of sounds is not effective.

~~~
ikeboy
Yes really.

Suppose there's a hundred different possible sounds, and each sound has a 1%
chance of randomly correlating . The probability of at least one sound
correlating would be very high, but once that's identified you would calculate
a 1% chance there.

That's selection bias. You selected the cough to look at _because_ coughs were
the ones that correlated. If something else correlated you might have looked
at that one.

~~~
celticninja
are you familiar with the program and specific episodes?

~~~
ikeboy
Nope

~~~
celticninja
it may aid your understanding if you were. specifically this guy used up his
lifelines in the easiest questions, and then had a clear run on the difficult
ones. it was a questionable performance, hence there being an investigation
and refusal to award the prize immediately.

------
im3w1l
I wonder what standard of evidence would be appropriate. He says 0.3% of
happening randomly. Another way of stating that is that his test has a 0.3%
false positive rate. And given the number of shows produced, that ends up a
high absolute number.

~~~
goodcanadian
He analyzed one episode. The 0.3% probability is of that pattern happening in
that episode. It would be different if he went searching for patterns through
all episodes, but that is not what he did.

~~~
im3w1l
He selected an episode known for having suspicious coughing. So essentially
someone else searched for patterns through all episodes for him.

~~~
goodcanadian
I think that is still an uncharitable interpretation of events. No one is
claiming (so far as I know) that Who Wants to be a Millionaire? scoured all
episodes looking for suspicious patterns. This is one episode where cheating
was already suspected (though I would like to know on what grounds they
suspected it).

~~~
carlmr
>No one is claiming (so far as I know) that Who Wants to be a Millionaire?
scoured all episodes looking for suspicious patterns

Because it's normally not worth it. This selected a show with a big win where
it was worth analyzing it.

Big winners will always either know a lot, or be lucky (and bold) with their
guesses. So this is already a pattern that was selected for.

Now given this backwards reasoning every time the guest said something else
but changed his mind will look like cheating. But you already selected only
the cases where they did get lucky, so that shouldn't factor in to your
thinking, but it obviously does because humans are bad at separating this
rationally.

No matter the percentages, absent any proof of collusion I don't think this
conviction holds up, since nervous cough when seeing the right answer is
enough of an explanation to make this plausible without any connection between
the two.

~~~
goodcanadian
You are arguing a different point than the original comment I was replying to,
however. The original argument was that even with only a 0.3% probability of
being coincidental, you would still expect to find one episode out of the
entire run where it occurs. This is true. However, that doesn't make it likely
that this episode is the one. If they had trawled the whole series looking for
weirdness, and picked out the one episode with weirdness, that would be a
major issue. They didn't do that. They picked this specific episode for a
different reason (that I am still not completely clear on). The analysis shows
that the coughing pattern was extremely unlikely to be coincidental. You are
correct that doesn't necessarily mean cheating, but it is an interesting
result that shouldn't simply be dismissed with hand wavy statistical
arguments. The correct follow-up is as mentioned in the article. If the
nervous coughing explanation is correct (for example) then most episodes
should show a similar pattern which would undermine the conviction.

~~~
carlmr
>If the nervous coughing explanation is correct (for example) then most
episodes should show a similar pattern which would undermine the conviction.

That doesn't necessarily follow. It could still be that this one person has a
nervous cough response to correct answers. Doesn't mean everyone with a cough
does the same.

------
ngneer
I wish I had that kind of time and exploratory spirit on my hands. Good job!

------
syntonym2
I haven't heard of that case before, but I found the evidence presented in the
blogpost not very convincing. The first argument states that assuming that
coughs are independent, the cough pattern is unlikely. But the data later
shows that they are hardly independent but cluster, often two or more coughs
are in a short timespan.

The second argument goes through each of the questions, but doesn't really
show any evidence for cheating. Question 8 has no suspicious coughs (coughs
close to the correct answer), and Ingram knows the answer quickly. Question 9
contains one suspicious cough, but only after the second mention of the
correct answer. Again Ingram is quite sure about this answer and does not
consider any other answer.

Question 10 is the first question (of that day) Ingram struggles with. There
are 7 coughs recorded, 5 cough "clusters" (coughs very close to each other), 2
cough cluster could be suspicious. The first potentially suspicious cough
cluster is pretty far away from the answers , while the second one is close to
the correct answer. The next notes that these coughs didn't come from the
phantom cougher, but from his wife. On the other hand there are lots of better
place to insert a cough to cheat. This is one of the few questions where he
changes his mind.

Question 11 has five coughs in total and three suspicious coughs, but cough
2-5 are very close together (and very late). Ingram is focused on the correct
answer from the beginning, and does not really consider any other answer.

Question 12 has one cough after an incorrect answer at the very beginning.
There is one cough cluster later on after the correct answer, but the correct
answer is said 5 times, whereas the only one other answer is mentioned once,
and only considered for a few seconds. The other coughes are relatively far
away from any answer.

Question 13 has a cough after the correct answer in the very beginning, but it
is not called as a significant cough in the linked youtube video from WWTBAM.
Only the correct answer is considered.

In Question 14 Ingram struggles similar to question 10. there are 12 coughs. 6
coughs are heard shortly after a wrong answer (coughs 1, 2, 3, 4, 5 and 9),
and 4 coughs are heard shortly after the right answer (coughs 6, 10, 11, 12).
The next mentions a muttered "no", but any muttering loud enough to inform the
contestant surely must be heard by the host and the crew. If they cheated
using coughs, why muttering very obviously?

Question 15 has 23 coughs, 7 are close to wrong answers and 10 are close to
the right answer. Ingram speaks the correct answer 11 times, and lists the
incorrect answers 4 times. Most of the suspicious coughing happens in the very
end, when he basically made his decision. Most of the coughing close to the
incorrect answers happens early. If coughing was used, why did he not lock
into one of the incorrect answers? The text mentions Ingram muttering "I think
I know..." and calls that suspicious, but that phrase seems a very natural
choice here.

The questions, coughing and answering doesn't follow a pattern: Sometimes
there is a cough very early after the correct answer and Ingram picks that
option, sometimes there is a cough very early after the incorrect answer put
Ingram doesn't pick that option. Sometimes for unclear questions there are
multiple coughs after the correct option, and Ingram picks that option;
sometimes for unlcear options there are multiple coughs after the incorrect
option and Ingram does not pick that option. On some unclear questions there
is no coughing at all, but Ingram still guesses correctly. On some clear
questions there is coughing, although Ingram did not even consider any wrong
answer.

Next the blog post compares distributions of elapsed time since last answer
for correct and incorrect answers. It does not use a statistical test (e.g.
Kolmogorow-Smirnow-Test), but just counts coughs to some threshold. Resolution
is very rough, the subsampling for that plot only done once. While simulations
show that there are more coughs after an correct answer then expected, there
are also more coughs after an incorrect answer then expected. The simulations
disregard that for most incorrect answers there is not a lot of time until the
next answer is spoken. Indeed often the incorrect answers are simply listed
and immediately disregarded.

While the presented data might call for some scrutiny, it is far away from
justifying the damning tone.

------
JoeAltmaier
Forget random. Maybe it was unconscious? Kind of a 'tell'. Doesn't mean there
was malicious intent, or planning ahead of time.

And what's the big deal? Is it actually a crime to win a game show? I'm
confused.

~~~
celticninja
the crime is to dishonestly acquire something, in this case the actual charge
was fraud IIRC.

They conspired to win the prize in a way that was outside the rules, by
obtaining help that they were not allowed to use. so the crime isn't winning
it's how you won, and then did you really win or did you cheat.

much like in a sporting event, you can win by taking performance enhancing
drugs, just because you came first doesn't mean you won.

~~~
JoeAltmaier
Fraud is a tricky thing. It requires not only deceit, but 'legal injury'. The
show producers clearly had great ratings, a popular show and considerable
exposure due to the sensational nature of this family's win. Their winnings
was a tiny fraction of a marketing budget. Arguably not a lot of legal injury
there.

'But it was not fair!' is not a legal argument. Neither is 'outside the rules'
since rules are not laws or regulations. Just some words the director spoke in
their direction; hot air.

~~~
DanBC
This is the law he was prosecuted under:
[http://www.legislation.gov.uk/ukpga/1968/60/section/20/1994-...](http://www.legislation.gov.uk/ukpga/1968/60/section/20/1994-11-02)

> A person who dishonestly, with a view to gain for himself or another or with
> intent to cause loss to another, by any deception procures the execution of
> a valuable security shall on conviction on indictment be liable to
> imprisonment for a term not exceeding seven years; and this subsection shall
> apply in relation to the making, acceptance, indorsement, alteration,
> cancellation or destruction in whole or in part of a valuable security, and
> in relation to the signing or sealing of any paper or other material in
> order that it may be made or converted into, or used or dealt with as, a
> valuable security, as if that were the execution of a valuable security.

He intended to gain, and he collaborated with someone else to cheat. Loss
isn't required.

Under the newer law (Fraud Act 2006) it's not necessary to show any gain.

[https://www.cps.gov.uk/legal-guidance/fraud-
act-2006](https://www.cps.gov.uk/legal-guidance/fraud-act-2006)

> It is not necessary to prove or demonstrate any consequences of fraud
> (though they will clearly be material to sentence, compensation and
> confiscation). "Preddy" type difficulties will not arise (where the property
> obtained had not belonged to another);

To me it feels odd that you'd need to show a loss. How would you prosecute
people who attempted but failed to defraud others?

~~~
JoeAltmaier
But...'cheating' at a game, making that have the force of law, makes game-show
hosts into lawmakers. That seems a strong stance. We're not talking a
contract, nor a physical exchange of goods and services, nor anything else
normally associated with civil cases. We're talking, the accused may have
operated outside a set of arbitrary rules made by some civilian for fun.

~~~
DanBC
> makes game-show hosts into lawmakers.

How? They're not adding anything into law.

> nor anything else normally associated with civil cases

This is a criminal, not civil, case.

~~~
JoeAltmaier
Miss the point any? To be criminally prosecuted for not following a rule some
producer made up, makes that rule into law. Because its enforceable in a court
apparently. Really?

Being a criminal case makes it more egregious.

------
cryptica
This statistical analysis does not account for the fact that there is such a
thing as 'nervous cough' which is anxiety-related. Stress hormones are likely
to spike during stressful moments around the time a question is asked which
may trigger a cough reflex from audience members.

The analysis seems to focus on the wrong things. A proof of cheating should
focus more on identifying specific and reliable patterns in the coughs. I
don't think generic statistical analysis of this kind makes a strong case at
all.

~~~
have_faith
He mentions at the end of the article trigger related coughing.

