
Nate Silver Finds Unusual Patterns in a Polling Firm's Data. Was it fabricated? - gluejar
http://www.fivethirtyeight.com/2009/09/comparison-study-unusual-patterns-in.html
======
isamuel
This is pretty damning stuff. In a well-functioning media environment, it
would be sufficient to seriously damage the credibility of the pollster
involved (Strategic Vision), or at least shift the burden to them to explain
their behavior.

Instead, we already see clues about what will happen: the pollster is
threatening a lawsuit, which will be a sure loser but nonetheless drain
resources and time from its target (either Nate himself or fivethirtyeight).
The polls from the offending pollster will, I'm completely certain, be
reported with utter credulity in the 2010 election cycle and beyond. The fact
that the books are cooked will not even be mentioned in media reports, and
will ultimately matter not at all.

This is because we simply do not have a well-functioning media environment.

~~~
cwan
I think their lack of disclosure has already severely damaged them.

Ultimately, in this case it would seem the damage will be pretty brutal unless
Nate Silver was wrong. Either they don't sue and don't disclose and the result
is no one trusts them enough to pay for their polling (after all, what's the
point in paying for a poll if it's going to come under question afterwards?)
or they'll sue while not releasing the support for their polls/data on their
merits - and that also opens them up to discovery which could be even more
damaging.

The best case scenario at this point is if they can publicly prove Nate Silver
is wrong. And based on their policies in the past of not releasing the
additional supporting data, I suspect this won't happen.

------
callmeed
With newspapers suffering so much, it's refreshing to see a blogger taking the
time to perform investigations like this. We need more of this.

~~~
cwan
My sense has been newspapers have been suffering so much _because_ they don't
take the time to perform investigations like this. This is hardly the only
example with the Acorn videos quite recently for instance. We do definitely
need more of this.

~~~
philwelch
The trouble with the Acorn videos is that they're a partisan attempt to
discredit the opposition. While the videos themselves seem legit, I'd rather
not leave exposes of political groups up to their political enemies. Political
enemies only want to discredit their opponents--they're not interested in
truth, and for every Acorn video there's a Swift Boat smear campaign with no
basis in fact.

~~~
anamax
> The trouble with the Acorn videos is that they're a partisan attempt to
> discredit the opposition.

We already know how the civility argument works. The tea-party folks are
bringing violence to US politics and the G20 thugs are crumpets and milk.

> While the videos themselves seem legit, I'd rather not leave exposes of
> political groups up to their political enemies.

Bully for you, but they're pretty much the only folks willing to do the work.

If you don't like the way news is made, make your own.

~~~
philwelch
I wasn't making a civility argument and I don't have the time in my life to
fix every social problem I can identify.

Do you have a constructive point to make? You've taken everything I said
completely out of context and made completely irrelevant responses. As it
happens, the last sentence is the central point of my comment, and serves to
motivate and justify the preceding two sentences. I'm sorry if that wasn't
clear enough.

Maybe it would be useful to me to remember that a lot of people read English
as if they were finite automata--once they read a given sentence they get
their heads into a given state, even if the sentence immediately following it
provides essential context to what I actually meant.

~~~
anamax
> I don't have the time in my life to fix every social problem I can identify.

I didn't say that you did. However, you do live in a world with those nasty
social problems....

> Do you have a constructive point to make?

My point is that either you just bitch or you set up unreasonable constraints.
The difference between the two or your reasons for them may be important to
you, but the end result is pretty much the same.

> You've taken everything I said completely out of context and made completely
> irrelevant responses. As it happens, the last sentence is the central point
> of my comment, and serves to motivate and justify the preceding two
> sentences.

Oh really? Let's review that last sentence "Political enemies only want to
discredit their opponents--they're not interested in truth, and for every
Acorn video there's a Swift Boat smear campaign with no basis in fact."

That sentence says that you find it likely that the Acorn tapers managed to
capture some "fact" and may have been interested in truth. Yet, that's not
good enough for you. Since the folks with appropriate motives aren't doing the
work, dismissing folks with "bad motives" means that you're going to do
without.

Someone else already pointed out that your "no basis in fact" statement is
false so it's unclear why you'd bring up that sentence.

------
DarkShikari
Particularly interesting is the further analysis done in the comments where
similar statistical methods are used to analyse other polling agencies'
results. Nearly every single one seems to line up to look almost exactly like
the Quinnipiac results, further damning Strategic Vision. But what interests
me more is the question of exactly _why_ this distribution comes up; it's not
Benford's Law, since probabilities aren't a power-law distribution, yet it
seems consistent across multiple sources.

------
jonknee
A true statistics hacker. I love reading his analysis--he was the go to source
during the election for anything poll related.

------
danger
Here's a bit more on the subject: <http://news.ycombinator.com/item?id=846514>

------
kevinpet
It seems to me that if there were one data source I would expect to _not_ be
consistent with Benford's law it would be polling data. Benford's law is
usually seen in numbers that follow some skewed distribution. Polling data
groups around maybe 45% depending on the number undecided. I don't know what I
would expect, but polling extremely close races, or polling close to the
election, I would expect to see lots of high digits (6 7 8 9). I wouldn't
expect many 41 or 31 (those questions are uninteresting) so the 1s come from
mostly 51s.

He may be right, but I'm going to need a little more convincing if someone
wants to assume that polling data would exhibit Benford's law.

