Nate Silvers wrote an entire book on this subject called "The Signal and the Noise" [1]. Humans are so often taken in by people claiming to be able to make predictions by combining new data points. The more unusual, or unrelated to the subject matter, the better. They make good headlines but (not surpsingly) almost always turn out to be heavily flawed in practice.
You can basically measure how much a pundit/expert is going to be wrong in their predictions by how ideological they are in their analysis. The best indicator is when they use only one or two metrics as a basis of a prediction of an otherwise very complex scenario.
One example from the book is how a researcher became famous before the 2000 US presidential elections by claiming to predict races with 90% accuracy [2]. He claimed that by measuring a) per-capita disposable income combined with b) # of military causalities you can determine whether democrat or republicans get elected. He said historical data backs up his theory. He then proceeded to fail to predict that years election and faded into obscurity.
Nate did his own historical analysis and demonstrated it was only 60% accurate instead of 90%. Plus that was only if you ignore 3rd party candidates as the model assumes a two-party system.
Plenty of other examples are provided in the book which makes me highly suspicious of the value of the predictions made in this article.
The general idea is that we need to stop looking for simple one-off solutions to complex problems. Instead we should adopt multi-factor approaches which suffer from fewer biases and are better grounded in reality. Otherwise these predictions are just another form of anti-intellectualism.
The law of averages will tell you that the more metrics (however random) you throw into your prediction engine, the closer your prediction will be to the actual result. But it's not very remarkable and it will never put you ahead consistently.
You can basically measure how much a pundit/expert is going to be wrong in their predictions by how ideological they are in their analysis. The best indicator is when they use only one or two metrics as a basis of a prediction of an otherwise very complex scenario.
One example from the book is how a researcher became famous before the 2000 US presidential elections by claiming to predict races with 90% accuracy [2]. He claimed that by measuring a) per-capita disposable income combined with b) # of military causalities you can determine whether democrat or republicans get elected. He said historical data backs up his theory. He then proceeded to fail to predict that years election and faded into obscurity.
Nate did his own historical analysis and demonstrated it was only 60% accurate instead of 90%. Plus that was only if you ignore 3rd party candidates as the model assumes a two-party system.
Plenty of other examples are provided in the book which makes me highly suspicious of the value of the predictions made in this article.
The general idea is that we need to stop looking for simple one-off solutions to complex problems. Instead we should adopt multi-factor approaches which suffer from fewer biases and are better grounded in reality. Otherwise these predictions are just another form of anti-intellectualism.
[1] http://www.amazon.com/Signal-Noise-Many-Predictions-Fail--bu...
[2] the "Bread and Peace" model by Douglas Hibbs of the University of Gothenberg http://query.nytimes.com/gst/fullpage.html?res=9803E5DD1F3DF...