

Wrong Tomorrow - holding pundits accountable for predictions - gabrielroth
http://wrongtomorrow.com/

======
sethg
I have this mental list of "things I would love to see and am too lazy to
actually implement right now and maybe someone else will come up with the same
idea and spare me the work".

And now I can cross another item off of that list.

~~~
smanek
Yep, I just wish people could bet on these predictions too. Unfortunately,
that is illegal in the US ...

~~~
ashot
virtual currency

~~~
smanek
That defeats the whole point, unless the virtual currency is worth something
in the real world - at the very least considerable social currency or
(ideally) it should be exchangeable for real goods and services.

People bet differently when they have something real at stake - ever tried to
play poker with no money behind the chips? It's a completely different (and
completely boring) game. It's the reason the market instantly knew who was
responsible for the Challenger crash (when it took Richard Feynman and the
rest of genius scientists on the panel months to even have a plausible idea).

~~~
gamache
I'd love to see a reference on that Challenger remark -- seriously.

~~~
smanek
[http://www.scribd.com/doc/8142600/the-Stock-Market-
Reaction-...](http://www.scribd.com/doc/8142600/the-Stock-Market-Reaction-to-
the-Challenger-Crash)

Sorry for the scribd link. It's the first copy I found that wasn't behind a
pay wall.

The gist is that within a few hours of the Challenger crash the stock for the
company responsible for the component that failed went down far more than the
stocks for other companies involved in building challenger. Interestingly, the
equity value the stock lost (~200 Million USD) is almost exactly the same as
the estimated total loss in profit (fines, lawsuits, updates to safety
standards, lost contracts, etc).

~~~
ryanwaggoner
Hmmm...so traders had a 1 in 4 shot of picking the correct company and they
happened to do so? I remain unconvinced. I also remain unconvinced that this
data is useful. It's not like we should let trader behavior influence safety
investigations, right?

~~~
smanek
Go read "The Wisdom of Crowds" - it is chock full of examples like this.

No one is saying we should use markets for safety investigations - we are
saying that we should use it predicatively.

------
smanek
Reminds me of overcoming bias' stuff. For example:
[http://www.overcomingbias.com/2009/02/who-are-macro-
experts....](http://www.overcomingbias.com/2009/02/who-are-macro-experts.html)

We really need this sort of thing to take off ...

Unfortunately, the public hates them. When DARPA created a prediction market
(PAM) there was so much public outrage that the head of the agency was fired.

~~~
adamhowell
The public hates prediction markets that try and predict terrorist attacks.
Ones that make Scoble look like an ass are something else entirely.

------
Eliezer
This is extraordinarily important work to see done. Whether or not anyone is
bothering to keep track of successful predictions affects the whole quality of
predictions that will be publicly made, which affects the sanity of the whole
society. Mod parent up and link to this project on your blog.

------
msluyter
Are there any categories besides right and wrong? I'd imagine that the success
of some predictions is open to interpretation and/or ambiguous enough to defy
easy categorization. A perfect example would be "Podcasting will be a factor
in the 2006 congressional elections in the US." How much counts as "a factor?"
Any amount > 0? How can we be sure one way or the other about this one,
without some extensive analysis?

At the very least, I think some explanation of how the determination is made
should be required.

~~~
Semiapies
The "WRONG" ones have some commentary:

<http://wrongtomorrow.com/predictions/26>

~~~
msluyter
Not all of them. For example:

<http://wrongtomorrow.com/predictions/47>

~~~
Semiapies
Good eye. (Not that I can think much of any counter-example, there's still no
explanation.)

------
sam_in_nyc
Very, very cool. Kind of like the Obama promise-keeper website that I thought
was cool, but then promptly forgot about. I'd like to suggest a couple of
features, without which I probably will promptly forget this site as well:

    
    
      1)  E-mail or alert me when a prediction is verified correct, or expires, whichever comes first.
      2)  Mirror the URL at which the prediction is made.
    

Great work launching the site, and best of luck to you.

------
snewe
If you peruse the Bloomberg wire and keep track of stock analyst predictions,
this site could grow into a "forecast analyzer" of stock "experts." Of course,
someone needs to login and edit whether the prediction was correct.

~~~
HSO
Actually, I remember reading sth about a hedge fund that employed just that
approach. They had every analyst or broker send in their predictions or
collected them from somewhere and kept track of who did well and who did not.
The better someone did (the article did not say which measure the fund used),
the more business the fund would send their way (in terms of order flow),
betting on their recommendations to go long or short. Presumably, their
portfolio would have been optimized to account for forecast accuracy, expected
gains, and some measure of risk. The difficult question here would be which
measure to take as "forecast accuracy", and that also touches on the site. How
wrong is wrong, and how right is right? You would need some measure of
dispersion to judge whether someone was right. If there is a lot of noise in
some variable (say the oil price), then basically anyone will have been right
sometime when the forecast horizon is long enough. On the other hand, you
couldn't really say someone was "wrong" in his forecast if he said X and, by
the measure of dispersion used, the actual number lies within reasonable
bounds around X. Another question I already mailed the site's creator is how
the variables are actually defined and tracked. Yet another question is
whether and how the forecasters can adjust their forecasts. Obviously, you
should be able to update your 5y forecast if tomorrow something happens that
topples your assumptions. It's totally legit to do that, and I would argue
such a mechanism is the real point of the otherwise pointless forecasting
game.

EDIT: There is one more question I find interesting. The measure by which
forecast quality is judged should also somehow account for time dimension.
Concretely, if the process you are tracking conforms 9 times out of 10 with
your forecast path and then "explodes" or "collapses", are you good or bad?
How far does it have to collapse or explode that you are "bad", how is the
situation if the granularity is finer and you can observe 100 times instead of
10 (so the process is 98 or 99 times fine and then explodes)?

~~~
joshu
One hedge fund? This is a fairly standard strategy. Analysts publish their
predictions. Services like First Call aggregate them, etc.

~~~
HSO
I didn't say "one" hedge fund pursues the strategy. I said I read about one.
Also, aggregators do the exact opposite of what this hedge fund was doing:
averaging. Whereas the point would be to measure their predictive accuracy
individually.

~~~
joshu
Actually, the data providers just give you all the data, not average anything.

I've actually worked in that business. Have you?

------
acgourley
I, and I think lots of others, all have had this on a todo list. I think the
reason they never seem to get built or take off is that there is such a lag
between their establishment and their payoff.

I assume the trick is to have some dedicated submitters for a year, and then
to really push the interesting right/wrong answers into the right social news
sites. That should get them over the hump. I hope this site manages to do
that. Bookmarked.

------
ivankirigin
Related: <http://www.longbets.org/>

------
cjlars
I love this in concept, but somehow feel like it'd be better established as
some sort of data blog. Perhaps you could find some aspiring young writer out
there willing to work for equity?

------
davidw
Seems to be down at the moment... and now it's ok. Weird.

I agree completely with sethg... I had this idea floating around, and it's
nice to see it. Do make it easier to see some statistics, and see wrong/right
predictions, as that's a bit more interesting for the casual browser.

BTW, it's _really slow_ again.

------
vaksel
For some reason the #s in authors, don't match up with the actual author
profile.

------
banned_man
Of all those, I tend to think T. Boone Pickens is the one who's most likely to
turn out right.

The question is, if someone turns out to be right, does he get removed from
the list?

