
The Experts Have Spoken - colinprince
http://www.theawl.com/2016/05/the-experts-speak-its-good-to-be-wrong
======
YuriNiyazov
It seems silly to expect that the experts never get anything wrong. It even
seems silly to expect that they don't get any of the big questions wrong. The
real question is whether the ratio (or some other function) of wrong
predictions to right predictions, suitably weighted by their strength and
complexity, is decreasing.

~~~
jimhefferon
My mom is sick, so I've been hanging around her house to sit with her. She has
some 5-10 year old magazines, including some of one of my favorites _The New
Yorker_. I like the intelligence of the commentators and I feel that they
often put into words things that I vaguely feel.

I was stuck reading them (or else: Fox News and the Catholic channel). Boy,
they get just about every prediction wrong. It was shocking.

Makes me doubt we do much better than people in the past.

~~~
nl
We _can_ do better, but it isn't expertise that is the difference.

It's worth taking a look at the Good Judgement Project and their success at
making forecasts.

[http://www.nesta.org.uk/sites/default/files/1502_working_pap...](http://www.nesta.org.uk/sites/default/files/1502_working_paper_-
_prediction.pdf) is a good overview.

------
nl
So, I work in this area and follow a lot of the work on ways to make
predictions (Superforecasters, etc)

One thing I see time and time again is that what we think of as predictions
are really aggregates of large number of assumptions and projections of how
these assumptions interact with each other and the world.

"Experts" generally are very good at estimating the likelihood of some of
these assumptions changing. BUT it is only a limited number of these things
they are experts in, and that make them no better in other areas.

~~~
jamesrcole
> "Experts" generally are very good at estimating the likelihood of some of
> these assumptions changing.

Is that really true? I'm dubious, and I'm curious whether there's ever been
any empirical studies.

Also, I'm not entirely clear what you mean by "some". Do you mean "some kinds
of assumptions" or "some number of assumptions?

~~~
nl
_Is that really true?_

I think so. For example, in fields relying on scientific progress, a scientist
working in the field is generally fairly accurate in predicting the likely
progress in their exact field over the next couple of years. For example in
materials science the researchers are relatively good at estimating what
progress will be made.

There was an IARPA program around estimating scientific progress that may have
done some empirical studies.

 _Also, I 'm not entirely clear what you mean by "some". Do you mean "some
kinds of assumptions" or "some number of assumptions?_

I mean they are good at estimating the likelihood of specific estimates
changing.

------
qwtel
There's a difference between binary and open-ended predictions. "Will the
iPhone be a commercial success?" is easier than "How many iPhones will be
sold?". It's the later that matters when you are directly involved. This can
lead to seemingly paradox statements like "The iPhone will most likely fail,
but assuming that it doesn't, it will be wildly successful".

Philip Tetlock and some controversial author have written a paper about it:
[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2284964](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2284964)

------
maxxxxx
Why do all these "experts" always try to predict the future? I would much
prefer if they actually gave some background about the present and the past so
we can make better decisions.

