
Surprisingly popular - tosh
https://en.wikipedia.org/wiki/Surprisingly_popular
======
Noumenon72
The capital is not Philadelphia, it's Harrisburg. So the idea must be that
most people think they are right and others will agree with them, but there
are people who get it right and know that others get it wrong, and they are
responsible for the difference in scores.

~~~
cecilpl2
There are four groups of people:

A - "Philadelphia is the capital, and others will agree."

B - "Philadelphia is the capital, but most others won't know that".

C - "Harrisburg is the capital, and others will agree."

D - "Harrisburg is the capital, but most others won't know that."

This technique eliminates groups A and C from consideration, and measures the
difference in size between groups B and D.

Both groups B and D think they know something other people don't, but B is
wrong and D is right.

In cases where people feel like they have "inside" knowledge, generally
speaking it's because they are correct and knowledgeable (group D), not
because they are misled (group B).

You often arrive in group D by virtue of having been in group A and then
learning the actual truth of the matter.

You can arrive in group B by falling prey to a conspiracy theory, which makes
this technique perhaps invalid in such cases? I wouldn't be surprised if the
question "Do vaccines cause autism?" had a surprisingly popular answer of
"Yes".

~~~
news_hacker
This is a great summary / reduction of the article. I appreciated how you
broke it down into groups and made the example more concrete.

~~~
lonelappde
It's not a summary/reduction, it's a a better article with more information.

~~~
cecilpl2
Wow, thank you!! :)

------
simias
The way I understand it it's effectively it's a good way to identify trick
questions and the people who correctly identify the "trick" (by saying that
other people would fall for it and reply differently) are more likely to be
right overall and it's reflected in the percentage difference.

Wouldn't you get the same result simply by saying "let's just consider the
answers from people who said others would respond differently because they're
more likely to be right"? That's more intuitive to me and I believe it's
effectively the same operation.

~~~
brownbat
Yeah, it's a bit like "second order" wisdom of the crowds.

First order: "How many jelly beans are in this jar?"

Second order: "What percentage of people will get the wrong answer to this
question about Pennsylvania's capital?"

There's also the interplay with the Keynesian beauty contest:
[https://en.wikipedia.org/wiki/Keynesian_beauty_contest](https://en.wikipedia.org/wiki/Keynesian_beauty_contest)

In some sense the stock market is an infinite regress of these "what will
other people think about value..." questions.

~~~
whatshisface
Shares do actually have a fundamental value, based on their lifetime
dividends, buybacks and liquidation. So unlike, say, Bitcoin or the Dollar,
there is something to be gained from knowing the zeroth-order reality.

~~~
brankoB
Disagree. Think about what makes a share liquid: lots of potential buyers
(i.e. people perceiving it as a safe investment).

~~~
whatshisface
If you were offering a long period of healthy dividends for a good price, I
wouldn't care if I could sell it.

~~~
brankoB
Sure but where does the amount of dividends you're being given come from? All
i'm saying is there's no objective "baseline", it's all perceived value no
matter how you look at it.

------
carbocation
It's an interesting heuristic, but without an exposition it's hard to know if
there is any reason to believe it. Has this technique been assessed across
various domains? Does it work better for political facts than for scientific
facts? Where does it fail? It's a provocative article and a cool thought, but
it seems more like a hypothesis.

(In particular, I'm referring to the assertion at the end of the article: "
_Because of the relatively high margin of 10%, there can be high confidence
that the correct answer is No._ ")

~~~
tofof
One of the wiki's sources might have been a better linked article.

The MIT summary [1] notes _" The researchers first derived their result
mathematically, then assessed how it works in practice, through surveys
spanning a range of subjects, including U.S. state capitols, general
knowledge, medical diagnoses by dermatologists, and art auction estimates."_
Across all those areas, this technique had error rates about 20% lower than
other competing techniques. Those techniques included simple majority vote to
two different kinds of confidence-weighted scoring.

The paper earned a prestigious publication in _Nature_.

1: [http://news.mit.edu/2017/algorithm-better-wisdom-
crowds-0125](http://news.mit.edu/2017/algorithm-better-wisdom-crowds-0125)

~~~
carbocation
Awesome, thank you. And 'nacc found the link to the Nature article as well:
[https://news.ycombinator.com/item?id=20547787](https://news.ycombinator.com/item?id=20547787)

~~~
bscphil
One interesting thing about the paper is that it seems that the Wikipedia
article incorrectly describes the procedure: respondents were not asked to
guess whether the majority would agree with their position. They were asked to
guess what _per cent_ of other respondents would agree. I think that's a
pretty severe difference in the method.

------
nacc
Digging down a bit this method is published in Nature!
[https://www.nature.com/articles/nature21054](https://www.nature.com/articles/nature21054)

------
inlineint
Looks like an idea for a semi-supervised ensemble method for machine learning:

Prepare two equally sized ensembles of classifiers, let's call them A and B.

1\. Train each classifier in ensemble A on labelled data to predict does a
picture contains a cat.

2\. Take some other unlabelled dataset and collect answers from classifiers
from A for each picture from this dataset.

3\. Train each classifier in ensemble B to predict average answer of
classifiers from A for each picture from the unlabelled dataset.

Then for a picture from the test dataset it would be possible to get answers
from ensemble A and from ensemble B and calculate what would be the
surprisingly popular answer.

~~~
londons_explore
Please do this.

------
cygx
Let's look at another toy example, using made-up numbers:

 _Q: Is the earth flat?_

    
    
        Yes: 10%
        No:  90%
    

_Q: What do you think most people will respond to that question?_

    
    
        Yes:  3%
        No:  97%
    

_Analysis:_

    
    
        Yes: 10% -  3% =  7%
        No:  90% - 97% = -7%
    

_Conclusion:_

The earth is flat, I guess ;)

~~~
identity0
A more realistic example of that would be:

Q. Is the earth flat?

    
    
        Yes: 0.1%
        No: 99.9%
    

Q: What do you think most people will respond to that question?

    
    
        Yes: 10%
        No: 90%
    

Analysis: Yes: 0.1% - 10% = -9.9% No: 99.9% - 90% = 9.9%

Conclusion: the earth is not flat.

Not sure where you're getting the "10% believe the earth is flat" from, but if
that's just your guess then it's pretty reasonable for people to say they
think 10% of people will respond yes to the question.

~~~
cygx
If there are only 0.1% of flat earthers, why would 10% of people think it to
be the most popular answer? That does not seem realistic at all.

Or did you interpret the question as suggested by bena in his sibling answer,
ie as an estimate for the answer distribution (that would then have to be
averaged) instead of a choice of the most popular answer (as was done in the
wiki article)?

~~~
Grue3
>That does not seem realistic at all.

That does seem realistic considering how often the topic pf flat-earthers
comes up in discussions on social media, despite it being an extremely niche
view.

~~~
cygx
OP suggested that 10% of people think there are more than 50% of flat-earthers
when the real number is 0.1%. Phrased another way, 10% of people expect half
the people they meet will be flat-earthers when it's 1 in 1000.

Doesn't seem realistic to me.

------
mathgeek
I’m not sure what to think of a Wikipedia article without any sections on
Uses, and the only Example being unreferenced.

------
espeed
Notice the relationships between Dempster–Shafer theory -- AKA the theory of
belief functions / evidence theory -- for reasoning with uncertainty...

[https://en.wikipedia.org/wiki/Dempster-
Shafer_theory](https://en.wikipedia.org/wiki/Dempster-Shafer_theory)

------
jmpman
How does this play out in the stock market?

~~~
beefman
In the cases of Brexit and the 2016 US Presidential election, betting markets
reported a majority of money on Remain and Clinton with a majority of bets on
Leave and Trump.

Perhaps the number of bets is analogous to what people believe, while total
money bet is analogous to what people think others believe. If so, the
_surprisingly popular_ answer is the one with the smallest average bet.

The obvious analog in the stock market is average order size, but this isn't a
generally-available statistic, and it's not clear to me whether it would be
meaningful.

Volume (shares traded per time unit) is closely-watched, however, and several
technical analysis "indicators" compare price changes to volume.[1][2][3][4]

High liquidity is a state of relatively small price changes per unit of
trading volume. It generally means traders are not paying large
premiums/discounts to previous trades. In a way, this corresponds to a smaller
bets on future returns, and thus, to _surprisingly popular_ returns...

[1]
[https://en.wikipedia.org/wiki/Ease_of_movement](https://en.wikipedia.org/wiki/Ease_of_movement)

[2]
[https://en.wikipedia.org/wiki/Force_index](https://en.wikipedia.org/wiki/Force_index)

[3] [https://en.wikipedia.org/wiki/Volume-
price_trend](https://en.wikipedia.org/wiki/Volume-price_trend)

[4] There are similar concepts in modern portfolio theorgy, such as Amihud
illiquidity.

------
ttimebomb
I see people mentioning whenever the answer is right or not, but that's
actually irrelevant. The heuristic only measures how surprising the popular
answer is to the population.

~~~
jobigoud
It's not presented as irrelevant in the article. The article presents it as "a
wisdom of the crowd _technique_ ", and ends with "there can be high confidence
that the correct answer is No".

So it is assumed that people using this technique could be inferring the
correct answer by finding the surprisingly popular answer. Not just a
observation about what the population thinks.

------
powrtoch
Using this on subjective questions (e.g. “Is Cymbeline one of Shakespeare’s
best plays?”) would help you find... what? Underrated things?

~~~
derefr
I'm not sure the algorithm would emit any answer in the case where everyone
has an equal level of objective foundation for their subjective belief.

But it would probably help in cases where popular opinion is entirely
misinformed about the subjective question, not having _any_ basis other than
(already misinformed) hearsay on which to form their own subjective opinion.

So, for example, if there was a musician who had an _absolutely terrible_ song
that somehow became the song they were best known for (being a "one-hit
wonder" whose song wasn't really a "hit"), the public might believe that that
song is their best song, since it's the only song of theirs the public has
ever heard of. Experts (i.e. people who have heard more than the one song of
theirs), on the other hand, would tend to agree that it's certainly not their
best song.

(Given that example, I'm inclined to suggest that you could use this algorithm
to determine when people are being judged overly-harshly for things, e.g.
whether to ban someone from a website just because they've received a lot of
reports about that person's behavior.)

------
kleton
Have any political pollsters ever done this sort of question format?

~~~
jobigoud
You mean like this:

1\. Who will you vote for?

2\. Who do you think will win?

I don't know if we can learn anything from the surprisingly popular answer
here, but now I wonder if question 2 would be a better predictor of the
election result. But it will be tainted by previous polls and news.

------
randyrand
Forgetting one thing, surpassingly popular _to who_? You need to answer the
"to who" part.

------
awillen
The wisdom of crowds is a truly fascinating thing... the fact that people
don't really need to know what they're talking about as long as there's enough
of them really blows my mind.

~~~
jobigoud
The concept in the article is pretty much a counter example of that. It says
that if there is a core part of the crowd that are expert on the topic at
hand, and they answer differently from the crowd wisdom, there is a way to
find out about it using a meta-question about the crowd wisdom itself.

------
busfahrer
I wonder if this could be viewed as an application of
[https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect](https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect)

~~~
theelous3
Nope. That is overconfidence of the ignorant, and this is dealing with
trusting the high confidence of the correct.

It doesn't really matter here that people answered incorrectly.

~~~
jobigoud
I don't think so. It's not saying anything about how highly confident the
correct are. It's about how they know that most people are incorrect. It's
about misconceptions.

There is this core part of the unpopular-but-correct answerers that knows that
the incorrect answer is popular. The larger this core part is, the higher the
"score" of the "surprisingly popular" answer is. It's about finding something
that is "mistakenly popular".

~~~
theelous3
Yeah but this is in the context of not knowing the correct answer.

Saying that you believe the popular option is different to yours is a display
of high confidence in your own answer, and low confidence in other people's
knowledge.

Dunning kruger only deals with highly confident wrong people, which isn't the
interesting group in our surprisingly popular measurements.

Bringing dunning kruger in to it would make more sense if there was an "im not
100% confident" option, but instead it was an absolute dichotomy.

It doesn't really mean that all of the people answering incorrectly are
completely confident and therefore are dunning krugering. Someone who is not
overconfident at 60% confidence adds the same "yes" data point that someone
who is 100% confident and wrong does.

------
lonelappde
In October 2016:

* Do you think Trump will win the US Presidential election?

* Do most voters think Trump will win?

------
bengarvey
Philadelphia should be the capital of Pennsylvania

~~~
ncmncm
Pittsburg/Philadelphia should be the capital.

~~~
JohnBooty
There is already widespread belief, in the rest of the state, that the
Philadelphia area (as the largest concentration of population and wealth) has
an unfairly outsized influence over Harrisburg.

The result of moving the capital to Philadelphia would surely strain the
legitimacy of the state government even more.

Moving it to Pittsburgh would accomplish the same exact thing except it would
be arguably even less just.

There are any number of reasons why keeping it in Harrisburg is imperfect but
it's probably the least bad solution.

~~~
ncmncm
No, I mean really Pittsburg/Philadelphia, like Minneapolis/St. Paul,
Seattle/Tacoma, or Raleigh/Durham.

Yes, the distance between respective city centers is farther than those
others, but that makes it more inclusive.

