
A Quick Puzzle to Test Your Problem Solving - granfalloon
http://www.nytimes.com/interactive/2015/07/03/upshot/a-quick-puzzle-to-test-your-problem-solving.html
======
ddlatham
I'm curious to see more about the distribution of questions and answers people
had, and how the HN population may differ from the NYT's. There will certainly
be self selection bias here, but if you're willing to share how you did with
others, please enter it here:
[https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCV...](https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCVLC8TbMxsPApGcSGg/viewform)

The result summary is visible here:
[https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCV...](https://docs.google.com/forms/d/17e5BIL0lH8OHsGj89Zdtdl8GeCVLC8TbMxsPApGcSGg/viewanalytics)

The raw answers are visible here:
[https://docs.google.com/spreadsheets/d/1ZxR2_eOUtNLXwgKfLO1J...](https://docs.google.com/spreadsheets/d/1ZxR2_eOUtNLXwgKfLO1JWhiXx-
UQOxsYlKaBFe7f3m0/)

~~~
veli_joza
People familiar with unit testing and test driven development will feel at
home with this kind of puzzle. That doesn't mean that they will be less biased
in social/political decisions, it just means that this test will fail to prove
a point.

~~~
diminishedprime
That's exactly what I was thinking. I (sometimes) follow TDD, and I applied it
to this problem. I made sure to include negatives, 0, positives, and include
primes here or there to help avoid issues with multiplication/exponentiation.
After a few of these, I felt pretty confident that the rule was simple.

~~~
joezydeco
Did you try floating point numbers? I didn't see anything in the text that
said integers only.

~~~
tptacek
I tried floating point numbers. Also, at 28 decimal places, the test breaks;
it's not arbitrary precision. So, technically, the answer isn't simply "any
ascending sequence of numbers".

~~~
SiVal
I actually avoided going that far to avoid getting bad data. I was trying to
answer the question, "What does the experimenter THINK his rule is?" rather
than what will the computer do. Since the computer can't be infinite, it will
inevitably fail with overflow, underflow, and such.

I was relieved, in fact, when it worked with negatives and floats in a "safe"
range.

I also tested with 1,1,2 and 1,2,2 to make sure that the required increase
applied to ALL of the values, not just a specific pair.

~~~
mjevans
I too tried to test if it was only one pair that was significant. However I
grew impatient and didn't try to come up with more tests when I thought I had
a sufficient answer to explain my most vexing observation (negative, positive,
positive out of combinations involving negative numbers).

The observation to brainstorm for ways of proving that a statement is in fact
wrong, and exhausting them, is such an eloquent way of wording the hunt for a
negative.

------
tmd
It responds "No" to (10000000000000000, 10000000000000001, 10000000000000002)
so the rule is not so simple after all :)

~~~
kittenfluff
Responds "Yes" to

    
    
      9007199254740990, 9007199254740991, 9007199254740992
    

but "No" to

    
    
      9007199254740991, 9007199254740992, 9007199254740993
    

Presumably this is due to how Javascript handles integers, i.e. it uses the
integer part of a float64, to wit

    
    
      > parseInt('9007199254740992')
      9007199254740992
      > parseInt('9007199254740993')
      9007199254740992
    

Edit: I think this is the code that actually reads the numbers the user
enters, see [0]

    
    
      function l(){
          var a=h.exec(m[1]),f=null,g=null,n=null;
          return a&&(null!==a[1]&&a[1]&&(f=parseInt(a[1],10)),
              null!==a[2]&&a[2]&&(g=parseInt(a[2],10)),
              null!==a[3]&&a[3]&&(n=parseInt(a[3],10))),
          new e(f,g,n)
      }
    

Edit(2): Actually, I'm not so sure that's the correct code at all. They NYT
game is capable of parsing floats correctly (e.g. it accepts 1.1, 1.2, 1.3 as
a "Yes") so it's not just using parseInt.

[0]
[http://a1.nyt.com/assets/interactive/20150612-151638/js/foun...](http://a1.nyt.com/assets/interactive/20150612-151638/js/foundation/lib/framework.js)

~~~
janka102
The actual code seems to be from here [0]

on line 588 is the comparison

    
    
        var rightWrong = (inputData[0] < inputData[1]) & (inputData[1] < inputData[2]) ? right : wrong;
    

With a variable declaration on line 545 being

    
    
        var inputData = [NaN, NaN, NaN],
            revealed = false,
            right = "<p class = 'g-answer g-yes'>Yes!</p>",
            wrong = "<p class = 'g-answer g-no'>No.</p>";
    

And `inputData` is changed on text input on line 662

    
    
        $("#g-input input").each(function(i) {
            var val = $(this).val();
            inputData[i] = $.isNumeric(val) ? Number(val) : NaN;
        });
    

It uses the `Number()` function to convert from the input text to an actual
number, so it can convert any number format defined by ES5[1] or ES6[2]. So in
ES6 you can use binary (0b, 0B) and octal (0o, 0O) formatting along with
exponential (1e-2) and hex (0x, 0X). Binary and octal works for me currently
on Chrome 43 OS X.

[0]
[http://graphics8.nytimes.com/newsgraphics/2015/06/16/puzzle/...](http://graphics8.nytimes.com/newsgraphics/2015/06/16/puzzle/3c265b64cd66937d132444db8e9f7edb6a231f29/build.js)

[1] [http://www.ecma-
international.org/ecma-262/5.1/#sec-9.3.1](http://www.ecma-
international.org/ecma-262/5.1/#sec-9.3.1)

[2] [http://www.ecma-
international.org/ecma-262/6.0/#sec-7.1.3.1](http://www.ecma-
international.org/ecma-262/6.0/#sec-7.1.3.1)

------
abecedarius
I ran this experiment for a while (code at
[https://github.com/darius/wason](https://github.com/darius/wason), derived
from
[http://lesswrong.com/lw/g2/positive_bias_test_c_program/](http://lesswrong.com/lw/g2/positive_bias_test_c_program/)).

In my logs most people seem to have gotten it right, though presumably that's
because it was linked from LessWrong.

For an actually-fun game like this, see
[https://en.wikipedia.org/wiki/Zendo_%28game%29](https://en.wikipedia.org/wiki/Zendo_%28game%29).

~~~
TeMPOraL
> _though presumably that 's because it was linked from LessWrong_

No surprise there, given that this experiment is discussed in Sequences, the
link to the post present in the article you link to :).

------
Kenji
"We’ve chosen a rule that some sequences of three numbers obey — and some do
not. Your job is to guess what the rule is."

My mistake was to assume that choosing the first number uniquely defines the
next ones in the sequence. Since, you know, like all the sequence puzzles I've
seen before worked like that, and I didn't read it rigorously enough. Oh, by
the way, the doubling thing is wrong if you use negative numbers (wrong as in
it gives false positives, instead of just false negatives). But the problem
definition doesn't even tell what set of numbers we're operating on.

Finding the rule the sequences obey is impossible since it could be that all
cases follow a simple rule except for one triplet which you're unlikely to
find. It's trivially easy to fool the user into finding a wrong rule.

~~~
mikehawkins
Same here - I was also guilty of feeling a bit clever and avoiding the obvious
"oh, the numbers double every time" answer. So, when I found out that if the
spelling of each number was one letter longer (4 = four letters, 8 = five
letters, 11 = six) I got smug, and didn't bother testing further.

Good article - and a humbling experience. :)

~~~
TylerE
I went for the slightly more clever "series of powers" after 3/9/27 validated.

~~~
dllthomas
That was a hypothesis I considered, as was x,2x,4x. Invalidated both of those,
and ultimately drew the intended conclusion (which is not quite "correct",
given javascript numeric precision issues).

------
m4r71n
Derek Muller of Veritasium has done this test over a year ago on people in one
of his videos:
[https://www.youtube.com/watch?v=vKA4w2O61Xo](https://www.youtube.com/watch?v=vKA4w2O61Xo)

I've given the test to various people since then and never once came across
someone who'd guess it right away, or within a short period of time. The
breaking time came usually after a few minutes when they gave up and started
throwing out random numbers that coincidentally did not meet the rule. Once
you hit the first "No", it took a very short time to figure out the rule for
almost everyone.

------
aaronbrethorst
More or less related: when I'm looking for constructive criticism from
someone, I'll ask them "what do you dislike about this?" or "what's wrong with
this?" instead of "what do you think?"

I tend to get much more interesting and useful feedback this way.

------
cabirum
function judgeSentence(sentence, numNo)

var probablyWrong = ["doubl", "expon", "multipl", "^", " __" , "power", "two",
"2", "twice", "as big", "nth", "rais"];

var seemsRight = ["larger", "increas", "greater", "small", "less", "big", ">",
"<", "go up", "ascending"], weaselWords = ['but ', 'not ', 'odd'];

Been expecting something more interesting than that

~~~
TeMPOraL
It's surprising how far you can go by such heuristics. I run an IRC bot that
uses matches like this (I'm working on a proper solution right now though) to
parse natural language queries and I managed to trick few people into thinking
they're talking with human. As long as it's ok for 90% of most common cases,
people often won't notice.

~~~
tomerico
Which is another case of confirmation bias

------
MarkMc
The comments here suggest people are missing the full significance of this
problem. It's not just a cute number puzzle - it demonstrates a profound human
weakness that has a deep impact in everything we do.

1\. People that think having a gun in the house makes it safer will not try to
design an experiment designed to demonstrate the opposite.

2\. People who think organic food is better for you than regular food will not
try to look for evidence that the two types of foods are equally healthy.

3\. An Israeli who believes the area where he lives was uninhabited before
1948 is not going to think about what kind of evidence would contradict that
belief.

I'm not saying the views above are incorrect. It's just that we are all guilty
of falling in love with our beliefs when they should be mere acquaintances.
Hence the quote, "People don't change their minds. They die, and are replaced
by people with different opinions." [1]

[1] [http://www.paulgraham.com/quo.html](http://www.paulgraham.com/quo.html)

------
nothrabannosir
My mathematical logic is rusty, but if I recall correctly, Gödel's
incompleteness theorem basically states that it is impossible to solve this
kind of question. No matter how many tests you run, there will always be an
uncertainty.

An incredibly stupid example is that the rule could be "yes for strictly
increasing, OR if one of the numbers is -18273192783127897981." You'll never
know.

I understand this is contrived, especially when the test subject doesn't know.
But if you do realize this while doing it, it makes the test a little
frustrating..

EDIT: I see people are making a connection with unit testing, and the irony is
poetic. This is precisely the problem Dijkstra was talking about when he said
that "Testing shows the presence, not the absence of bugs."

~~~
scarmig
Remotely related: I've been interested for awhile in how the same initial
terms of a sequence could possibly be generated by multiple rules.

For example, you might have

2,3...

And the rest of the sequence might look like either

2,3,4,5,6...

or

2,3,5,8,13...

or

2,3,5,7,11...

or even

2,3,5,10,20...

Clearly, on some level those sequences are all much less complicated than one
defined as "The first term is 2, the second term is 3, the third term is
919243, the fourth term is -1234..."

It's unclear to me how one might rank them in complexity, though. The maximum
amount of memory necessary to get an arbitrary nth term? The number of
operations necessary to get to the next term?

Another interesting question to me: if there is an ordering of ways to
generate a sequence of numbers, given the first couple terms of a sequence of
numbers, what are the simplest N ways to generate the full sequence?

~~~
drewcrawford
Fun fact: Mathemetica has a method FindSequenceFunction which does exactly
what you describe. In my experience, however, it generally requires at least 4
terms.

I'm not totally sure how the math behind it works (maybe it's similar to
Eureqa?) but the results speak for themselves and are rather incredible.

For example, if I run FindSequenceFunction on this input:

    
    
        {0, 1, 3, 8, 19, 43, 94, 201, 423, 880}
    

Which is the number of 0,1 sequences of length n that contain two adjacent 1s

Mathematica produces the result:

    
    
        1/10 (5 2^(1 + x) - 5 (1/2 - Sqrt[5]/2)^x + 
        3 Sqrt[5] (1/2 - Sqrt[5]/2)^x - 5 (1/2 + Sqrt[5]/2)^x - 
        3 Sqrt[5] (1/2 + Sqrt[5]/2)^x)
    

Which, astonishingly, is correct for all the values I've tried. So apparently
Mathematica understands more about this sequence than I do, and I know its
definition.

Another party trick is to use the input

    
    
        {-(1/6), 2/15, -(13/140), 23/315, -(83/1386), 305/6006, -(2269/
         51480), 4259/109395, -(16103/461890), 30616/969969}
    

Which is the integral x^n (1 - 2 x)^n for x from 0 to 1, for n = 0..<10\. Here
it seems 10 numbers are required. This yields the solution

    
    
        (2^(-2 - 3 x)
        x! (Sqrt[\[Pi]] (1 + x)! + 
        3 (-1)^x 2^(
        2 + 3 x) (1/2 (1 + 2 x))! Hypergeometric2F1[1, 3/2 + x, 
        2 + x, -8]))/((1/2 + x)! (1 + x)!)
    

Which as far as I can tell, is a closed-form solution (!) to the integral. A
solution it worked out to an integral it has never seen, but only the first 10
elements in the sequence.

So it's safe to say Mathematica knows a lot more about math than I do.

~~~
thisrod
_Which is the number of 0,1 sequences of length n that contain two adjacent
1s_

It sounds a lot simpler than that. There are n-1 places to put the adjacent
1s. For each of those, there are 2^(n-2) ways to complete the sequence with 0s
and 1s. So the answer should be (n-1)*2^(n-2).

Edit: it isn't quite that simple, I'm counting sequences with multiple pairs
of 1s multiple times.

By the way, Mathematica's formula looks a lot like the closed form for the
Fibonacci sequence.

~~~
thisrod
OK, got it.

Let a_n be the number of sequences of 0s and 1s of length n, that do not
contain a pair of 1s, and end in 0. Let b_n be similar, except that the
sequences end in 1. These satisfy the recurrences a_{n+1} = a_n + b_n, and
b_{n+1} = a_n. It follows that a_n satisfies the Fibonacci recurrence a_{n+1}
= a_n + a_{n-1}. Starting from a_1 = 1, and a_2 = 2, we have a_n = F_{n+1}.

The total number of sequences of length n is 2^n, so the number we want is 2^n
- a_n - b_n = 2^n - F_{n+1} - F_n = 2^n - F_{n+2}.

And this time it works :-). There might be a point about confirmation bias
here, in that the trick is to count sequences that _don't_ contain a pair of
1s.

One drawback of Mathematica is that it has a very poor idea of the formulae
that human readers will regard as simple.

------
phantarch
Funnily enough, I notice confirmation bias quite a bit in a D&D game I am
currently DM of. I'm playing with a group of friends who are big into video
games, and as a result they consistently seek resolutions to conflicts in D&D
by way of what they know from shooters: kill everything in sight. Yes, it's at
times a valid answer, but it's not the only one and it's certainly not the
most interesting one. The best way that I've seen the confirmation bias
dissipate from their thinking is to put them into situations where their bias
just doesn't help at all.

~~~
pizza
Maybe some positive/negative feedback built into the campaign could help; e.g.
for each act of benevolence/violence, add/subtract a 'karma' point from some
running total, and alter the gameplay as needed.

~~~
eru
Just don't tell them explicitly about it. (That'd be taking away all the fun.)

------
pbnjay
If anyone's curious about the raw numbers, I found the counts here:

[http://int.nyt.com/newsgraphics/2015/2015-06-26-rule-
guessin...](http://int.nyt.com/newsgraphics/2015/2015-06-26-rule-guessing-
game/numno.json)

    
    
        {"count": 27723, "numNo":     0, "share": 0.7716051100782098}
        {"count":  2921, "numNo":     1, "share": 0.08129922903504133}
        {"count":  1883, "numNo":     2, "share": 0.05240891758746417}
        {"count":  1285, "numNo":     3, "share": 0.035764980934621056}
        {"count":   880, "numNo":     4, "share": 0.024492749589468118}
        {"count":   525, "numNo":     5, "share": 0.014612151743716774}
        {"count":   288, "numNo":     6, "share": 0.008015808956553202}
        {"count":   156, "numNo":     7, "share": 0.004341896518132985}
        {"count":   108, "numNo":     8, "share": 0.0030059283587074506}
        {"count":    56, "numNo":     9, "share": 0.0015586295193297892}
        {"count":    45, "numNo":    10, "share": 0.001252470149461438}
        {"count":    11, "numNo":    11, "share": 0.00030615936986835146}
        {"count":    22, "numNo":    12, "share": 0.0006123187397367029}
        {"count":     8, "numNo":    13, "share": 0.00022266135990425562}
        {"count":     4, "numNo":    14, "share": 0.00011133067995212781}
        {"count":     5, "numNo":    15, "share": 0.00013916334994015976}
        {"count":     3, "numNo":    17, "share": 0.00008349800996409585}
        {"count":     1, "numNo":    18, "share": 0.000027832669988031953}
        {"count":     3, "numNo":    20, "share": 0.00008349800996409585}
        {"count":     1, "numNo":    21, "share": 0.000027832669988031953}
        {"count":     2, "numNo":  null, "share": 0.000055665339976063906}

~~~
buckbova
Interesting totals. I had 4 No's and would have had at least 5 had I
considered negative digits.

------
binarymax
The puzzle is not nearly as interesting as the code being able to understand
my answer!

My answer: _The numbers increase from left to right_

Application response: _As you seem to have guessed, the answer was extremely
basic_

~~~
Retr0spectrum
I said "The numbers are in ascending order", which it didn't seem to
recognise.

~~~
matchu
"ascending" is one of the words it checks for. Might've typoed it? :/

~~~
aylons
I tried, coincidentally, the exact same phrase as he did, with no effect.

------
stygiansonic
Neat - as others have pointed out, I feel that being familiar with unit
testing would help in this situation. Having negative test cases is just as
important, if not moreso, than having "happy path" tests.

~~~
david_shaw
That's a great point. Perhaps we should show something like this to new
developers who don't understand the value.

------
shmageggy
This phenomenon has had a profound effect on the history and philosophy of
science. There have been entire schools of thought based on verification of
hypotheses, and entire movements based on refuting those schools. The most
effective strategy in this puzzle(and the one that is unintuitive for many) is
to systematically generate alternative hypotheses and falsify them. Karl
Popper claimed that this method is actually at the core of how we gain
scientific knowledge, and his brand of philosophy of science is the most
popular and arguably the most successful today.

------
noreasonw
I think that the "quick" adjective in the title is purposeful misleading. You
are supposed to learn quickly the most general rule, but that is not so easy
because there are many possible rules that could fit such a pattern. It seems
that you should be rewarded for solving the puzzle quickly and then you fall
in the trap. I propose to change the title to "A puzzle to test your
Generalization Abilities", and state clearly that you should try to find the
most general rule that satisfies all patterns you can think of. In that case,
I would expect the conclusion and results of the experiment to be completely
different. So to summarize: the so "quick" adjective in the title has a very
strong anchor effect.

Edit: changed for grammar and to express more clearly what I think.

~~~
Retra
Maybe it should be "a puzzle many people already know the answer to" in which
case the conclusion and results are already obviously biased.

I was able to solve the puzzle without testing any numbers at all. Which
really skews the relevance of "only nine percent of people saw three 'no's
before answering."

~~~
crgt
You mean you correctly guessed without testing at all - and got lucky. I'm not
sure this is the same thing as 'solving' it.

~~~
Retra
I "got lucky" in the sense that my experiences have primed me to recognize
that particular kind of question. If the article had actually modified the
question framing at all, rather than just copying the existing one, it maybe
wouldn't have worked that way.

It's the same reaction I have to the Monte Hall problem: I don't have to think
or be clever to get the right answer. So naturally you couldn't effectively
teach me anything by simply posing the question, and any information you
collect by doing so won't accurately reflect what I learn or how well I think.
You'd just be testing topical familiarity.

~~~
crgt
The question could have read the exact same way, with a different final rule
as the answer. Without testing at all, you wouldn't/couldn't know that...and
so you didn't solve it..you just guessed...

~~~
Retra
Everyone guessed it. I guessed it and got it right. That's what 'solving it'
means.

------
damoncali
Cool. The funny thing is I inserted a constraint of my own invention without
even realizing it: "Use the fewest number of examples possible." Of course,
this meant failing miserably, and was nowhere in the problem statement.

Perhaps that's an additional factor - not exactly confirmation bias, but not
unrelated.

~~~
Nimitz14
Eliminating assumptions baby.

[https://www.youtube.com/watch?v=9-9VLVkm8R4](https://www.youtube.com/watch?v=9-9VLVkm8R4)

------
rtkwe
Wonder how many people here immediately knew what the rule was?

This is the standard example of to right way to test a hypothesis/theory and
the power of Confirmation Bias, testing sequences that are invalid under the
theory instead of testing what you think is correct.

~~~
rockdoe
I guessed that immediately (but still verified it). It was obvious it was
going to be a trick question, so...

------
FilterJoe
While the HN crowd mostly gets this right when framed as a math puzzle, my
guess is that confirmation bias is alive and well in high tech just like in
any other field. One example:

Young 20s entrepreneur vs. early 50s entrepreneur. Without knowing anything
about either person, which startup is more likely to succeed? Even if you have
the business plans for both, and you meet both - which one are you going to be
more skeptical about as you evaluate which one of them gets funding?

~~~
eru
I'd trust the older guy.

------
harryh
Reminds me of the game mastermind which I loved playing as a kid.

Here's a javscript version: [http://www.archimedes-
lab.org/mastermind.html](http://www.archimedes-lab.org/mastermind.html)

------
moo
I've heard of intelligence failures to explain the Iraq war, now this author
says it was confirmation bias which caused a completely erroneous
justification for the invasion of Iraq. I think the author has confirmation
bias in too easily using the term to explain government and corporate policy
choices which have been based on false justifications.

------
yequalsx
Math person here. I'm curious to know if anyone used decimal numbers in their
tests and if negative numbers were used. The rule is increasing real numbers
and one can guess that the rule is increasing numbers without realizing this
includes all real numbers and not just integers.

In addition to getting it right did you use an exhaustive set of tests?

~~~
MalcolmPF
I tried a negative series and a decimal series, just in case the rule was
increasing natural numbers. I tried very large numbers to see if there was a
limit to the rule, and a series that had large contrast in between each
element. For fun I tried to see if the app would recognize "pi", "i", or "e",
but, perhaps unsurprisingly, it did not.

~~~
yequalsx
I would have checked complex numbers for fun but there is no reasonable
ordering of the complex numbers in the way there is for real numbers. It's too
bad it didn't recognize pi or e.

------
smilefreak
Does laziness have anything to do with the responses? You get a rule that
seems to work and so you seek the reward early. It takes effort to prove
yourself wrong.

I was trapped by this and guessed it was exponential series n^1,n^2 etc for n
starting at greater than 2. While technically true this was not the rule they
had in mind.

~~~
eru
Why is it technically true?

~~~
smilefreak
As in every number in that set is a subset of the larger set of x < y < z.
Poor language choice it's not true, yes I was wrong. I am just curious as to
how much laziness and not necessarily confirmation bias has to do with the
result. If getting it wrong had some kind of penalty or getting it right had
some kind of reward ( money etc. ..), how much better would people do then?

~~~
eru
Yes, it seems most people's rules actually were subsets of the true rule.

Laziness is involved to a certain degree. Though I don't know how much---no
one is forcing anyone to do the puzzle, so the truly lazy filter out, too.

------
dhruvbird
People try to make the fit be as tight as possible to the sample data -- the
explanation is that simple. I don't buy the explanation provided in the
article.

~~~
mpu
Pretty good point. That's how I felt.

Additionally, this setting is probably too close to usual situations you get
in school where there is little to no interaction and negative answers from
the teacher are seen as failures by students. (Speaking about education in my
country only.)

------
sp332
I got 7 yes, 5 no, and was still wrong. Maybe I'm just dumb.

~~~
EvanKelly
I'm curious about how you came to your answer and what led you there?

Were you testing a pre-supposed hypothesis that confirmed itself?

~~~
sp332
I got the part about increasing eventually, but I thought the third number
also had to be the sum of the other two. I came up with the sum idea after
trying (3 6 9), so only 2 tests. The idea that they had to be increasing came
later. I don't have it open but I'm pretty sure one of my tests was (1 2 5)
which should have tipped me off... in conclusion yes, I'm probably dumb.

~~~
snowwrestler
I know you're joking, but I think this is important:

Failing the test does NOT mean a person is dumb. The point of the article is
that confirmation bias seems to be a fundamental default in the way everyone
thinks. Certain people with specialized training in inductive problem solving
(scientists etc.) have learned to compensate.

If folks think it's an issue of intelligence, then they might be willing to
think "but not me, because I'm smart." (After all, many programmers believe
that they are smarter than the average bear). But while programmers are well-
trained to think carefully about sequences of numbers, they might be as
susceptible as anyone else to confirmation bias in other areas.

~~~
carussell
I wonder how much games like Twenty Questions plays into conditioning towards
this kind of approach, since the implication there tends to be that the fewer
questions you ask, then the better you've done.

------
sambe
The "Check" buttons weren't enough of a hint for me and I jumped into "This is
a numerical reasoning problem" mode. I'd argue that this kind of situational
bias is as much a factor here as confirmation bias.

------
cristianpascu
I understand the power of confirmation bias. I believe it to be natural for
anyone. It's perfectly normal to seek an explanation that fits the already
built cognitive structures, developed through experience. It's unreasonable to
simply jump into new paradigms everytime we encounter a new fact. It takes
time to prove that it doesn't fit, and then we start looking for new
explanations.

However, the test simply required a possible solution. There are plenty
solutions and it's absurd to think they have the simplest one. The simplicity
of the rule is subjective, in that is evaluated differently by different
people. The famous 'simple but no simpler' is relevant here. As long as we
were not told to look for the simplest solution, ALL solutions are equally
probable. That being the case, I started to with the first solution that
popped into my mind. I sticked with that because of my psychological state.
Some searched for other solutions.

I don't think that getting an YES was the main driving force. Of course it
feels good to get an yes. This is fundamental in human relations. But it's not
the whole story. People do not disbelief globar warming because they want to
get an YES. The reason is much deeper. Just as many, so many people go to the
wave of climate change because it's fashionable, it makes them feel good,
accepted , part of the mainstream. Being a climate change denier is being a
disident this day (not my flavor of disidentism), and being a disident is not
for everyone. And perhaps disidents picking their fight have complicated
reasons for doing so.

I went with the doubles.

------
danielvinson
I remembered this puzzle from HPMOR, but either way this is just writing a
unit test.

~~~
Tyr42
Having just re-read HPMOR a few weeks ago, I could answer it right away, but
there was actually a difference from HPMOR's version, which required three
positive increasing numbers, while this allowed negatives.

------
mc808
This looked like a puzzle I had seen before, so I assumed this was the case
(testing a few sequences just to verify) and turned out right.

I guess the conclusion is that if a problem looks suspiciously like one you've
encountered before, there's a good chance that they are the same or similar.
The world is self-organizing, not completely random where you must obsessively
second-guess your accumulated wisdom.

------
Gravityloss
Anyone played Monkey Island 2?

There's a puzzle with a doorman, there's a few distracting clues where the
answer is actually very simple.

It doesn't offer confirmation bias though, and it took some time to figure it
out.

I consider it a very similar test.

So one could actually construct such a test without the confirmation bias
part, and then look at how long it takes for people to realize the simple
model.

------
qznc
> A mere 8 percent heard at least three nos

I guessed correctly with only 2 nos. Since there is no penalty for guessing
incorrectly here, I felt safe enough with my theory. I might have checked for
more nos, if I had to announce my theory publicly (Twitter, comment, etc).
However, I also knew about Confirmation Bias beforehand.

~~~
rockdoe
I guessed correctly with only one no.

You can enter all kinds of crazy random sequences which only have The Rule in
common and get a yes, which seemed to be enough assurance. If you're trying to
get it to say "no" but failing, is that still confirmation bias? Doesn't sound
like it.

~~~
reagency
That highlights a great point: on this simple decision rule, using _random_
test data will draw the solution more quickly than human cleverness.

------
phaemon
I got this wrong, but oddly enough my first attempt _was_ a No. I tried, "6,
1, 8"

After that, I went with doubling and it worked 3 times with various sizes of
number, so I went with that.

Double bonus points if you can guess what I was testing with the first
sequence (which the given numbers do satisfy).

~~~
o0-0o
Were you guessing random area codes of midwestern states? :)

~~~
phaemon
Hah! No, I'm not American :)

But, in case no-one guesses, the answer is (rot13):

erirefr nycunorgvpny beqre

~~~
reagency
You vastly overestimated NYT's estimate of the cleverness of their readers

~~~
phaemon
A bit cruel, but I admit I laughed at that.

------
js2
For a more extensive version of this article, see "Mistakes Were Made (But Not
by Me)":

[http://www.amazon.com/Mistakes-Were-Made-But-
Not/dp/14915141...](http://www.amazon.com/Mistakes-Were-Made-But-
Not/dp/1491514132)

------
afitnerd
Seems busted now - clicking the "I think I know" button does nothing. I
thought the answer was:

Let the first number be x. If x is 0, then the second number is 1. Otherwise,
the second number is two times the absolute value of x. The third number is 2
times the value of the second number.

~~~
gervase
The answer they were looking for was: Let the numbers be x, y, and z; x < y <
z must be true.

Their hypothesis was that most people would guess as you did (they mentioned
78% of people did so).

------
harryjo
Veritasium covered this classic puzzle / bias test nicely, last year.
[https://www.youtube.com/watch?v=vKA4w2O61Xo](https://www.youtube.com/watch?v=vKA4w2O61Xo)

------
nsxwolf
So, am I supposed to feel bad if I assumed it was some tricky, hard to figure
out function? It just reminded me of those questions on the ACT or whatever
and I froze up and got frustrated.

Am I a dim bulb?

------
dude_abides
I'm a data scientist, and it relieved me no end that I got this one right:
[http://i.imgur.com/V5oJ4i4.png](http://i.imgur.com/V5oJ4i4.png) I would have
had second thoughts about my career choice if I got this wrong :)

The correct approach for any data modeling problem is to think in terms of
entropy. Each subsequent approach should minimize entropy, until you reach
diminishing returns.

~~~
Sadgrinner
Isn't your answer technically wrong, though?

The sequence is not monotonically increasing. It's strictly increasing. If you
test [1, 1, 2] or [1, 1, 1] or [1, 2, 2], you'll get "No" answers even though
those sequences are monotonically increasing.

~~~
thaumasiotes
Not so.

[http://mathworld.wolfram.com/MonotoneIncreasing.html](http://mathworld.wolfram.com/MonotoneIncreasing.html)

~~~
ddlatham
So.

[http://mathworld.wolfram.com/MonotonicFunction.html](http://mathworld.wolfram.com/MonotonicFunction.html)

[https://en.wikipedia.org/wiki/Monotonic_function](https://en.wikipedia.org/wiki/Monotonic_function)

~~~
thaumasiotes
You think the definition of "monotonic function" is more relevant to the
meaning of "monotonically increasing sequence" than the definition of
"monotone increasing" is?

[1,1,2], [1,1,1], and [1,2,2] are not monotone increasing. They're also not
monotonic functions.

~~~
Sadgrinner
In all seriousness, the definition of monotonically increasing that I was
taught is the same as exists in wikipedia:

"A function is called monotonically increasing (also increasing or non-
decreasing), if for all x and y such that x <= y one has f(x) <= f(y).

[https://en.wikipedia.org/wiki/Monotonic_function#/media/File...](https://en.wikipedia.org/wiki/Monotonic_function#/media/File:Monotonicity_example1.png)

This definition allows for 'flatness' in a graph, since the derivative does
not change sign.

Or, from [http://www-history.mcs.st-
and.ac.uk/~john/analysis/Lectures/...](http://www-history.mcs.st-
and.ac.uk/~john/analysis/Lectures/L8.html) :

    
    
        A sequence f(n) is monotonic increasing if f(n+1) ≥ f(n) for all n ∈ N.
        The sequence is *strictly* monotonic increasing if we have > in the definition instead of ≥.

~~~
thaumasiotes
You can draw that contrast (increasing vs strictly increasing), but what I was
taught was to contrast increasing functions/sequences with nondecreasing
functions/sequences.

A book or paper will make it clear what they mean by "increasing" by using the
definition. There, it doesn't matter at all -- they could just as easily coin
new words, since they immediately give the full definition. But the people
hanging around this thread, telling people who are using a very common
definition of "monotonically increasing" that (in paraphrase) "I hate to be
pedantic, but you've made a mistake, in that I would have phrased that
differently" have failed to contribute anything _or_ to be pedantically
correct. There's no case to be made that, if I say a "monotonically increasing
sequence" must be increasing rather than nondecreasing, I've made a
terminological mistake. This is a term with different definitions in different
treatments.

~~~
ddlatham
Thanks for your perspective. It's interesting that there appears to be
ambiguity in the terms and a diversity in what is being taught.

------
pbreit
I guess I'm the opposite: I got more No's than Yes's. My immediate sense was
to find No's. I guess that's why I like 1 star reviews.

------
sushirain
My explanation to the scarcity of "No"s: people are used to seeing in these
puzzles mostly sub-types of increasing series: exponential, linear, etc. By
the time they had ruled out these sub-types, and had resorted to guessing
"ascending", they wouldn't have encountered even a single No.

I predict that if the rule was narrower, like "exponential", much more guesses
would have yielded No's.

------
Lorenzo45
Wish I could have done this with no previous knowledge of the puzzle, I knew
it right away because I saw the exact same problem in a Veritasium video.

------
yellowapple
I think the first-known matching pattern plays a huge role. The original 2,4,8
sequence, for example, locked me in immediately to doubles of the previous
number (causing me to test 1,2,4 and 7,14,28 and such). Had it been a
different starting sequence (like 3,9,27), I might've based my guesses
differently. Same for 1,2,3.

In other words, first impressions really _are_ important.

------
mordrax
_They don’t want to hear the answer “no.” In fact, it may not occur to them to
ask a question that may yield a no._

So the author's obviously never heard of sanity checking, in fact that's the
second thing that I always do once I confirm a solution is to confirm it's not
a fallacy.

Having said that, my solutions were -10 -20 -40 -10 -8 -4 1024 1026 1030

and I said it was +2, +4 and got it wrong!

------
__z
Veritasium video on the same thing

[https://youtu.be/vKA4w2O61Xo](https://youtu.be/vKA4w2O61Xo)

------
SZJX
Umm... Not sure what's so special about it. Isn't that what all programmers
and science people do all day?

------
nerdo
Reminiscent of the folding table libertarians with their questionnaire and
that political cartesian coordinate chart. "You answered that 'it's wrong to
steal', [...psychobabble...], on this science graph it appears you've always
secretly been a libertarian, we meet at the cinnabon on sundays".

------
Gabriel_Martin
My process was [1, 2, 3], and then I guessed each number is greater than the
last.

It was totally a possibility that they wouldn't apply the simplest rule, but I
felt it highly unlikely. This "rule" is a meme of the rationality community,
especially given the example, so it seemed pretty likely that it was
sequential numbers.

------
ljk
did anyone else only try even numbers? guess "increasing even numbers"

pretty interesting how in-the-box my thought process was

------
drhdylan88
Veritasium, a pretty interesting youtube channel, posted a video on this
experiment a while back. I found the discussion afterwards to be more thought-
provoking than this article.

[https://www.youtube.com/watch?v=vKA4w2O61Xo](https://www.youtube.com/watch?v=vKA4w2O61Xo)

------
decisiveness
Constraints allow for only integers <= 9999999999999999 and >=
-9999999999999999 which is interesting considering (-)10000000000000000
through (-)9223372036854775807 are also within the bounds of a 64 bit integer.

It's ironic that these facts are not mentioned considering the article is
about confirmation bias.

------
progmanal
The attached reading material is interesting, but this question is too similar
to problems where you guess the next one in the sequence and none are missing.

A rule where the numbers are increasing does not explain why 3 or 5 or 6 is
missing from the sequence in that version of the question that is much more
common.

------
pmelendez
I failed it again... Every time I take a test like this I ended being fooled
by the confirmation bias.

------
jasallen
There is no penalty to being told "no". That _also_ applies to "I think I know
the answer". There is no penalty for being told "no" there as well, so once
you have a reasonable guess why not check it? Disagree with their analysis.

~~~
ddlatham
There is an implied penalty because it says about the answer box "Make sure
you’re right; you won’t get a second chance" in contrast to "You can test as
many sequences as you want" about the sequences.

~~~
Falcon9
This is absolutely correct. Perhaps Hacker News culture is such that they
realize such a warning can be overcome through a refresh, or possibly an
incognito browser window if a cookie is preventing a second guess. Still
though, within the rules of the game, there is no drawback of testing negative
sequences, but there's a clearly defined drawback of entering an incorrect
answer.

------
theVirginian
I don't seem to be able to submit my answer or follow the link to see the
answer for "just tell me the answer" is this supposed to be a trick or is my
browser just not capable of properly following those links? Using Chromebook.

------
Geee
Anyone tried entering letters in the boxes? I didn't. I think _that_ would
have been an unexpected twist.

So, after all, I think I fall in the trap of confirmation bias that the
sequence must consist of numbers only.

~~~
dEnigma
Well, after all they tell you that the sequence consists of numbers:

>We’ve chosen a rule that some sequences of three numbers obey

>Now it’s your turn. Enter a number sequence in the boxes below, and we’ll
tell you whether it satisfies the rule or not.

------
gesman
After total of 13 answers including 6 "no", I guessed it right :)

------
arikrak
the questions is somewhat ambiguous. it would be more interesting to see what
happens of people really understood the question and if they had a real
motivation to get it correct.

------
jjuhl
The same experiment:
[https://m.youtube.com/watch?v=vKA4w2O61Xo](https://m.youtube.com/watch?v=vKA4w2O61Xo)

~~~
zeidrich
What's funny is that I felt smug and self satisfied for getting the correct
answer even though I watched that video when it came out.

------
Kluny
Decimals and negative numbers still get you a "yes" as long as you obey the
rule. However, the button to submit your answer didn't work for me.

------
bluker
My first assumption was x,2x,2(2x) - then I tested 0,0,0 and got a NO which
disproved my first assumption... the answer is obvious after.

Did anyone else test 0,0,0?

------
adamc
I got it, but it took a bunch of guesses before I had the pattern. It's a good
example of how our preconceptions shape our answers.

------
ernsheong
We should all be doing good on this one if we have been practicing making our
test cases are red first before turning them green :D

~~~
lugg
Looking at my result there, seems to be something I need to start doing. TIL
:P

To be fair, I did check 3/6/12 - just to make sure it was double and not
powers of two. Guess that's the articles point though isn't it!

------
0xdeadbeefbabe
Confirmation bias affects Corporate America and Government Policy, but not
science? Talk about confirmation bias :)

------
jessaustin
Have large numbers of HN people really not seen this riddle before? Maybe I
read weird things.

------
elwell
After programming daily for more than a decade, I question my assumptions
annoyingly often.

------
brianwillis
This doesn't seem to work right for negative numbers. The article says the
rule is "each number must be larger than the one before it", but if you try -2
-4 -6 it says that pattern doesn't match the rule.

Maybe I'm just being pedantic here, but last I checked -6 was larger than -4.

~~~
Tloewald
It works for floating point numbers -- 0.01 0.02 0.04 for example. So it's a
geometric series that has to start with a positive number and doubles. (The
submit button and the show the answer button were broken for me.)

~~~
eru
That's unintentionally hilarious.

~~~
Tloewald
Yeah I was wrong in exactly the way the article expected :-)

But the submit button being broken did kind of suck.

------
ammaar
for anyone who hasn't seen it Veritasium did a video on it.

[https://www.youtube.com/watch?v=vKA4w2O61Xo](https://www.youtube.com/watch?v=vKA4w2O61Xo)

------
vincston
Sadly ive already seen Veritasiums test. so i was biased.

------
PSeitz
Nice game, got it right after 7 yes and 7 no

------
eridal
I thought..

    
    
        F(n + 1) = n * 2

------
limeyx
Got it right with 7 nos.

------
flint
Yup - Dick Cheney!

------
mastre_
-10, 0, 0.01

------
725686
That page prints the nyt logo in the javacript console (will probably look
messed up here):

    
    
           0000000                         000        0000000
         111111111      11111111100          000      111111111
         00000        111111111111111111      00000      000000
         000        1111111111111111111111111100000         000
         000        1111       1111111111111111100          000
         000         11       0     1111111100              000
         000          1      00             1               000
         000               00      00       1               000
         000             000    00000       1               000
      00000            0000  00000000       1                00000
      11111            000 00    000000      000                 11111
      00000          0000      000000     00000              00000
         000        10000      000000      000              0000
         000        00000      000000       1               000
         000        000000     10000        1     0         000
         000        1000000 00              1    00         000
         000         1111111                1 0000          000
         000          1111111100           000000           000
         0000          111111111111111110000000            0000
         111111111        111111111111100000          111111111
           0000000              00000000              0000000
    

NYTimes.com: All the code that's fit to printf() We're hiring:
developers.nytimes.com/careers

~~~
heartbreak
Every nytimes.com page displays that in the JS console.

~~~
stephengillie
Putting your engineer hiring notices under the hood is becoming a common
"Easter-egg" practice. It's also a form of targeted recruitment advertising -
the only people who see it are your target audience.

------
elektromekatron
I liked this puzzle. It got me. I am thankful for it.

------
bediger4000
The official answer to this puzzle makes a huge assumption: that there is one
correct answer. There is not one correct answer. (x, 2x, 4x) gives you a "yes"
every time, therefore it is a correct answer, at least as automatically
checkable, and there's an infinite number of such tuples. To find a "no"
you're reduced to random guessing. That's not a puzzle, that's crap. The
confirmation bias material might be true, but the puzzle does not illustrate
it.

~~~
myNXTact
It's amusing that you went to a website on confirmation bias, did the puzzle
incorrectly, presumably read the material on confirmation bias, but still
suffer from the effects of confirmation bias.

~~~
harperlee
Amusingly human, if I may add. Reading "Thinking, fast and slow", by Daniel
Kahneman, one key idea that I got was that even knowing against biases you are
very, very likely to suffer from those biases. Disheartening results were
gathered from studies done on well-trained psychologists and people prepared
for the experiment, to no avail. Can't remember the details right now, but
just read the book, it's awesome. Another good one was "Influence" by
Cialdini, but they gave you tips on trying to avoid those biases that, upon
reading Kahneman, I don't think anymore that are very useful.

~~~
reagency
You can greatly improve your performance by setting up internal and external
reminders to check your assumptions and commo falacies.

Related: read The Checklist Manifesto.

