

The odd one out - inspired by IQ tests - cema
http://blog.tanyakhovanova.com/?p=182

======
goodside
This may not apply to the author of the linked blog post, but it's long been a
pet peeve of mine how casually people dismiss intelligence testing after
seeing one or two examples of poorly designed questions. The rarely considered
reality is that IQ tests vary widely in quality.

Some reasons are obvious: The test might be intentionally designed to measure
something other than IQ (success in college, performance on a job, money spent
on College Board® prep courses). The test might be designed to test within a
particular IQ range, whether high or low. The test might be designed only to
be taken by people fluent in a particular language, or who received an
education from a particular school. And so on.

But the biggest and least obvious reason why bad IQ tests are everywhere is
that good IQ tests are not only difficult to make, but _expensive_ to make.
Good tests are not made by geniuses in armchairs thinking up riddles. Test
design is an _empirical science_ , and the final judges of any question's
value are the statistical relationships that show up in the answers when the
test is given.

Empirical evidence sterilizes tests from the biases of their authors, and does
so much more effectively than waiting for complaints from morally outraged
observers. Questions that penalize takers of different cultural backgrounds
are bad questions not because they're socially repugnant, but because they're
noisy--you can demonstrate that people who otherwise score highly get the
question wrong more often than they should. Questions like those addressed in
the blog post, where some really smart people are attracted to a "wrong"
answer, will have negative inter-question correlations among the brightest
subjects, which is a giant blinking neon sign in any analysis.

This sanitizing effect is so strong that the tests which show the most
desirable psychometric properties, such as Raven's Progressive Matrices, often
evoke the opposite complaint: they're so minimal and homogenous in form that
it's hard to accept they're a better measure of IQ than scattershot tests of
imperfect analogies and odd-one-out puzzles. If you're curious, RPM goes a
little something like this: <http://iqtest.dk/main.swf>

IQ testing is a complicated science, and discrediting it because there's one
or two (or fifty) dumb questions on the SATs is like giving up on journalism
after reading an issue of USA Today.

~~~
hooande
Standardized tests of any kind are fundamentally flawed. You can't measure a
person's intelligence using a test. You can only measure very specific things,
and even then you can't expect to do a good job of it. Does a high SAT score
predict success in college? People with great test scores drop out / fail out
/ join the peace corps by the thousands every year. Testing can a good
indicator, but it's not an ideal solution.

I'm not against testing. There has to be some way to decide who gets into what
college, and the SAT is a published standard. But there's no point in
defending a flawed system. Let's just accept that IQ tests are a bit
ridiculous and move on.

~~~
asciilifeform
> You can't measure a person's intelligence using a test.

Do you actually believe this dogma?

This Usenet post should be read by all "IQ deniers": <http://tiny.cc/F7gxA>

_"according to IQ doubters, knowing how smart I am should tell you nothing
about my IQ. It is as likely to be 50 as 150."_

The same poster once proposed a beautiful test of the depth of an IQ denier's
beliefs:

You are to be operated on, and have been given a choice between two surgeons.
You know nothing about them, save for the IQ of each. Will you choose the
surgeon with the lower IQ? What if the difference between them is an entire
standard deviation? Two standard deviations? If you truly believe that IQ
tests measure _nothing_ \- that the result of any such test is essentially a
random number - you should happily choose the surgeon with the lower score. If
you do not believe it, we are merely quibbling over the accuracy of existing
tests.

Moreover, the only way to seriously dismiss the existing IQ tests as
hopelessly inaccurate is to borrow tricks from the Creationists' playbook
wholesale (as Steven Jay Gould, the patron saint of IQ denial, often did) and
close your eyes (and more importantly, those of your audience) to reams of
high-quality data.

More references, for those actually interested:
[http://www.psych.utoronto.ca/users/reingold/courses/intellig...](http://www.psych.utoronto.ca/users/reingold/courses/intelligence/cache/1198gottfred.html)

And if you want to know the core driving force behind the IQ denialists:

[http://www.gnxp.com/blog/2007/10/james-watson-tells-
inconven...](http://www.gnxp.com/blog/2007/10/james-watson-tells-inconvenient-
truth_296.php)

 _"Watson was thinking like a scientist. Which is exactly why he was punished.
The moral laws of our society dictate that we are not allowed to think
scientifically about some issues. Especially not in public."_

~~~
hooande
If IQ is a valid measurement, then why isn't it used to make important
decisions? While your surgeon example may be a bit contrived, it raises a
valid point: Why haven't you ever considered the IQ of your surgeon? Does
anyone even know their surgeon's IQ?

Investors don't seem to care about IQ at all. It wasn't an issue of public
consideration during the last presidential election. If it could reliably
predict anything, don't you think more people would ask about it? I'm sure
there are places where IQ scores are considered (applying to college or
Google), but these are the exception, not the rule.

There is generally a difference between people with an IQ of 50 and an IQ of
150. But you don't need a standardized test to see those differences. The
problem is that there is very little difference between an IQ of 120 and an IQ
of 130. If all IQ is good for is telling the difference between the very
intelligent and the learning disabled, then it isn't very useful in daily
life.

~~~
asciilifeform
> If IQ is a valid measurement, then why isn't it used to make important
> decisions?

It absolutely _is_ used. We are simply forced to measure it through indirect
means, given that IQ tests are verboten.

> Does anyone even know their surgeon's IQ?

Medical school is (perhaps above all) a proxy IQ test.

> Investors don't seem to care about IQ at all.

They are quite interested in heavily IQ-correlated traits.

> If it could reliably predict anything, don't you think more people would ask
> about it?

U.S. law de-facto prohibits the use of IQ tests for anything of economic
importance:
[http://caselaw.lp.findlaw.com/scripts/getcase.pl?court=US...](http://caselaw.lp.findlaw.com/scripts/getcase.pl?court=US&vol=401&invol=424)

> But you don't need a standardized test to see those differences.

There is also no need to use a thermometer in laboratory work - we're all
equipped with fingers. At least ten measurements' worth, at that.

Indirect measures are useful.

Want to determine which teenagers would make decent electrical engineers?
Instead of $500 worth of IQ test (proper IQ tests are administered by private
psychologists, who do not work for free) one can use $150,000 worth of
college/trial-by-ordeal - a poorer measure of general intelligence, overall.
Yet it continues to be the only allowed measure, because IQ tests are
Officially Evil.

> there is very little difference between an IQ of 120 and an IQ of 130

On what basis do you say this?

> If all IQ is good for is telling the difference between the very intelligent
> and the learning disabled

This is merely its last permitted use.

~~~
hooande
I agree with you that intelligence does exist and it needs to be measured.
Measured by the completion of medical school/college, by whatever heavily IQ-
correlated traits that investors look at or by a wide variety of indirect
measurements or proxy tests.

Intelligence should not by measured by a standardized test. Standardized tests
simply do not have a history of predicting success at anything on a micro
level. If they did, people would use them. It would be trivial for large
employers to pay $500 per employee to have an IQ test taken. And it isn't
illegal for them to do so. From the court decision:

"Nothing in the Act precludes the use of testing or measuring procedures;
obviously they are useful. What Congress has forbidden is giving these devices
and mechanisms controlling force unless they are demonstrably a reasonable
measure of job performance."

The law doesn't say that you can't use IQ tests for anything of economic
importance. It says that you can't use them to discriminate. Do you think
Google could show that a high score on a general intelligence test bears a
demonstrable relationship to successful performance as a software engineer?
And if it doesn't...then what good are the test scores?

It is possible that there is a great conspiracy against IQ tests. They could
be the best measurement of intelligence, but people just don't like them. It's
also very possible that the test scores aren't used in society because are
relatively meaningless.

~~~
asciilifeform
> Standardized tests simply do not have a history of predicting success at
> anything on a micro level.

This is empirically false. Read about the Longitudinal Study of Youth.
Children took tests, grew up, lived their lives. You cannot argue the data out
of existence.

> The law doesn't say that you can't use IQ tests for anything of economic
> importance. It says that you can't use them to discriminate.

There are many practices which are not illegal per se, but put you at dire
financial risk liability-wise.

Read about the _actual consequences_ of the precedent set by the Griggs v.
Duke Power Co. decision. Talk to a lawyer about whether it can ever be safe to
include a traditional IQ test in your company's hiring process. The problems
involved in demonstrating "non-discriminatory intent" and "performance
applicability" are insurmountable. Unless you have extraordinarily deep
pockets, the sheer expense (not to mention PR debacle) of fighting such a
lawsuit will sink you before you are at any risk of overturning the precedent.

> It's also very possible that the test scores aren't used in society because
> are relatively meaningless.

Once again, the falsehood of this statement can be uncovered by anyone who
cares to look. We aren't talking about an organized conspiracy here - only a
generation's worth of toxic political correctness.

------
pyre
I think that it's funny that a lot of people are saying that the 'odd one' is
the 'normal one'. It's just a matter of common features. The large, red,
bordered square is the only one that shares more than 2 features with all of
the other objects.

    
    
      Features
      ========
    
          Color   Frame   Size    Shape
          -----   -----   ----    -----
      1.  red     yes     large   square
      2.  red     no      large   square
      3.  red     yes     large   circle
      4.  green   yes     large   square
      5.  red     yes     small   square
    
    
      Features in Common
      ==================
    
        1 2 3 4 5
      1 - 3 3 3 3
      2 3 - 2 2 2
      3 3 2 - 2 2
      4 3 2 2 - 2
      5 3 2 2 2 -

------
hughprime
Those sorts of questions always bothered me in IQ tests too. I was also always
bothered by questions like:

"What comes next in this sequence? 1,4,9,16,25,"

Sure, the answer _might_ be 36, but then again maybe we're looking at some
subtler sequence of numbers than that. Maybe it's "the number of bald men who
walked past my house every hour since 5am". Maybe the next number is 345 due
to the annual Patrick Stewart Lookalike Parade. Who knows?

~~~
mattmaroon
You're probably taking it too seriously then. There's a certain subtext, the
catching on to of which might be part of the test, that it's a rational and
discernible pattern, and that if the answer is not 36 then the question is
broken.

~~~
hughprime
_You're probably taking it too seriously then._

Oh, certainly. I've never found a case where there's _genuine_ ambiguity, it's
just that there's a certain part of my brain which takes delight in pointing
out all the potential flaws in everything I read. It's much harder to
concentrate on the test when half my brain is busy visualizing what four
hundred Captain Picards would look like marching down my street.

~~~
jamesbritt
"Oh, certainly. I've never found a case where there's genuine ambiguity ..."

I recall a test that asked to pick the one word out of four that was different
or didn't fit or something.

The thing was, three out of four words described red objects. But a different
three out of four words had one syllable; the other had two.

So it them became a game of, What was the test creator likely thinking?

~~~
blinks
This is why a proper test needs to be administered by a professional, I would
expect.

------
kabdib
I had to read one of the comments below to find out that one of the objects is
red instead of green. I'm red-green "colorblind" (about ten precent of males
are).

Be careful what you are testing.

~~~
barrkel
I'm red/green colour-blind too, but it relates to contrast. Red doesn't stand
out as vibrantly as other colours do, that's all.

For example, in dim light, I can find it difficult to tell the difference
between brown and red on a snooker table. But close up, or in bright lighting,
there's no problem. For example, there's a red, a green and a brown ball in
this picture:

[http://upload.wikimedia.org/wikipedia/commons/e/e6/Snooker_t...](http://upload.wikimedia.org/wikipedia/commons/e/e6/Snooker_table.jpg)

I can tell the difference between the red and the green more easily than the
red and the brown. If the red wasn't visible on the table for comparison, and
the brown wasn't near its home spot on the centre of the baulk line, I
wouldn't be sure it's not red.

I doubt that 10% of males have the same level of difficulty distinguishing red
from green as you indicate.

------
keefe
To me, this says more about the importance of rigorous test writing
methodology than the silliness of IQ tests. You're trying to get a particular
signal, the fact that noise exists does not eliminate the need to read that
signal. However, IQ tests were first developed to determine how far behind
normal certain types of children were, rather than as a measure of excellence.

------
Gormo
I don't necessarily see this as a bad question. The correct answer is "it
depends on which attribute is most important".

If that is an available answer, then the question can effectively separate
subjective, presumptuous approaches from objective, analytical ones.

If "it depends..." is not an available choice, then it demonstrates that the
test itself is subjective and presumptuous, but this doesn't necessarily
implicate standardized tests in general.

------
tel
I wonder if there's any pattern to the order in which people discover the
special properties. I saw it as: green, no frame, small, circle.

~~~
krolley
I saw it as: no frame, green, small, circle.

It just demonstrates that this question might be a good one, if what you're
testing for is the order that people find "the odd one" (not necessarily in an
IQ test).

------
peterhuesken
The first one is special, for two reasons: 1) as stated by others earlier:
it´s the only one that isn´t special

The next reason states it more clearly imo.

2) the first symbol differs from the other four symbols in exactly 1 way; the
other symbols differ from each other in 2 ways.

------
xiaoma
The one on the left is the odd one out because it doesn't have a single unique
feature.

------
mildweed
Or perhaps rephrased as the 'odd one in'...

~~~
NathanKP
Yes the odd one out is the only one that isn't odd in some way.

~~~
cema
True. But in general things may be more complicated.

On a typical test, when given a similar problem, I would have to try and
figure out the thinking style of the test designers.

------
maxklein
That was a bit easy, I think most people would figure this one out, mostly
because there is no decoy, and there is little chance of your trying too long
in the wrong direction.

