Hacker News new | comments | show | ask | jobs | submit login

Speaking as someone with a cognitive science background (and yes, probably too lazy to dig up specific references), I'll say that there exist a lot of poorly designed or poorly implemented tests of "intelligence". Even the concept of intelligence is poorly defined, and lots of different measures are just proxies. Even if the test is good and has some psychometric utility, often giving the test is a giant pain to do well.

Even if the test is good, it's probably calibrated on WEIRD [0] subjects, so is really only good as another kind of gatekeeper - you can probably use old-school IQ tests as an excuse to reduce diversity. People who aren't WEIRD may not do as well on these tests (even though they're just as "intelligent" (parenthetically bracketing the ill-defined term for a moment)), so you have your reason that you only hire white dudes. But that's a cynical extension of logic there.

Most people, when approaching a psychometric test like one of intelligence or personality make the first incorrect assumption that modern cognitive science can actually form a construct like "personality type" or "intelligence" that's stable across all cultures and norms, testable, and repeatable. That's just not really true, and by and large what makes it to public consumption is pseudo-science.

Diverting just a little bit here, but MBTI [1] is another good example of bad use of psychometrics in business. There's about as much science supporting Myers-Briggs tests as there is astrology, yet people still use MBTI for actual decision making. Types aren't shown to be stable and they're not well-clustered (meaning you can be a mix of introvert and extrovert, or show signs of both depending on the day, context, mood). Zodiac Sign and MBTI Type should only be used as pick-up lines in a bar. "Hey, babe, I'm an INTJ, so I'm not going to say anything else."

There are good uses of psychometrics, but they're rare enough that I take any reliance on any kind of psychometric during hiring as kind of a bad sign.

0: http://blogs.scientificamerican.com/primate-diaries/the-weir... 1: https://www.psychologytoday.com/blog/give-and-take/201309/go...




To be fair, I think that Raven's matrices probably get around this. If someone explains the process to you, you don't actually need to be able to read to do well on the test. https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices

I do find it surprising that Raven's appears to be more susceptable to the Flynn effect (people keep getting better at IQ tests, for an unknown reason).

Completely agreed on the MBTI, but its surprisingly difficult to convince people that's its useful.

FWIW, I agree with most of what you said above, I was just pointing out a counter-example. (Psychologist/psychometrician here).


Yes, I agree that Raven's matrices are pretty good at measuring something in a way that's not super-reliant on language. I'm not inherently opposed to the notion of generalized intelligence as a construct either, I just think measuring it is pretty tricky. The Flynn effect is another great example - these instruments are nuanced, and it's important to really understand them in order to interpret their results.


Totally agree, I'm somewhat in sympathy with Shalizi's argument that g is just the result of iterated factor analysis. That being said, done well, these kinds of tests can provide useful data, especially if you're willing to randomly hire to calibrate performance (which no one ever is, sadly).


I find the "WEIRD" thing a bit tough to support, at least with regard to IQ. East Asian populations (Hong Kong, South Korea, and Japan right at the top) seem to consistently score higher on IQ tests than "WEIRD"os. Maybe the less pathetic and self-hating way to look at it would be "EIR" populations. Even then, it's hard to argue that spatial and linguistic reasoning are not universal values; even if you're uncomfortable that some populations have lower averages than others. I feel as though this must come from a lack of respect for the obvious personal value of Education, Industrialization, and Riches.


You're making two different arguments here and kind of tying them together. First: you're claiming that there actually is some kind of universal "spatial reasoning" skill. Usually people who claim that also believe that this skill is somewhat generic - it applies to lots of different contexts and transfers easily from (say) visualizing rotations in space to finding your way around a 3d environment and so on. There's actually a fair bit of research that questions this supposition (take a look at embodied cognition to challenge the theoretical model, or stereotype threat effects to challenge the assessment model).

Secondly, in this context, the assumption is that not only does this generalized attribute exist in humanity and we can measure it, but this attribute causes success in $job. That's a further assumption that I'd call into question. Sure, plenty of people who do well on $test do well on $job, but those could easily be covariate or epiphenomenological relationships rather than causal ones.

I don't doubt that for some values of $test, there's some discriminatory power predicting the presence of success, especially in more monocultural environments (affluent, often-white, often-male, often-young startups, to caricature). But I do believe that this kind of test would suffer from strong false negatives as many kinds of people who would otherwise be excellent programmers (or whatever other $job) are rejected for not doing well on $test.

That's why I worry about anyone who puts real stock in these kinds of tests - it is (generally) indicative of a kind of science blind spot in the user. People are critical and skeptical of all kinds of other scientific claims, but psychological measures tend to get a pass.

I imagine the fact that a lot of us tend to score well on IQ tests has a lot to contribute here. When I found out how many standard deviations above the mean my IQ score represented, I was pretty excited about how awesome I was. When I realized that the test was probably baloney, I had to figure out more interesting ways to shore up my self-esteem. Personally: I bake bread and fish for compliments on the quality of my sourdough.


I prefer stronger evidence than "I don't doubt" when positing an international raciosexual conspiracy. Especially if that conspiracy would presumably have heinous and far-reaching consequences.

I learned, as an adult, that I had taken an IQ test while young and had scored in the 99.97th percentile. I think that I got that not everything benefits from what IQ tests measure. That said, I had already started seeing great success in a knowledge industry. It is clear that you also have great linguistic intelligence. I don't think you could completely dissociate your IQ test results from that intelligence.


I don't think you could call the general lack of diversity in tech a "raciosexual conspiracy". Rather, it seems a lot more like the result of classic pressures - institutionalized racism and sexism, combined with the general tendency of people to prefer other people who look like them.

The fact that certain groups of people tend to test well (or, taken conversely, that there are so many reasons why otherwise "intelligent" people don't test well) just exacerbates this problem and I think the central claim is still reasonable: things like IQ tests aren't necessarily strong predictors of tech skills, are dangerously close to pseudoscientific when misapplied, and have enough other theoretical problems that they should probably not be utilized during a hiring process.

As to my own linguistic skills, I contend that a lifetime of reading as well as a MS (in computer science), a Ph.D (in cognitive science and education), and ten years afterwards in academics including a professorship has prepared me as a writer. Did I have some initial Potential that gave me a head start? Maybe. If nothing else, I had the head start of my general introvertedness and a love of both reading and geekiness. A short test that purports to measure a fixed potential somehow inherent in someone is going to be pretty flawed.

But my "I don't doubt" phrase was more along the lines of ceding a central point: there probably do exist certain inherent characteristics that vary between people and provide some kind of predilection or head start. I'm just not yet convinced that the scientific community has really identified them yet, or that they're really able to effectively measure them yet. Instead, we get proxies that have a very high false negative rate, especially among otherwise-marginalized groups.

Anyway, four paragraphs is probably enough here.


Could there possibly be a chance that with your majestic IQ might make you a bit biased on the subject? Mind you, I have a uselessly high IQ, too, and I still think they're are an insult to hard working and bleedingly knowledgeable people who could outperform you and I, easily. I'm just not sure what the value of the tests are, considering the discriminatory ease.


Thank you! It's refreshing to hear that from anyone with a cognitive science background, as you put it.

I had an MBTI test pretty much forced on me a while ago (senior engineer would be unhappy if I didn't take it). I argued very forcefully against it, saying it was like astrology, that it made me feel like the company was asking me to participate in some religious ritual when I was an atheist and so on. I'm not in that company anymore and one major reason I ran was that they allowed pop psychology like that to dictate decisions.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: