Hacker News new | past | comments | ask | show | jobs | submit login

Huh, good catch. The too-short time limit issue was a methodological issue I had definitely not thought to check for. (Haier doesn't mention it.)



There are actually several distinct issues with Jaeggi 2008. The speeding is a concern but it increasing looks like it's not driving the score increase. The passive control group is a big issue (although Jaeggi et al, via Au, are doing their best to fight to the grim end and deny it as long as possible). As is the subsamples using different IQ tests.

Finally, as Haier spends a lot of time explaining, pre-post test gains is fundamentally invalid as a way of proving testing increases the g-factor, because such gains are necessary but not sufficient; an increase in intelligence should show up as a gain, but many non-increases will also show up as a gain because despite the best efforts, IQ tests do not measure solely intelligence.

I'm increasingly understanding that this is as big a problem for n-back as the passive control group inflation. Many attempts to increase intelligence in the past, some where there was no dispute at all that test scores did increase, have failed as soon as a latent variable analysis was applied; I'm not aware of any which passes. All of the interventions had increased scores on only some subtests, and there were no general gains; they had not increased intelligence, only taught a specific subdomain. So our prior is highly against n-back being the exception. Worse, several studies have already tried to apply the latent variable analysis, and they all indicate that n-back flunked. So our already exceedingly skeptical prior gets reinforced by evidence of no effect on intelligence, whatever one subtest like a matrix test might say. (If Au or Jaeggi have addressed this, I have not seen it yet. They prefer to concentrate on the fact that the test scores increase and that you can cast doubt on the active/passive criticism by trying to correlate with whether a study was done in the USA or not, or whether the control group gained enough pre/post.)

You can see some links discussing the topic, including Haier's methodological argument here, in point #3 in http://www.gwern.net/DNB%20meta-analysis#analysis




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: