Weak science. They ran a battery of 12 tests and looked for one with a statistically significant difference at the 5% level. If there were no actual differences, how many tests would you need to run to find at least one that by chance shows significance? If you run 20, you’d expect 5% to do so, on average, which is 1. Running 12 and finding 1 is not a significant finding.
Then suspiciously, they found one significant result on trial 5 of the verbal learning test but little to no improvement on the other trials. That makes no sense.
Finally the brain changes had a p-value of 0.046, which is barely significant by even the weak 5% threshold, and the changes weren’t correlated with the test result improvements.
the only way to get replicated is by posting the poorly funded weak science version and having people talk about it
maybe this could be a job for AI, the LLM reads studies and highlights ones worth replicating and funding, sparing the rest of us from every seeing them at all
would probably be better than the politics involved with getting recognized
Have you ever tried to use LLMs to navigate around papers in academic research areas? They are not good in tasks like that, eg finding what is important, often they cannot even summarise papers properly, getting confused with causal directions and relationships between different parts.
Then suspiciously, they found one significant result on trial 5 of the verbal learning test but little to no improvement on the other trials. That makes no sense.
Finally the brain changes had a p-value of 0.046, which is barely significant by even the weak 5% threshold, and the changes weren’t correlated with the test result improvements.
Perhaps interesting but only if replicated.