I agree with the study participants. Using LLMs makes me feel less capable at some cognitive tasks, but it's hard to say whether I've actually declined or if use of AI has dispelled some of the Dunning-Kruger effect for me around my own skills.
For instance, LLMs are excellent at rewriting text in a specific style/tone. I used to believe I was quite good at that, but LLMs always do better. Now I no longer believe I am quite good. Did I become worse at text synthesis, or did I simply become more aware of my skills and limitations?
This is emblematic of a broader issue with self-reported data. It'd be good to measure critical thinking skills against earlier benchmarks for a clearer picture.
It's also important to not only focus on one skill, because while AI might make certain areas of our mind atrophy, others might now be more engaged. Just like when pocket calculators became popular and we all got worse at mental arithmetic but much better at applied mathematics overall. High level programming languages have made software engineers worse at comp-sci, but much better at applying it; commodification of cars made the average driver much less capable of understanding car mechanics, but much better at driving; and so on. My thesis is that human brainpower is not generally reduced by technological innovation, but changed.
One thing's for sure - cognitive changes in society (and especially how they relate to technology) is an area I'd like to see more research in. I think there are a few discoveries to be made there.
That's a great point about how using the models may actually just be revealing our own incompetence, not causing it.
Maybe it's a quantity over quality issue. What if we're actually doing more of the thing - or at least we're guiding the process - than we would have if LLMs weren't available, which enables us to become better at recognizing how LLMs are actually beneficial for the task.
I remember the story of the two groups of pottery students, one tasked with making as many pots as possible, and the other with making one perfect pot. At the end of the day, the quantity group's pots were also higher quality than the quality group's single pot, simply because they had far more practice.
“Just like when pocket calculators became popular and we all got worse at mental arithmetic but much better at applied mathematics overall. High level programming languages have made software engineers worse at comp-sci, but much better at applying it; commodification of cars made the average driver much less capable of understanding car mechanics, but much better at driving; and so on.”
I don’t agree with any of these conclusions. I guess it’s debatable of course.
For instance, LLMs are excellent at rewriting text in a specific style/tone. I used to believe I was quite good at that, but LLMs always do better. Now I no longer believe I am quite good. Did I become worse at text synthesis, or did I simply become more aware of my skills and limitations?
This is emblematic of a broader issue with self-reported data. It'd be good to measure critical thinking skills against earlier benchmarks for a clearer picture.
It's also important to not only focus on one skill, because while AI might make certain areas of our mind atrophy, others might now be more engaged. Just like when pocket calculators became popular and we all got worse at mental arithmetic but much better at applied mathematics overall. High level programming languages have made software engineers worse at comp-sci, but much better at applying it; commodification of cars made the average driver much less capable of understanding car mechanics, but much better at driving; and so on. My thesis is that human brainpower is not generally reduced by technological innovation, but changed.
One thing's for sure - cognitive changes in society (and especially how they relate to technology) is an area I'd like to see more research in. I think there are a few discoveries to be made there.