You act like that's a "gotcha" instead of a normal thing. All they mean by that [0] is that can't mathematically prove their developers/tasks/tool are representative for of the majority of worldwide developers/tasks/tools.
You're demanding an unreasonable level of investment for anyone to "prove a negative."
The burden of proof lies on the people claiming zillion-fold boosts in productivity across "enough" places that they don't really define. This is especially true because they could profit in the process, as opposed to other people burning money to prove a point.
If you want to shit talk LLMs, you better come armed with research, buddy. Claims about how it will revolutionise every profession just need n=1 anecdata though.
Historical comparison: "I just had a pizza delivered on the new Segway and it was super duper cool because they came right into the conference center, so say goodbye to cars and bikes, by 2025 it's all going to be Personal People Movers!"
That said, I think LLMs will have a bigger effect than a self-balancing scooter, both positively and negatively.
Correct, the study showed that it slows down experienced developers. We don't know what it does to inexperienced developers so that sounds like a good research topic. But it still leads to the question of why experienced developers should be told to adopt it, given that it slows them down.
