I'm actually kind of flabbergasted that people -no matter who they are- are automatically given the benefit of the doubt, without question.
I'll bet that a lot of folks just assume that anything they do will be taken at face value, without question or inspection. I also suspect that many "brought and paid for studies" are done this way.
For my own work, I generally assume that most of these studies are pretty much worthless, and tend to do some of my own homework before accepting them. Since most don't concern me at all; it's not a big deal.
Health is just one place this kind of thing happens. Software Development is absolutely rife with bad implementations. I am not in AI, but I have heard from a number of people that AI has a big problem with irreproducible results.
I work in ML research and I used to do experimental physics. I'd agree that specific results in papers can be hard or impossible to reproduce, but that never really bothers me because at least in my work, the specific experimental result is rarely material to why I'm interested in the paper. It's more of a demo, and like a demo, you know its orchestrated to look good. What I'm interested in is what is the mechanism behind the advance and do I think its applicable or relevant to what I'm doing. If the paper is really just a random observation of something that worked better, without a causal explanation, it's not very interesting, but I don't see those often.
Maybe health research is very different, and people are latchjng on to surprising results they find in papers, but I doubt it's a big problem in academia, much more likely in the media. If I was a doctor and saw an out of the blue study claiming a surprising result, I'd discount it accordingly. If I saw a causal explanation with evidence, I'd give it closer scrutiny and follow up if it seemed relevant to me. That is how research works in my experience.
I'll bet that a lot of folks just assume that anything they do will be taken at face value, without question or inspection. I also suspect that many "brought and paid for studies" are done this way.
For my own work, I generally assume that most of these studies are pretty much worthless, and tend to do some of my own homework before accepting them. Since most don't concern me at all; it's not a big deal.
Health is just one place this kind of thing happens. Software Development is absolutely rife with bad implementations. I am not in AI, but I have heard from a number of people that AI has a big problem with irreproducible results.
https://web.archive.org/web/20190926055757/http://www.jir.co...