Intro to stats says that studies are very difficult and that bias is everywhere. Reinforces how stats can be whatever you want them to be.
Larger Number of people in the sample is a good start. However, bias is also a human element. Therefore you need that study reproduced independently, multiple times.
Researchers don't like this because studies are expensive and hard, and their boss is telling them to publish and get funding. The system is broken.
For starters, experimental bias isn't really an issue, here... This was an observational study, with crystal clear objective criteria for coding the dependent & independent variables.
A cheap, simple observational study like this is merely a first step. No reasonable expert is claiming that this study proves anything worth making medical policy changes over. This study's sole purpose is to establish whether it's worth investing in further studies, or if we can just shitcan the idea now. Subsequent studies will cost more money, and involve larger samples, better controls, and get a lot more scrutiny from peers.
Time and money are finite resources. You have to have some kind of system for deciding where to spend those resources.
This is how medical science works. It's got a ton of pitfalls, but we still do it this way because so far nobody has come up with a better alternative.
> experimental bias isn't really an issue, here... This was an observational study, with crystal clear objective criteria for coding the dependent & independent variables.
I have no position on this particular study, but in general you can make your coding as simple and objective as you like and it won't do much about experimental bias. Objective processing of bad data isn't better than subjective processing of bad data.
All studies have some level of risk for bias... But that doesn't mean all studies have the same level of risk.
The essay you linked is very good, but it's actually illustrating my point... The mouse study ran into bias problems because they had to go and directly measure something in nature, and then turn that into a number. That introduces several opportunities for error.
But this SSRI/COVID study isn't doing that. They're literally just looking at patient records, and counting hospitalization vs current SSRI usage. They're not picking up mice to count ticks... They're exporting records to a spreadsheet, and summing columns.
Now, these guys might have problems with the quality of the records they're relying on... Who knows whether the records are accurate or not. But that's a problem of data quality, not experimental bias.
You might say "Who cares? Either kind of error undermines the results, just the same!" But it definitely suggests that the other guy doesn't know what he's talking about... Because if he did understand stats, he would have used the right terminology.
> The essay you linked is very good, but it's actually illustrating my point... The mouse study ran into bias problems because they had to go and directly measure something in nature, and then turn that into a number. That introduces several opportunities for error.
> But this SSRI/COVID study isn't doing that. They're literally just looking at patient records, and counting hospitalization vs current SSRI usage. They're not picking up mice to count ticks... They're exporting records to a spreadsheet, and summing columns.
> Now, these guys might have problems with the quality of the records they're relying on... Who knows whether the records are accurate or not. But that's a problem of data quality, not experimental bias.
Well, no, I have to disagree with this analysis. The mouse study described in that essay didn't run into bias problems, and the reason it didn't is that it was counting the ticks itself. But other tick-counting studies did have bias problems (due to bad methodology), and the Lyme-disease studies based on the bad tick-counting studies had the same bias problems (because they were using the bad data). The bias doesn't go away when you wash it through two publications instead of one. It's not a different kind of problem -- or even a different instance of the same kind of problem! -- and there's no reason to give it a different name.
You sound like you just took a stats 101 class and now are bragging to a bunch of adults about how much you learned in stats 101. Sincerely, someone working in the field for last 7 years.
Sounds like I don't know nearly as as you... I minored in stats 20+ years ago, but since then I do a lot more arguing about stats than actually using them for work.
You're kinda right about one thing, though... I am definitely talking down to OP, and a couple of other folks.
I'm frustrated. There are real problems in how science uses statistics, and it sounds like OP & co have heard about those problems. But the way they talk about this study, they're just throwing crap against the wall to have something to say. They don't really seem to understand the problems they're talking about, or how those problems apply to the study we're currently talking about.
I should change my attitude, and stop taking out my frustrations on these folks. If they're wrong, I'm probably not going to change their minds by insulting them.
> For starters, experimental bias isn't really an issue, here... This was an observational study, with crystal clear objective criteria for coding the dependent & independent variables.
Just a nitpick: it was not an observational study, it was a true experiment, because they controlled the treatment: it was assigned by researchers at random. It is by definition not an observational study which have a distinctive feature that it doesn't control for a treatment.
Larger Number of people in the sample is a good start. However, bias is also a human element. Therefore you need that study reproduced independently, multiple times.
Researchers don't like this because studies are expensive and hard, and their boss is telling them to publish and get funding. The system is broken.