
The incentives of academia naturally select for less reliable results (2016) - dom0
https://www.theatlantic.com/science/archive/2016/09/the-inevitable-evolution-of-bad-science/500609/?single_page=true
======
avs733
>Low statistical power is an obvious symptom of weak research.

No it isn't. It is a simple one but it isn't obvious. Low statistical power is
just one of many indicators of research quality. Qualitative research with
sample sizes in the single digits can be done well and research with large
statistical power can be done poorly. The measure of quality in research is
much more conceptual and cannot be reduced to rule based measures without
failing completely.

~~~
tlb
Statistical power includes effect size, so "qualitative" research (with large,
easily measurable effects) can have high statistical power with only a few
samples.

[https://en.wikipedia.org/wiki/Statistical_power](https://en.wikipedia.org/wiki/Statistical_power)

~~~
avs733
I'm really curious as to why you put qualitative in quotes.

------
nonbel
The proposed "solutions" of pre-registration and increase sample size are not
going to work because they are not fixing the underlying problem of NHST use
(where a researcher tests a nil null hypothesis of zero effect and then
concludes their favorite explanation is correct if they reject this null
hypothesis).

The solution is to go back to the old way of science (pre-1940):

1) Collect data and figure out what is repeatable and consistent (eg come up
with "natural laws").

2) Come up with theories that explain the observations from #1 that also make
_precise_ predictions about some different type of data.

Reward researchers who figure out methods that lead to reliable, repeatable
observations and those that make precise predictions (if your theory predicts
nothing more than a positive/negative correlation between two variables it is
too vague).

~~~
madhadron
Your solution is historically inaccurate. Statistical tests of significance
were created as a tool to improve the objectivity of _evaluating_ results.
Garbage in, garbage out still applies. Nor is it relevant. Statistical
techniques are a symptom, not an underlying cause.

Indeed, everything in the article or in discussions of this that I see is
always treated symptoms far from the causes. Let's do a quick five whys:

Why are publications not reproducible? Because positive results are being
published as fast as possible with little checking on them.

Why are they publishing so? Because the people involved need publications to
get out of their current transient position and secure another transient
position, with hopes of someday being able to settle down somewhere.

Why do they need to secure their next transient position? Because they have
been inculcated that this is the career to be in by their milieu and that is
the only way to stay in it.

Why is that the only way to stay in it? Because there are no permanent
positions that aren't at the very end of this gauntlet.

Why are there no permanent positions except at the end of the gauntlet?
Because this is a pyramid scheme built on an apprenticeship system.

Don't get distracted by all the other stuff. It's the fundamental structure of
academic science that leads to the rest of this.

~~~
ethanwillis
This is exactly why I am aiming to bootstrap my own lab outside of academia
and traditional research positions available private industry.

There's a lot of "BS" in these power structures that keep people from being
able to dedicate resources and a proper amount of time to generate quality
research.

I'm fairly "green" in my research career, but I've been in the game for a few
years now in the Biology/Bioinformatics realm. I've seen and heard stories
from plenty of colleagues about how when they first got started and well into
their first P.I. positions (and even until this day when they're 10-20 years
into their work) where they get abused by their institutions. Aside from
"publish or perish" there's also your institution skimming 50-60% of your hard
won grant funding to let you "rent" their facilities where you're fighting
over shelf space with other P.I.s Absolutely insane.

Seeing this I decided I can bootstrap my own lab for much less and have true
autonomy. For example a rough breakdown of my costs for setting up a
bio/bioinformatics lab for myself and a few friends:

Land Zoned for Light Industrial and Scientific Research? about $1000/acre or
less in my area. (hint: buy tax liened property)

CO2 incubators? Don't pay $4k-8k new. I can buy refurbed or liquidated lab
equipment from the unsustainable institution labs for literally pennies on the
dollar. Try $100-200 for this equipment.

High RPM centrifuges? Once again instead of $2k-5k I can spend $300 for a 15k
RPM centrifuge. and something like $20 for desktop centrifuges.

PCR machines? Try < $100 each. And I've even gotten "extras" with them in that
price range.

Compute power/I.T.? I can get 256 core, 1tb ram blade centers for about $1k.
Modern Core i5/Core i7 desktops for $80 each.

By my estimates I can buy the land, build a building(admittedly a very
utilitarian one), and equip a new lab for sub $30k. Now my only costs are re-
agents(if I'm doing research that requires it), power, and cheap land taxes.

~~~
analog31
You'll find that any large state university has dozens if not hundreds of
little businesses that sprout up in exactly this way. You see them in the
little business parks -- most of them lease rather than buy. You see them at
the trade shows.

It goes without saying that the idea is just like any other business venture,
with ways to succeed and to fail.

But a useful thing is to try and get a survey of what kinds of businesses are
out there, what they are doing, and how well they are doing. Next time you're
at a conference that has a trade show, go and look at the booths in the "low
rent district," because it will give you an idea of what people are doing.

Oddly enough I work for a F500, and we buy second hand gear. One reason is
that if somebody is advertising a particular instrument, it means that they
actually have it, and you can get it right away. With new gear, sometimes it
can take weeks or even months. Also, if it's under a couple grand, I can buy
it on my purchasing card without management approval.

~~~
ethanwillis
I have a business park a literal stones throw from my home. I thought about
going that route, unfortunately for me I have some higher power requirements
that I can't get in those units. I thought about leasing correctly zoned
warehouse space with the right power options but the cost is actually more
expensive in year 1 than buying and building in my area.

I'll definitely take a look at what some other small businesses are doing.
I've seen some up close, but the trade show run through is a great idea.

And yea. I buy stuff at auction within 300 miles of my area and drive the same
day to pick up any gear I buy. What kind of department do you work within
inside your F500?

~~~
analog31
I'm in "advanced" R&D, playing with technologies that will be in future
products. For my group, we often need something on a temporary basis. So it
doesn't make sense to invest in a brand spanking new machine. Plus, we might
be intending to modify it.

