
You Want to See My Data? I Thought We Were Friends - dnetesn
http://nautil.us/blog/you-want-to-see-my-data-i-thought-we-were-friends
======
kerkeslager
One big problem is that journals agree to publish studies _after_ they are
completed, which means that publication in prestigious journals is based on
the novelty of the result rather than on the validity of the study/experiment.
It's an absolute fundamental of science that you go into it with an open mind,
admitting that you do not know what the result will be. A good
study/experiment is not one which produces an interesting result, it's one
which is properly designed to answer the question it's asking. Evaluating the
quality of studies/experiments based on their results is an anti-scientific
practice which should be excised like the cancerous tumor it is. The best way
to do that is to accept and commit to publish studies/experiments based solely
on their design, _before_ the experiment/study has actually been performed.

~~~
kashyapc
I like what you're saying. Do you (or anyone on this thread) have any
comments/thoughts on the following?

During a conversation with an academic researcher (non-Computer Science)
friend, when I brought up the topic of data sharing, especially in context of
the infamous "replication crisis", they have their reasons not sharing. I'm
loosely paraphrasing here, while trying hard not to misrepresent/misremember
their exact views:

"I want to protect my data; I don't have enough time to present my data in a
presentable form; and more importantly, they'll just steal my idea and go
present it as theirs—and I might lose funding" ... and so on.

I can empathize with the academic pressure of "publish or perish". And not
least of all, "need some food on the table, and roof over my head".

But I still wonder, there must be other effective ways to gently persuade a
said researcher (especially in the 'soft sciences'—I'm not using the term
derogatorily) on the importance of sharing data that allows reproducibility of
a given experiment?

~~~
kerkeslager
To be honest, I don't think this is going to be solved from the bottom up. I
think a lot of scientists know that they are making compromises between doing
science and pursuing their career. But we can't reasonably ask people to do
better science when better science means living on an adjunct salary for the
rest of their life. The change has to come from publishers.

The replication crisis will continue until publishers incentivize replication.

~~~
kashyapc
One researcher-friend did admit of the compromise you speak of, and the
internal struggle that goes with it.

What remains is the "small matter" of persuading the publishers to action.

~~~
kerkeslager
That may work with some publishers--MIT Press in particular seems particularly
amenable to change, probably because their funding isn't dependent on the
popularity of their publications. But the profit-motivated publishers who have
a "working" business model aren't going to be persuaded to change it easily.
We've watched hundreds of brick and mortar video stores and newspapers go out
of business rather than step into the future--if history is any teacher here,
the Elseviers of the world will go bankrupt rather than change.

And that's a good thing. A revolution in academic science led by Elsevier
would be like King George leading the American Revolution. Real fundamental
change isn't going to come from the profiteers who caused the problem in the
first place.

------
James_Henry
A lot of the problems in academia, I believe, come from incompetence, a lack
of questioning or at least of doubting others' competence and your own
competence, and reliance on unsound ideas about scientific methodology. Andrew
Gelman has some good thoughts here that I feel are related:

[https://statmodeling.stat.columbia.edu/2020/07/29/the-
crooks...](https://statmodeling.stat.columbia.edu/2020/07/29/the-crooks-get-
the-headlines-but-the-real-problem-is-bad-science-done-by-non-crooks/)

Some people will have the best of intentions, they will really be doing
"science" out of the goodness of their hearts and out of their desire to help
mankind, but they'll end up publishing trash that gets acceptance and
sometimes even praise.

What are the incentives that need to be instilled to fix the problems we
currently have? I'd say we need to incentivize competence and humility. How? I
don't know exactly, but values like these seem to be able to be instilled
through cultural practices and traditions.

Also, especially humility seems to be lacking from many cases of bad science.
If people accepted criticism and accepted that they don't really understand
all that much, I believe scientific quality would improve. You do have a lot
of reasons to not be humble in academia though, as this comic lays out.

~~~
bjornsing
> I'd say we need to incentivize competence and humility.

Problem is I think that the competent are few, and when the cultural norm is
that they must be humble then they stand no chance against the many
incompetent.

IM(H)O: Science shouldn’t be humble in the face of non-science. As long as it
is it will lose. The idea of conflict free great science is a pipe dream. We
need a culture that accepts (intellectual) conflict.

~~~
gregmac
> the cultural norm is that they must be humble then they stand no chance
> against the many incompetent

I don't think this is a cultural norm so much as just the Dunning-Kruger
effect [1] in play. People who are highly competent still realize there is
much they don't know, and that makes them humble. I suspect if you go and find
someone widely recognized as an expert in pretty much any field, and ask them
if they know all there is to know, you'll find no one says yes.

[1]
[https://en.m.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effec...](https://en.m.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect)

~~~
bjornsing
I don’t know... I often feel like we’re talking about many kinds of humility
at once. You seem to be talking more about “epistemological humility”, while I
was talking more about “social humility”.

Was Richard Feynman humble?

~~~
James_Henry
Yes, from what I've read of him, he was humble in quite a few ways.

~~~
bjornsing
I agree: He was very humble before reality (as all good scientists have to
be). But I argue he wasn’t very “socially humble”...

------
Sebb767
I generally agree, but their solution (fund boring research, publish only in
journals with high standard) is in direct contrast to what they stated
earlier. It's basically saying yes, we have bad incentives, but we could
ignore them. That's not going to happen; shiny new research _will_ attract
people and funding. Especially more than "boring" _what we found before was
indeed a finding_ -research.

Now, I don't have a good solution either, unfortunately. What might work is
that we require replication work for a PhD or have a certain percentage of a
journal dedicated to verification. That, combined with some meta-studies to
reward people with citations for replication, might work without fully
swimming against the current.

It's a hard problem, really.

~~~
setgree
Tyler Cowen writes [0] that the most important question in economics, to him,
is

> how do differences of culture — however defined — interact with traditional
> economic mechanisms involving prices, incomes, and simple comparative
> statics? Are those competing explanations, namely cultural vs. economic?

Richie's answers are mostly focused on changing the culture of science, and
while there are lots of ways we could change the incentives, none of them
would be pretty.

Example: let's say we want to more closely align research and actionable
results, e.g., a product a company can use (Brian Armstrong argues for
something like this [1]).

Solution: radically reduce public funding for scientific research and for
university education as a whole (in line with Bryan Caplan's arguments in "The
Case Against Education" [2]). Academics, who would be many fewer in number,
would then have to get more of their funding from companies, who (presumably)
would:

A) guide them towards asking market-relevant questions, and

B) have a clear incentive to check the data, re-run the code, etc. -- so that
the product they built based on that research didn't flop.

I think most people would recoil at this proposal. But that's what comes to
mind when I think about fixing the incentives rather than the culture.

P.S. Small nitpick: Richie gives an _example_ of a perverse incentive in lieu
of a definition.

[0]
[https://marginalrevolution.com/marginalrevolution/2017/01/im...](https://marginalrevolution.com/marginalrevolution/2017/01/important-
unsolved-problems-field.html)

[1] [https://medium.com/@barmstrong/ideas-on-how-to-improve-
scien...](https://medium.com/@barmstrong/ideas-on-how-to-improve-scientific-
research-9e2e56474132)

[2]
[https://en.wikipedia.org/wiki/The_Case_Against_Education](https://en.wikipedia.org/wiki/The_Case_Against_Education)

~~~
michaelt
If you think funding from industry makes academics' results more robust, you
might want to look into tobacco-industry-funded research on cancer, oil-
industry-funded research on global warming, and ridesharing-industry-funded
research on drivers' working conditions.

~~~
JackFr
No one thinks "research on drivers' working conditions" is science. Manifestly
self-interested nonsense like that is relatively easily ignored. And if that's
the trade we make to get Bell Labs, Xerox Parc and IBM Research, I'll take it.

------
beagle3
I don't know if the comic properly represents the book, but all of the
suggestions are somewhere between ridiculous and useless.

First, about identifying bad research: The "extraordinary claims require
extraordinary evidence" is already practiced. It's not that "we've overturned
quantum theory" articles that are causing problem - those are quickly and
effectively shot down. And it is rare that the "perfectly aligns with a
political interest" can be applied. The only actionable one is "see what
others think about it", and it's no panacea either.

Bad and fraudulent science like the recently retracted Surgisphere covid paper
is abundant. I was trying to track down the origin of the "reduce salt intake"
and "limit egg consumption to no more than 2 per day / 2 per week"
recommendation in the past, assuming there was hard science behind them. There
isn't; and indeed they're slowly being reversed everywhere - but they were
prevalent for half a century, with a lot of other research taking them as
axioms.

The remdesivir trials have been p-hacked to death - anyone who took an
interest was seeing it happen in real time - yet, the scientific community
turns a blind eye.

The ketogenic diet is vilified in every mainstream media and most nutritional
"science" publications; the headlines are rarely inline with the the actual
results, but that's what people remember (Not that nutrition science is really
science)

And the other recommendations about fixing it are comparable to "lets solve
evils of the US 2-party system! All we have to do is make those two parties
vote to take away their power". Academia and science publishing are where they
are now because it benefits essentially all the incumbents (at the expense of
the rest of society).

The problem description (at least in the comic) is good. Any suggested action
... not so much.

~~~
joppy
I think the suggestion that "universities could change their hiring policies"
is a very good recommendation. There are a lot of problems in academia (bad
research, lack of diversity, pressure to publish) that could be reduced if
universities changed their hiring policy to something further away from the
metric of (number of papers published since receiving PhD) / (time since
receiving PhD). Of course no university hires explicitly on that metric, but
many of the metrics they use are not far from that, mixing in quality of
journals and norms for the field, etc.

Each of the suggested actions, taken together, would seem to have a very
positive improvement on the status quo. Would you care to explain why the
suggested actions would be "somewhere between ridiculous and useless?"

~~~
meow1032
Not OP here, but my issue with the recommendations are that they've pretty
accurately listed a whole bunch of mostly structural problems with academia,
but all of the suggestions boil down to "we all just need to try harder". You
can _say_ something like: "journals need to demand higher standards" but what
incentive do they actually have to do so? Then you can counter with
"scientists could vote with their feet", but what incentives do they have to
do that?? You're asking people to consider seriously damaging their career for
some nebulous quality metric.

Frankly, having worked in academia long enough to see at least a couple shifts
in culture, the only thing I can see that comes out of this is a couple more
things get added on to the ever growing checklist of publishing a
paper/submitting a grant application.

I think we need to get away from the sort of thinking where large structural
problems can be solved by tiny incremental improvements. If you really want to
solve the problem, one or more of [Granting Agencies|Journals|Universities]
has to be completely torn down and built back up.

~~~
joppy
It seems to me, still, that a lot of these problems you bring up can be
addressed by universities changing their hiring policies. Which makes sense:
academics ultimately rely on universities for their income, and so it is the
hiring policies which are setting the perverse incentives. And I don’t think
changing hiring policies would be an incremental change, it would be a huge
change (and not likely to be made by any university any time soon, since
students rank universities on similar metrics to how universities hire staff —
a prestigious university will lose prestige even if it changes its hiring
policies for the better).

~~~
meow1032
> academics ultimately rely on universities for their income

Sort of, a huge portion of income is from grants, particularly after the first
few years from being hired. More importantly, a huge portion of the University
income is from grants. When a researcher recieves a grant, there is an
"overhead" percentage that goes to the University. Universities hire, in part,
to maximize those overheads, which means getting the researchers with the best
chance at getting big grants.

Changing the hiring process may affect how PHD students act, but once they're
"in the system", they are subject to all the same problematic incentives.

~~~
tejtm
> academics ultimately rely on universities for their income

In my decades at it (digital side of bioinformatics) the cash flow is in the
other direction.

------
mabbo
Zach Weinersmith's ability to tell compelling non-fiction via comics is
something I truly love. He manages to take what the person is saying, convert
it into a comic form of them saying it, while adding humour. And throughout
the process, _what_ is being said does not seem to be degraded at all. There's
also a level of openness we all seem to have to something that comes at us as
a comic rather than hard text.

I don't even think we have a good word for what this practice is, but I'll go
with "Art" because it takes a lot of that.

His book on Immigration[0] is a large-scale version of this skill in practice
and I suspect a lot of HN readers might enjoy it, regardless of if you agree
with his points or not.

[0][https://www.amazon.com/Open-Borders-Science-Ethics-
Immigrati...](https://www.amazon.com/Open-Borders-Science-Ethics-
Immigration/dp/1250316960/)

------
ajuc
Is there something like negative-citation-index?

Where you spread refuted papers out through citations to other scientists and
newspapers.

It could be included as a factor when hiring scientists.

And of course the person who refuted a false paper should receive the
citations of that false paper. It's only fair.

~~~
crankishness
Expanding on this, suppose there is an anti-journal, tentatively titled
'Journal of Bad Science', which features thoroughly refuted, bad-faith papers.
Not just rejected papers, as the reasons can be as innocuous as a few spelling
mistakes, but clearly and unambiguously bad research.

This would form the basis of the Crank Index of a paper, which can be
simplified into a stoplight system: Good research with good sources is GREEN.
Getting featured in the Journal of BS earns a paper the esteemed distinction
of a blaring scarlet RED, citing a RED paper will mark a paper ORANGE, citing
ORANGE research leaves you YELLOW... Throwing together lots of ORANGE and
YELLOW citations will nudge your paper up the spectrum towards RED.

This would incentivize researchers to not only care about the quantity of the
citations they share with each other, but to be extremely vigilant of the
quality of those citations as well.

~~~
fsflover
Let’s say I cited a “red” paper and explained how and why they were wrong.
Does my paper become “orange”? I hope not, but that would require a lot of
rigorous manual verification by the journal editors...

~~~
ajuc
Ideally papers would start citing using a new format that makes explicit the
dependencies between papers.

For example:

    
    
        refutes: ...
        expands on: ...
        depends on: ...
        alternative approach to: ...
    

etc.

------
sradman
This comic by Zach Weinersmith summarizes Stuart Ritchie's recent book
_Science Fiction_. To combat the problem of low-quality science papers, one of
the panels suggests:

> [journals] can demand scientists share their data, and to prove that they've
> written down their analysis plans before they touch the data

I wonder if this doesn't gloss over a deeper underlying problem: journals have
traditionally assumed the copyright of the paper. Journals themselves have an
incentive to obfuscate and protect the underlying data and content.

Ultimately, any complex system or institution will be more susceptible to
gaming when it is mature and its value proposition clearly established. Anti-
gamification is hard to design into the early stages of a system when it is
needed most.

~~~
wizzwizz4
[deleted]

~~~
throwanem
The comic is the posted article.

------
Vinnl
It's an excellent analysis of the fundamental problems in academia, but
"people should just act against their incentives" isn't really a solution.

It really is the incentives themselves that are the problem: just looking at
number of publications and citations (or even: citations of articles _in the
journals that your articles happen to be published in as well_ ) when
determine who to fund or hire.

The problem there is that we _have_ those metrics, are relatively quick and
easy to obtain, which are accepted because they are what's been used so far -
even though plenty of research has pointed out their flaws yet [1]. And
anything new that is proposed as a replacement of those metrics (whether other
metrics, or other systems of evaluation) is dismissed for not being proven to
live up to a standard that the currently used methods do not either, or for
not being available quickly or easily enough. (Which is reasonable - e.g. it's
not viable to read and properly evaluate all research of your applicants.)

(Disclosure: I do volunteer for a project,
[https://plaudit.pub](https://plaudit.pub), that tries to offer an alternative
nevertheless.)

[1] [https://medium.com/flockademic/the-ridiculous-number-that-
ca...](https://medium.com/flockademic/the-ridiculous-number-that-can-make-or-
break-academic-careers-704e00ae070a)

------
adamnemecek
You need a better science publication platform. Like arxiv and github
combined.

~~~
sradman
Arxiv, github, and a self-publishing style platform that supports reproducible
digital artifacts, i.e., the published paper. IIRC, many flawed papers were
the result of data errors saved in a spreadsheet.

~~~
adamnemecek
And better citations. I want to be able to link to a particular sentence of a
particular version of a paper.

Also pull requests.

~~~
mattkrause
Could you explain what the sentence-level citation adds?

I've heard several people ask for this, but never understood why. Most
citation formats let you include page numbers; you can usually work in other
location information ("See Foo et al. (2020)'s Figure 3A") too.

~~~
adamnemecek
Maybe but most people dont do it.

------
amatic
I think the core problem is our ignorance of psychology - we don't know how
humans really work, what makes us tick, what are the 'incentives' that should
be put in system design to move toward better science, and whether
'incentives' are even a good conceptualization of human motivation. We will
not fix science until we understand what makes scientists behave as they do,
and until we figure out how to design systems for humans. Though, Maybe we
stumble upon a better system via blind variation in system properties and
selective retention, based on some novel metric. Scientific psychology is
rather weak in explaining and predicting how humans will behave.

------
goatinaboat
Nobody in the world gets to do the “fun” part of their job more than a
fraction of their time. I don’t think scientists are uniquely hard done by
here. They are enormously more privileged than the vast majority of people,
being funded to do something purely speculative, it’s not too much to ask them
to publish it so others can benefit from that spending too - which largely
comes from taxpayers doing less fulfilling jobs.

------
t0mbstone
I can't help but wonder how many of the issues with scientific papers couldn't
be solved with technology. The notion that science is so deeply rooted in
antiquated concepts like paper journals and academic constructs like tenure is
absurd when the internet has been around for as long as it has.

For example, imagine if scientific papers were voted up or down by a
community, kind of like stack overflow.

Or imagine if scientific papers had to publish all of their source materials
and instructions for replicating the experiment, and there was a system for
tracking and showing whether or not the experiment had been validated or
disproven?

What if you "game-ified" scientific papers and gave people points for
publishing, but also gave people twice as many points for disproving a paper?

Imagine if we had a platform for tracking scientific theories and experiments
that was a combination of democratic/meritocratic administration (like
wikipedia), change logging/tracking (like github), and reputation management
(like stack overflow)...

------
gentleman11
The fixes they propose are good ones, but aren’t grounded in reality. They
ignore how we got here in the first place. Citations matter as a proxy for
importance. Negative studies are inherently uninteresting. Companies fund
groundbreaking work because they want to be associated with a breakthrough.
Scientists publish in bad journals because their careers depend on it - it’s
an entire lifetimes work to get tenure and research grants. They can’t just
throw that away. Journals publish lousy studies because they don’t have enough
good ones - the journals will not self destruct in order to slim down for us.

To fix the system will take a more honest look at the incentives of the
people/institutions who create the incentives, and so on.

------
fritzo
paraphrasing to emphasize irony:

"Science should be based on solid data: published, auditable, peer-reviewed
numbers. Data is good, data is objective, data is truth.

"Academic hiring is broken. We can't base academic hiring on numbers because
people game the numbers. In academic hiring we need to be subjective, to
evaluate the intrinsic merit of each researcher. Data is corrupt, data isn't
sufficiently subjective, data is flawed.

------
einpoklum
A couple of years ago, Prof. Michael Stonebreaker gave a talk in ICDE (IEEE
Intl. Conference on Data Engineering) 2018 in Paris on the problems of the
pursuit of the "LPU", least publishable unit of work; and his impression that
few people pursue deeper and more significant work because of this and other
factors. If you can find a summary or a recording of that somehow, it's
worthwhile to listen IMHO.

------
csours
What you like to see at the top of any article covering the press release of a
study?

Something like an infobox with P-factor, whether it was pre-registered, sample
size, funding organization, double-blind, etc?

This comic tackles the Academia side of things, but a lot of that motivation
comes from press coverage. If the press has better capabilities to be critical
of bad studies, Academia will give less credence to the same.

------
drummer
Nothing proves that comic more than the current covid-19 'pandemic' which is
largely based on fear and BS (bad science). The doctors and scientists that
actually make sense get censored into obscurity while sensational and fear
agenda promoting info gets published.

------
ptero
There is no perfect solution, but requiring access to both the data and full
methodology for experimental sciences should help.

Even problem sof cherry picked data would be partially exposed eventually; and
eventual exposure is still a very effective deterrent in science. My 2c.

------
abdullahkhalids
One change that would help and easy to implement is reporting, for a
researcher/paper, citations excluding self-citations.

This doesn't take care of citation rings, but does move the needle towards
reporting the actual value of a paper/researcher.

------
giardini
OK, so "Science Fictions" was just released and is full of cartoons and we
have an obvious promotional push going on. But I simply must recommend the
(possibly) more mundane (no comics) but nonetheless excellent David H.
Freedman book:

 _" Wrong: Why experts_ keep failing us--and how to know when not to trust
them _Scientists, finance wizards, doctors, relationship gurus, celebrity
CEOs, ... consultants, health officials and more "_

[https://www.amazon.com/Wrong-us-Scientists-relationship-
cons...](https://www.amazon.com/Wrong-us-Scientists-relationship-
consultants/dp/0316023787)

which begins with an interview with John Ioannidis and goes on to discuss in
detail why so many academic (and expert) publications are wrong and how they
got that way.

~~~
sradman
Stuart Ritchie's book _Science Fictions_ was not illustrated by Zach
Weinersmith, only the article was.

~~~
giardini
Oops! My bad! Then I recommend Freedman's book even more!

------
im3w1l
I kindof think science isn't fixable. Maybe we can get it a little bit better
but we just have to live with the suck. At least it makes some progress
despite all the flaws.

------
pvaldes
LOL, The annelida part killed me

------
annoyingnoob
All of these problems plus low pay, why would anyone go into academia?

~~~
annoyingnoob
You can down vote me but I asked a serious question. There are certainly
science careers that are not in academia.

Am I missing something? Is there great pay before you've taken a lot of time
to move up the ladder, assuming you are fortunate enough?

Help me understand your reasoning rather than just down voting me.

------
kkotak
I think all scientific papers should be written in this format.

