Hacker News new | comments | show | ask | jobs | submit login

I came to the biological sciences trained as a chemist. As I began working on my Ph.D. and variously encountered papers on cell signaling research (the field that much "cancer research" would fall into) it was blindingly apparent to me, perhaps because of my chemistry background, what was going on...

Cell Signaling (and much of Biology) research is in its Alchemy phase. Now, alchemy often gets a bad rap as being completely worthless, snake-oil type stuff, but this was not the case at all. Rather, individuals were working on an area about which almost nothing was known, and (more importantly) for which the key central organizing laws were not yet revealed (for Alchemy: the atomic theory of matter and chemical bonding, for Cell Signaling: how individual proteins and small molecules interact). Furthermore, the goals were lofty and almost certainly unattainable (for Alchemy: turn base metals into gold, for Cell Signaling: cure cancer), driving people to do "rush" work. It's not that the results are completely invalid, or that the experiments are useless. It's just that everyone feels like they're so close to a solution (Alchemists were way off with Phlogiston, and I'll bet Cell Signaling researchers are similarly clueless as to what really matters) that no one takes the time to step back and synthesize the results in an attempt to understand the forest from the trees.

Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev.

From experience, the fact that 89% of these "cancer research" papers are not reproducible almost definitely has less to do with fraudulent data and so much more to do with crazy complex experimental setups that end up probing a half-dozen experimental variables all at once (without the researchers even grasping that this is going on).

Yeah, publish or perish sucks, but what sucks more is the death of basic science research. The Alchemists eventually became Chemists because they refocused on core principles (atomic theory, bonding) and forgot about the lofty goals (turn lead into gold)...

...but try telling any politician that they should decrease cancer research funding and refocus on genetics, structural biology, and evolution research. I'd love to know how they respond.

>Cell Signaling will, eventually, have its Joseph Priestley, its Dmitri Mendeleev.

That's very optimistic, but not necessarily true (even ignoring the question of when). Priestley succeeded so well because he developed simple, reproducible experiments (expose mercury oxide to sunlight to produce oxygen etc.) that he was happy to share for no financial gain (even traveling to others' labs to help them reproduce his work [1]). The current environment of paywalls and competitive grants seems hardly conducive to the rise of similar figures.

[1] http://en.wikipedia.org/wiki/Experiments_and_Observations_on...

Part of why it may not be true is that we lack the language or conceptual vocabulary to describe how living systems really work.

When we engineer things, it's one-part-one-function. Living systems are every-part-every-function network graphs with weighted edges that are subject to dynamic reconfiguration.

A cell isn't a computer that runs "code" in the form of DNA, nor is it a "machine" as we understand it. A "structured cloud of probabilistic quantum interactions among molecular nanomachines" is a bit more accurate.

Until we conquer these meta-problems, we won't understand the cell or the genome. I don't think they can be understood classically -- and I don't mean classically in the sense of omitting quantum mechanics. (Though that's true too.) I mean classically in the sense of linear fixed-relationship cause and effect machines.

What do paywalls possible have to do with the reproducibility of results? Anyone who has access to a lab capable of even attempting to reproduce such research is either 1) at an institution that has access to those journals or 2) can use google to find the papers anyway. Hopefully both.

The fact that tax-payer funded research is put behind paywalls is a travesty , but it seems wrong to claim it actually hinders researchers from doing anything given.

    > What do paywalls possible have to do with the reproducibility
    > of results? Anyone who has access to a lab
Actually, there's no institution subscribed to every journal. There are, in fact, many papers inaccessible to even the most funded labs. Spotty coverage.

No, no institution subscribes to every journal. However, see point 2. Given the title and author, how many of those paper's can't you find with google / google scholar(or emailing the author).

Paywalls for publicly funded research our wrong, but the cases when it prevents people at research institutions from doing research seem to be tiny.

I'm published in a highly-ranked journal that our university doesn't have access to.

Sorta funny.

Can you find the paper online by searching for it?

If they were open, you would have an online service that could automatically point contradicting studies, and even make suggestions.

Even if they are closed, you can link to the papers (unless the DMCA metastasized again). Granted, someone can't make an automatic tool to find such studies, but that would be hard anyway and presumably the authors are aware of the study they are refuting and could tag them.

Now, people trying to make policy decisions could not read or evaluate the studies. If we assume they are capable of doing so in the first place( i.e. their not Lamar smith), then we have an actual argument for getting rid of paywalls. I still think the best one is its a waste of money and a tax on universities/ grants.

I don't understand why science should be a closed guild in the first place.

Because these days it is institutionalized, and in our society that means it is funded by capitalistic entities, i.e. corporations.

In other words, the profit motive dictates that making money to enrich a few individuals is more noble than sharing knowledge for the good of society.

Science is funded entirely by corporations? Tell me, where can I buy stock in the NSF, DARPA, NIH? Or where can I find the articles of incorporation for the MacArthur Foundation or any of the other non-profits that also fund research. Or which corporation is paying my PhD stipend --- I really ought to short their stock since they were dumb enough to hire me.

Those are government entities, and the government's parent company is the corporatocracy that owns all the politicians and media.

Science is a business in the service of global capitalism. Truth hurts.

I will give you the MacArthur foundation and certain other non-profits. But most non-profits are also operated by the corporatocratic elite, like everything else on this planet.

You're right - if you define 'funded by corporations' sufficiently loosely (to include everything from governmental bodies to universities to charitable foundations) that the statement 'science is entirely funded by corporations' becomes a tautology, it becomes clear that science is entirely funded by corporations! I for one am shocked and appalled.

You're just being pedantic. The thrust of his comment was clear while not perfectly correct. "Fund" is well understood as a euphemism for "control"

It would be ideal if science were "funded" (read: controlled) by the public in the interest of the public good, but in reality it is controlled by the corporatocracy.

Is Goldman Sachs "funded" by the government or is the government "funded" by Goldman Sachs? The best understanding is that the government is a corporate subsidiary of Goldman Sachs, a profit-centre that Goldman Sachs invests in and controls by means of investment, and those investments reap profits.

Technically you could say that GS is funded by taxpayers, but funding implies both investment and control. So it's more accurate to say that GS funds the government, because this makes it clear where the locus of control is. Citizens are indentured servants that pay tribute to GS via taxes. Academics are indentured servants that receive money and position in exchange for their services to the subsidiary of GS called NSF.

Having said that, I think many foundations are genuinely independent of GS control, but they still can't be said to be funded by the public for the public. They are just altruistic private parties as opposed to the megalithic non-altruistic faction that is the corporatocracy.

I keep coming back to this essay called "Can a Biologist Fix a Radio?": http://protein.bio.msu.ru/biokhimiya/contents/v69/pdf/bcm_14...

The thing that keeps coming to mind is that it's easy to hand-wave pseudo-understanding of systems using enough jargon and vague diagrams. And if you use enough jargon and cloak things behind complex methodologies (the mechanisms to even begin investigating how cellular machinery works are enormously complex) it's easy to hide a lack of understanding. And I can't help think a similar sort of problem exists in software. The tools and terminology we have for explaining and understanding software systems leaves a lot to be desired, and one of the ways that is revealed is in the diagrams we use for describing systems. At a fairly fine grain we can use UML diagrams, but these tell us almost nothing and in complex systems simply become a giant hairball. Typically we end up describing a system as a series of layers, or as an interconnected set of services, but I can't shake the notion that such diagrams are too much like Figure 3a in the essay above than like 3b, they are the crudest sketch of a system, they don't touch on the nature or interrelation of the components.

> It's not that the results are completely invalid

Especially if only 11% of them are reproducible, they're only 89% invalid.

I'd go so far to say: If the reproducibility rate is 11%, you're not doing science, you're just pursuing funding.

You're assuming that only "reproducible" results are valid, which is incorrect. Results that are reproducible within a lab, but not when attempted by other people in different settings indicate that there's an unaccounted for variable, and that falls into the 89%. There's still something valuable there, it's just that we don't yet know all the variables.

We could just give up, or we could try to continue study of something incredibly complex. Given that we are gaining some ground, it's clearly not a useless endeavor, it's just an extremely difficult one.

If its not reproducible it is not science.

Define reproducible, in the terms of the precise actions that people take. Is it reproducible if the same scientist repeats the experiment in the same lab and gets the same results? Because that's the current bar of reproducibility, and that 89% that is not "reproducible" certainly passed that bar.

What was being tested was a different lab, with different materials, trying to get the "same" results, for some definition of same. If you give 100 programmers an algorithms book, and tell them to produce code for a binary search, and only 25% of the programmers are able to make something that works, does that mean that binary search is only 25% reproducible?

If five different companies benchmark five different web frameworks for their application, and come to 2-4 different answers about which one is the 'best,' does that mean that the benchmarks are not reproducible? Of course not.

What's being highlighted here in this study is the extreme diversity of biological models. And one doesn't necessarily expect exact reproducibility in other people's hands, because we simply don't have technology to characterize every single aspect of a biological model, and it's impossible sometimes to even recreate the exact same biological context. Is something "reproducible" if it means that it replicates in 5% of other cell lines, 25% of other cell lines?

>Is it reproducible if the same scientist repeats the >experiment in the same lab and gets the same results? >Because that's the current bar of reproducibility, and that >89% that is not "reproducible" certainly passed that bar.

I don't follow you here. The above does not seem to be the current meaning of "reproducible":


The same person doing the same experiment is repeatable, not reproducible. And I don't believe even the repeatable bar has been met, as very few projects have funding to do the same experiment twice.

The fact that a given investigator can "repeat" his experiment have very low weight among professional scientists, because we are all human. Irving Langmuir's famous talk about Pathological Science, and especially the sad story of N-rays, is a warning to every scientist.



So there is a theory that if something doesn't reproduce it's because the other guy was just incompetent, and that may be the case, but just like everyone wants to believe they're above average, everyone will want to go to the theory that the other guys just aren't any good, when I suspect that that will be much less of a factor. At any rate, when you start trying these drugs on the wide diversity of the patient population, if they're not super robust, they don't be of much use anyways.

Huge swaths of astronomy are functionally unreproducible. We can argue about the math but many phenomena exist as a single example and/or that's basically static on our time scales. The best we can do is see if the math seems to produce similar looking structures when (sparsely) simulated.

There's a difference between an observation and an experiment. Lab experiments should be reproducible. The fact that astronomical events are not reproducible does not make the study of them unscientific, but it also doesn't imply that lab experiments should be one time events.

Would there be a problem with just calling it something else other than science then? I don't see the need to bend definitions of words to account for inconvenient circumstances. I am a programmer and do hard stuff, I don't require people to call me a scientist. Mathematicians do hard stuff, they don't complain that they're not called scientists. Engineers, too, are not scientists. It's not pejorative, just a statement of fact. If what you say is correct, then what is the issue with just saying astronomy is not a scientific field?

Well attempting to exclude astronomy/astrophysics from the umbrella of science would be bending the definition far more than the current status quo.

Part of this is that science isn't about experimentation, it's about observation. We perform experiments when possible so that we have more stuff to observe, or more controlled events.

Much of astronomy, atmospheric physics, geology, medicine, the "soft" sciences and I'm sure plenty of other "hard" fields are at the mercy of certain phenomena having sweet FA for data points. And I'm sure they all do what astrophysicists do: make sure that what we do have plenty of data for works; make our extrapolations with as few assumptions/rounding errors as we can; and revisit existing models anytime we find a new data point.

It's as scientific as anything it's simply going to take longer to sort out in some cases.

I once had a very long and intense argument with a guy who was offended because I thought that I wasn't a scientist even though I studied Software Engineering (not even Computer Science).

Apparently people take that crap seriously.

I think this is a straw man. Astronomers, astrophysicists, etc. go to enormous lengths to address these issues over time and are well aware of the shortcomings of their work. When black holes were predicted, none had been observed. I'd suggest that astronomy is a terrible place to make claims about irreproducibility.

Cosmology on the other hand...

It's really critical that we don't confuse research that's nor reproducible with fraud. Very few scientific theories survive unmodified over time, so lack of reproducibility isn't a criticism and we really need to move the debate past this. Every theory is expected to be inaccurate as it only explains the data using the understanding of the time, but this isn't an indictment of the research or the researcher and studies of outright fraud indicate that that actually only happens around 1% of the time.

Reproducibility isn't about calling out people whose work isn't reproducible, it's about identifying and promoting the most robust stuff.

There are lots of parts of science that can't be experimented on (e.g. astronomy). Even for those parts that are experimental, just because you're wrong doesn't mean you're not doing science.

The current test (if I remember my philosophy of science correctly is about falsifiability - it's not science if its claims can't be disproven. From this perspective, bad experiments are still science - someone predicted that similar experiments would behave similarly, and their prediction was falsified. This is how science is supposed to work.

It gets problematic when any failure to reproduce instinctively gets explained away as experimental error on the part of the second experimenter. Even worse is when experimenters (as in this case) work to have failures to reproduce hidden from the scientific community (the authors of this study had to sign contracts that they would not identify specific failing studies before they were given the necessary data about experimental procedure.

That's not true. Part of science is doing experimentation. If an experiment doesn't reproduce, you need to find out why that is. The hypothesize and test part of the scientific process is every bit as much science as the rest, even if your tests show that an idea is wrong, or that more is going on then you thought.

What about the experiments run by using the LHC? No other organization has a similarly sized particle accelerator, so by your definition it is not science because it cannot be reproduced elsewhere?

If 11% of the papers are reproducible and people are trying to reproduce the results then your still making progress. Just the slow and expencive kind. Considering how complex the subject matter is I don's think it's reasonable to expect anything else.

The problem is people are not trying to reproduce results which harms the field and slows everything down.

Actually very few people are trying to reproduce results. It is a lose-lose situation. Either you confirm the previous results which won't get published or you can't confirm them which means you are either not as competent as the original researcher or you have to embarrass your colleague and bring shame on your profession. Neither of which gets you published or helps you career or gets you more funding. The incentives are all screwed up.

These Amgen researcher had to sign agreements that they would not publish the results of their attempts to reproduce these experiments before they could get enough detail to attempt to reproduce them. Clearly this is not how science is supposed to work but it is exactly how businesses operate. Very sad.

After seeing how much of the research is done, I'd agree with this more.

This is a problem that is getting worse as funding is getting cut more, people feel they have a need to get a paper out, regardless of the results. You get positive results, make a story up about it, then run to publish it before trying to reproduce it or look further into the data. While this doesn't happen in every lab, I'm unhappy to say that I've seen this happen in many "high impact" labs.

Why do you say funding is getting cut? NIH's budget has almost doubled in the last decade [1] and many of the other funders have seen similar growth as well as new funders appearing every year.

I don't think the problem is lack of funding but screwed up incentives. When medical reaseach became focused on funding the quality of the results suffered. And if the vast majority of landmark cancer research can't be reproduced much of that money was wasted.

The solution will require a huge cultural change which may be impossible. However step one is recognizing the problem. And some efforts are already underway such as journals like PLoS which publish negative results and more recently The Reproducibility Project and Reproducibility Initiative [2,3]. Still it will be difficult.

[1] http://officeofbudget.od.nih.gov/pdfs/spending_history/Mecha...

[2] http://www.openscienceframework.org/project/EZcUj/wiki/home



Already funded grants are getting cut ~20% across the board. There is a ton of cuts going on right now to the NIH budget, google it and take a look. This has been happening for years now, trying to get an RO1 (large research grant) is becoming more and more difficult and it isn't helped by the constant changing in requirements.

Every other point I agree 100% with, the culture change has to happen. Nothing is impossible, and it just takes the right people to make the right things to happen. Everyone recognizes the problem, I can't tell you how many times I hear people complaining about the same problems over and over. The problem is that they aren't taking action, and with no action, nothing is going to take place. While there are many efforts in place (my project being one of them, http://omnisci.org), they need to be implemented properly. The same rule of startups applies to science. The ideas/concept means nothing with proper execution.

While the culture change may be slow, the academic world is having a really hard time keeping up. NIH is also fighting to stay afloat. I have a few friends who work as program officers and they really have a negative outlook on the future of research funding.

You are right about the sequester cuts. I was looking at the annual numbers of the NIH front page which didn't include 2013. I wonder why 5.5% overall cuts translate to 20% cuts. The SciMag article makes it seems like they were only cutting the number of grants not the size which kinda makes sense. Perhaps they are treating funded grants worse which seems crazy. Wouldn't this potentially waste the money already spent if the project can't be finished on 20% less?

Good luck with omnisci.org this is the sort of thing that would help: open sharing of data, techniques and negative results. If this was the norm things could be very different. But one thing I have learned is it is very hard to change and organizations culture.

I've seen this happen as well, but I think the problem is more with the "make up a story" part and less with the "run to publish" part. I've seen really really interesting results that defy explanation get passed over for publication in favor of something more mundane that can "tell a story" because, it seems, stories get funded...intriguing research? not so much...

This is a pretty dubious justification. Whether or not Alchemy lead to later easily reproducible results, the alchemists couldn't have engaged in an effort to present themselves as what we now know as scientists.

Oppositely, it seems logical to expect that a parade of fake and often impressive results would act as a damper to real, modest gains getting attention and funding.

So, basically, it's a combination between: A) A really hard problem to solve; and B) A heck of a lot of pressure to get it solved.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact