
Cargo Cult Science (1974) - maverick_iceman
http://calteches.library.caltech.edu/51/2/CargoCult.htm
======
btilly
Everyone who cares about science has known his point for over 40 years now.

But Psychology, the subject he was discussing, continued to not take figuring
out how to replicate seriously until the recent replication crisis
demonstrated exactly HOW bad it was in terms that they couldn't ignore. See
[https://en.wikipedia.org/wiki/Replication_crisis](https://en.wikipedia.org/wiki/Replication_crisis)
for more on that.

And now it is a hot topic with lots of, "Look at what we've figured out about
how to make our research better!" With very little discussion of the fact that
this was something they should have known all along.

~~~
zevkirsh
yes the problem is politics , or what you might call institutionalized group
psychology of those with access to spending tax dollars, and the legislators
they need to 'persuade' to get the money spent.

science is not an art , it is generally an institutionally funded social
behavior. the art of science is such that a lot of shitty ass sciece research
with poorly constructed hypothesis and protocol gets funded............

can we do a better job? maybe. but the answer lies in better policy analysts
and politicians, not in better 'scientists'.

on heuristic for helping pinpoint bad and poorly constructed research is to
look for where the most 'growth' in research has occured the quickest; and by
that-----i mean look for the research 'bubbles'.

why? as with art in general, and clothign and every sphere of human behavior
-----that which gets popular very quickly is usually by nature fleeting and
without the most substance, like fast growing weeds. that which grows slowly
and withstands the test of time usually resembles an oak in quality.

that does not mean that fast growth is always undesireable. you want fast
growth in areas where there is great potential and low hanging fruit.

it is good that bubbles occur SOMETIMES. BUT NOT ALWAYS. some bubbles are
poorly advised. you must crack a few eggs to make an ommelet but sometimes you
are just breaking eggs inside the carton without ever making breakfast.

but regardless of whether or not a bubble is good or bad, all bubbles still
must pop at some point to let the overgrowth and wasteful growth be cleared.
all forests must eventually have a forest fire.

the longer fire and destruction is suppressed beyond a certain
point........the greater the amount of destruction is going to be when it
finally comes.

and in general, this is why our current economic bubble is so worrisome

------
jhallenworld
Well we should talk about Cargo Cult Management. I suspect most management is
Cargo Cult :-)

[https://gigaom.com/2009/06/21/cargo-cult-
management/](https://gigaom.com/2009/06/21/cargo-cult-management/)

~~~
wycx
Very much so. My pet theory is that most business management practices evolved
from manufacturing production line environments, and the application of these
management practices to, say, running a university or research facility is
cargo cult in the extreme.

------
kirrent
The parts that always strike me the most about this is that in 1974 Feynman
lamented a lack of scientific rigour in deciding how we teach and how we
manage crime and the criminal justice system. In both cases, despite 40 years
having passed, it's hard to say there's been any large improvement in the
application of rigour.

~~~
abecedarius
There's _some_ progress:
[https://en.wikipedia.org/wiki/Sequential_lineups](https://en.wikipedia.org/wiki/Sequential_lineups)

The part that boggled me was how recent (and incomplete) was the introduction
of blinding to police lineups. Apparently it really was (and is?) common for
the witness to pick out a suspect, and for the cop to know the 'real' suspect
and say things like "Are you sure? Take your time."

From another Feynman talk: "This is not yet a scientific age."

------
joelthelion
This applies extremely well to programming and software engineering in
particular. "Best practices" are often a good excuse to avoid thinking hard
about the problem at hand.

------
maxxxxx
I think there is a lot of cargo culting going on everywhere. The lucky ones
pick the right cult. How do you know whether your model of how the world works
is right?

~~~
sinxoveretothex
You're looking at this the wrong way. The question is not "does reality match
this model?" but rather "how well does this model match reality?".

You never know exactly to what exact extent your map matches the territory.
But you can have a pretty good idea. Like, we know (to the extent that we can
"know" anything) that models predicting the Sun will rise tomorrow match
reality better than those which don't. We know that models which predict that
turning on a switch will cause the lightbulb to emit light are more accurate
that those who say that it's magic or mysterious.

To quote Feynman:

> We’ve learned from experience that the truth will out. Other experimenters
> will repeat your experiment and find out whether you were wrong or right.
> Nature’s phenomena will agree or they’ll disagree with your theory. And,
> although you may gain some temporary fame and excitement, you will not gain
> a good reputation as a scientist if you haven’t tried to be very careful in
> this kind of work. And it’s this type of integrity, this kind of care not to
> fool yourself, that is missing to a large extent in much of the research in
> Cargo Cult Science.

~~~
maxxxxx
In physics or other hard sciences it's possible to validate a lot of models.

I was more thinking about other areas like politics, management or software
engineering issues. There it's often not possible to validate your model. It's
even hard to tell if something goes well or not. And the feedback loop from
decision to result can take years in many cases. So you pretty much are forced
to do cargo culting.

~~~
sinxoveretothex
Ah, I see. I thought you were going more along the lines of the "how can we
know anything?" sort of philosophical questions.

In the matters that you're talking about, it may just be that we can't really
know. Certain things take years to figure out (take Feynman's example of the
Milikan value): people still −I assume− had to use some value while the
"official" value was shifting.

I'm not sure I'd call that cargo culting necessarily. If the best case to be
made right now is in favour of the majority position, then I think one should
go with the majority position. I'd say that one should still keep a healthy
dose of skepticism and be ready to change their mind if the balance of
evidence changes… but then, it may be that managers/leaders need to feel
committed and that a "healthy dose" for them is very, very little.

------
NoMoreNicksLeft
I wonder how many cheer this, then turn around and do the sorts of things he
was complaining about.

------
m1n1
Ancient typos

She was very delighted with this new idea, and went to her professor. And his
reply was, no, you cannot do that, because the experiment has already been
done and you would be wasting time. This was in about 1935 or so, and it seems
to have been the general policy then to __nut__ try to repeat psychological
experiments, but only to change the conditions and see __hat__ happens.

------
kragen
In CS (Systems, not Theory) research there's a big push now to move toward
reproducibility by publishing code and dependencies;
[https://twitter.com/kragen/status/727220736633524228](https://twitter.com/kragen/status/727220736633524228)
talks a bit about some of the historical context and details.

Hackademia has a couple of synergistic efforts going on, directed mostly at
securing the free-software distribution infrastructure and somewhat at
simplifying system administration: on one hand you have reproducible builds
being done by Tor, Mozilla, and Debian, and on the other hand you have Nix,
Guix, and the like pushing purely-functional content-addressable package
management.

The upshot of all of this is that, in theory, in the near future, it will be
practical to publish machine-executable instructions for reproducing the bit-
identical executables and other output products you used in your research, at
a cost of only a few gigabytes, so they can be exactly reproduced at any time
in the future, and then modified to do new experiments; and it will be
straightforward to make archival copies of this entire set of dependencies so
that it won't be lost because some random guy died or decided to pull his
modules from npm.

Mike Hoye famously complained about how far we are from this in a Tweetstorm
the other day:
[https://twitter.com/mhoye/status/725010388547330048](https://twitter.com/mhoye/status/725010388547330048)

The Sprite-LFS affair I mention in that tweet is an infamous case of a famous,
influential systems paper by a famous, influential academic (Ousterhout) that
badly failed reproduction when a different famous, influential academic
(Seltzer) tried to reproduce its results by writing a new log-structured
filesystem. Nobody really knows what went wrong, but our best guess at this
point is that the Sprite operating system had unknown bugs in it that gave a
performance advantage to log-structured filesystems.

(ChuckMcM may want to weigh in, since his company at the time (NetApp) was
getting record-setting performance with WAFL, which is very similar to a LFS
in some ways, but not others.)

Jan Vitek, Jean Yang, Sam Tobin-Hochstadt, and Shriram Krishnamurthi are four
prominent researchers who are advocating this change; they and others have
created a new institution called "Artifact Evaluation Committees", which
review the artifacts submitted by authors to see if they are able to reproduce
the results the authors report. They've written an overview of it here, and
they seem to already be getting significant results:
[http://jxyzabc.blogspot.com.ar/2016/05/myth-cs-
researchers-d...](http://jxyzabc.blogspot.com.ar/2016/05/myth-cs-researchers-
dont-publish-code.html)

Hopefully over time this will create a bubble of real, reproducible science
inside of the systems part of Computer "Science" research.

[Earlier versions of this comment did not mention Krishnamurthi or Vitek and
the synergistic work in hackademia, which was a combination of cluelessness
and carelessness on my part. I regret the omissions.]

~~~
solipsism
I could be totally wrong, I don't read a huge number of CS research papers,
but aren't most published CS papers fully self-contained? They contain the
algorithms or ideas within them. Perhaps in pseudocode, but fully
reproducible.

The little experience I have is in computer graphics publications and (less)
distributed systems publications. Graphics papers very often have data sets
and code that go along with the paper. These are nice, and it would be awesome
to have a common and well-known way of publishing all that together with the
papers, but they are just examples. Mike Hoye's comparison to the loss of
greek fire doesn't seem applicable to the idea of _reproducibility by
publishing code and dependencies_. The meat is in the paper!

Same with my experience in distributed systems papers. I can implement Paxos
at any time using only the Paxos PDFs I've saved. Companion code is nice, and
again I'd be fully supportive of a system which gave permanence to not ony the
papers but companion artifacts... but the paper is the important part, isn't
it?

~~~
qb45
> I could be totally wrong, I don't read a huge number of CS research papers,
> but aren't most published CS papers fully self-contained? They contain the
> algorithms or ideas within them. Perhaps in pseudocode, but fully
> reproducible.

They may be not reproducible if going from pseudocode to working
implementation and from there to the pretty graphs published by author is
nontrivial. From parent post:

 _The Sprite-LFS affair I mention in that tweet is an infamous case of a
famous, influential systems paper by a famous, influential academic
(Ousterhout) that badly failed reproduction when a different famous,
influential academic (Seltzer) tried to reproduce its results by writing a new
log-structured filesystem. Nobody really knows what went wrong[...]_

> I can implement Paxos at any time using only the Paxos PDFs I've saved.
> Companion code is nice, and again I'd be fully supportive of a system which
> gave permanence to not ony the papers but companion artifacts... but the
> paper is the important part, isn't it?

Paxos is closer to the "theory" than the "systems" end of parent's spectrum.
Real problem is with things that take man-months to implement and validate,
not things that take man-months to invent.

------
pjlegato
The entire "agile" methodology strikes me as a prominent example of a software
engineering cargo cult.

Yes, good projects have been executed successfully using agile methodology.
No, this does not mean that using agile _caused_ these projects to work out
well.

Most agile projects fail, just like most non-agile projects. There is no
evidence whatsoever that agile methodology _causes_ projects to work out well
-- but there is a whole mini-industry of training, books, seminars, "scrum
masters" and so on built around the idea that it does.

~~~
solipsism
There's very little evidence whatsoever about _any_ software development
methodology or practice working well or poorly. That said, most experienced
developers I've interacted with consider most of the Agile principles to be
wise -- either common sense, or shown to be true via decades of experience.

Here are the principles:

* Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

* Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

* Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

* Business people and developers must work together daily throughout the project.

* Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

* The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

* Working software is the primary measure of progress.

* Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

* Continuous attention to technical excellence and good design enhances agility.

* Simplicity--the art of maximizing the amount of work not done--is essential.

* The best architectures, requirements, and designs emerge from self-organizing teams.

* At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Which of these would you argue aren't wise principles for all of us to use
when developing software? Now, if your beef is with Scrum (or some other
branded methodology) in particular, I'd be more inclined to agree.

~~~
trhway
>Which of these would you argue aren't wise principles for all of us to use
when developing software?

most of these aren't principles, these are wishes. Like i can say i have 2
principles - be wealthy and be healthy - and those 2 principles look like very
wise and hard to argue with, don't you think?

Except for the stuff like this :

> The most efficient and effective method of conveying information to and
> within a development team is face-to-face conversation.

>The best architectures, requirements, and designs emerge from self-organizing
teams.

which is just some personal beliefs or dreams of somebody, and is obviously
not true in general. There is a reason things are called 'manifest' and not
even a 'paper/essay', less 'peer-reviewed paper'...

~~~
solipsism
_most of these aren 't principles, these are wishes. Like i can say i have 2
principles - be wealthy and be healthy - and those 2 principles look like very
wise and hard to argue with, don't you think?_

Becoming wealthy takes time and luck. It's not something you can just turn on
at will. I just read through each of the principles, and I find exactly 0 of
them that share that quality. Most of them tell you _specifically_ what to do,
and they are things that you can start doing at any time.

Name one or more specific principles that are like "become wealthy" and we can
talk about them specifically. We may be reading them differently.

 _> The best architectures, requirements, and designs emerge from self-
organizing teams. which is just some personal beliefs or dreams of somebody,
and is obviously not true in general._

I've worked remotely for a long time and love it. However, I certainly
recognize that there's a legitimate cost to the team. I've also worked for
years with teams that are split geographically. Again, the distance is a cost.

Of course there are benefits too. That's why we pay the cost. But the
principle says _the best architectures, requirements, and design_. How exactly
could physical distance be a net negative to architectures, requirements, and
designs, all other things being equal? The methods of communication at your
disposal face-to-face are a strict superset of the methods at your disposal
when physically separated.

------
100ideas
16th time this has been posted to HN

[https://hn.algolia.com/?query=cargo%20cult%20science](https://hn.algolia.com/?query=cargo%20cult%20science)

~~~
harry8
I wouldn't complain if it were posted 160 times. It's brilliant, amazing,
relevant stuff that hasn't dated. I've re-read it at least 16 times since
first encountering it in "Surely you're joking Mr Feynman" in the 80s.

~~~
m1n1
it's been so long, why are there typos in it?!

------
pvsukale1
today is richard feynman's bday! happy bday feynman!!

------
Pica_soO
The problem can not be solved by education alone. Education alone does not
prevent the minds from making mistakes. Its about building and maintaining
social structures that destroy cargo science and at the same time give freedom
to groups/individuals who abstain from it. On envisions a credit system, in
which you earn credibility by proofing something you did works- or by
disproofing something that was wrong (obviously you get all credits that proof
held). New credits enters the system with every entering scholar.

------
dschiptsov
He probably could have been delighted to see all these "pendulums going in
both directions at the same time" and other marvels of modern science we so
often see on this site.

