
The High Cost of Not Doing Experiments (2015) - gwern
http://behavioralscientist.org/mindware-the-high-cost-of-not-doing-experiments/
======
Thriptic
I'm not sure if this is really plausible, but I would love to see a
legislative rule instituted which stated that programs, regulations, and
spending measures cannot be passed without doing the following:

* Clearly laying out the goals that bill sponsors are trying to achieve with the proposed change as well as the reasoning driving the change

* Clearly articulating a set of metrics by which we can judge the efficacy of the program or change

* Defining a period in which studies or review will be conducted to evaluate efficacy and who will be funded to do them

If the change is later reviewed and it is found to be ineffective, it should
either be forced to be modified or immediately rolled back. This would protect
against a whole host of legislative silliness such as pork belly / vote buying
projects, bill scope creep, the immortality of improperly validated programs
as mentioned in the article, and knee jerk reaction regulations after
tragedies or in the face of anecdata and moralizing.

~~~
dhimes
Like kweinber, I've also advocated this for years. I'll accept that it's hard,
but I won't accept that it's impossible. Think of it as "scientific
government." We institute falsifiable hypotheses and test them. This can only
be done by folks who want the _right answer_ however, and not too many people
in government are like that. They may want their answer to be right, but of
course that is not the same thing.

At the very least, set up a criterion (or criteria) at which we will agree
that a policy has failed.

Betsy DeVos is being disruptive in U.S. education. Will it work? Take the US
rankings now. If in four years our ranking has fallen, we try something else.

Try a tax cut. If the economy is x% worse 18 months after it going into
effect, it failed.

I'm sure when we first start this we will struggle to get the metrics right
and to ask the right questions. But we should aim in that direction.

I suspect that the reason we don't has to do with money, and in a bad way.

~~~
spiralx
I call it "evidence based government" to match evidence-based medicine. But it
is one of the things I'd most like to see happen, because the real world seems
like far too small an influence on politics nowadays.

~~~
dhimes
Evidence-based works too, but I chose scientific because of the falsifiable
hypothesis approach. We set it up so that we can eliminate things that don't
work from being retried under similar conditions just because a different
party takes over.

------
rlanday
> But in fact D.A.R.E., as it has been conducted for the past thirty years at
> least, doesn’t decrease children’s use of drugs.

That's an understatement. Numerous studies have found that D.A.R.E.
_increases_ drug use:

[https://en.wikipedia.org/wiki/Drug_Abuse_Resistance_Educatio...](https://en.wikipedia.org/wiki/Drug_Abuse_Resistance_Education#Studies_on_effectiveness)

~~~
MiddleEndian
Another problem I have with DARE is that it conflates some important messages
(like don't give in to peer pressure) with its message of all drugs will ruin
your life. When people learn that certain drugs they mention are just not that
bad, other important messages could be dismissed as well.

Note: I don't have any study to support this but it's my hypothesis.

~~~
Thriptic
Exactly this. The program is infantilizing and not based in fact. There are
great reasons why you may not want to do a number of drugs: opiates (high
addiction potential, incapacitating); meth (wrecks you); cocaine (less
problematic but you can't trust the cutting agents, it can exacerbate cardiac
conditions, it can give you transient ED if you're a guy); molly (one poor
formulation can kill you or leave you mentally incapacitated for life). I
don't recall this ever being reviewed at a D.A.R.E event. All I remember
hearing is "drugs are categorically bad and dangerous" which as you say falls
down under scrutiny when you think about alcohol, weed, or even cocaine to
some degree.

~~~
yorwba
Alcohol, weed and cocaine are pretty bad and dangerous, though. It just
happens to be that those are more or less socially acceptable ways to become
temporarily stupid at the expense of long-term health, which I agree makes
"abstinence only" education efforts seem hypocritical. Since people are going
to self-medicate one way or another, teaching responsible use seems like the
better choice.

~~~
tomrod
Weed and certain cocaine uses aren't bad.

~~~
yorwba
If by "not bad" you mean "not as bad as other drugs" or "I can live with the
consequences", then sure. Otherwise, messing with your body's chemical
pathways tends to have negative side-effects, especially when you overdo it.

[https://en.wikipedia.org/wiki/Cannabis_(drug)#Adverse_effect...](https://en.wikipedia.org/wiki/Cannabis_\(drug\)#Adverse_effects)

[https://en.wikipedia.org/wiki/Cocaine#Adverse_effects](https://en.wikipedia.org/wiki/Cocaine#Adverse_effects)

------
RcouF1uZ4gsC
The problem we have with the social sciences is that we don't have good
general theories to make sense of the experiments. Think of Newton's laws of
motion. We could do a few experiments and calculate g and frictional
coefficients and then we could know the answers to a whole class of problems.
We don't have to do experiments on how fast a x kg block will slide down an
inclined plane for every value of x.

Having a theory also enables us to get more out of a single experiment. Think
about how the Michelson–Morley experiment led to Einstein's theory of
relativity and all it's implications.

We don't have any such framework for the social sciences. We have trouble
generalizing from the results of one experiment to another experiment with the
situations slightly altered.

~~~
ejstronge
I accept your argument from the perspective that it would allow social
scientists/other interested individuals to assess new data in an easy
framework. But, I don't think it would change the political problem that the
author is describing.

Some (admittedly weak) examples illustrate this point: despite our knowledge
of physics and our exploitation of this knowledge in the form of aircraft and
rockets, flat-earth believers and those who don't believe/understand
acceleration abound.

We know the mechanisms by which vaccines work - and could go observe the
diseases they prevent by simply taking a trip, or reviewing public health data
from different countries. But, anti-vaccine belief is also increasing.

There are many more of these stories, where the 'easy' idea beats out the
well-established scientific consensus. Thus, I'm not sure that a fundamental
understanding of how influence/addiction/personality work would change
political support for these ineffective programs.

~~~
TheOtherHobbes
Flat earthers have zero influence on the development of physics, and their
ideas don't matter to anyone except themselves and a few opportunistic
journalists.

This is not true in the social sciences where it's very hard to do physics-
grade research, especially given that the social sciences are very
politicised, and research is often done to prove a political or moral point.

So in practice our understanding of personal and social/political psychology
is pre-Copernican. There's a lot of moralising - which you will agree or
disagree with, depending on your predisposition - but very little high quality
research into the way that moral and political decisions are made individually
and collectively.

We're left with a mythology of free objective democratic choice in politics
which is clearly naive and reliably breeds monsters, but as yet there isn't a
better model of choice to replace it.

~~~
ejstronge
I agree with your point - as mentioned in my first line, I state that such a
comprehensive model would be very helpful for academic purposes.

I don't believe that it would change popular opinion - indeed, flat-earthers
and other 'anti-intellectual' fads continue despite not being accepted by
serious researchers.

------
itissid
What works clearing house
[https://ies.ed.gov/ncee/wwc/](https://ies.ed.gov/ncee/wwc/) seems like a step
in the right direction. If there are more such efforts to summarize
research(especially to address the problem of reproducibility in Social
Sciences and Micro Economics) that would be awesome.

------
doglet
I think at the root of the problem is statistical ignorance, combined with
confused morale judgment, along the lines of "but if we think X is effective,
it would be wrong to withhold providing X for everyone equally!"

------
gumby
Finland now has a government program for doing such studies, called Kokeilun
Paikka ("Place to Experiment").

They are using it to do a random trial of basic income. The results to date
are ambiguous, showing both that it's hard to get a large _n_ and hard to pick
the measurement criteria.

Also: although I agree with the thrust of the book, there are additional
issues for programs like Head Start: 1> it's hard sometimes to _find_ or
identify the kids who need it most, so by making it broader you improve your
chances of reaching those most at need, at the cost of "helping" others who
may not really need it 2> when you make programs universal, you get broader
support than when you only extend it to those most in need. Look at the
additional restrictions just placed last month on those in the US getting
public assistance -- restrictions which make it hard for those most in need to
actually get the benefit! This is why the New Deal extended social security to
everyone and dressed it up it so that it appeared to be an insurance scheme
you paid into: everyone would feel they "earned it" so nobody would want to
cut it.

Thus in the US school meals are not universal or free, which stigmatizes the
kids who get them. In most developed countries the kids all get lunch in
school, which helps the nutrition of the poorer kids. Likewise, making public
transport free would pull cars off the road, making drivers happier...but when
they feel they are paying for "those people", voters typically hate using road
taxes to pay for streetcars and subways. Perhaps the ability to tax self-
driving vehicle fleets will change that.

This sort of goes back to the Finnish observation of the difficulty of making
measurements.

------
gervase
Needs a [2015] tag.

~~~
sctb
Updated. Thanks!

