
Want more policies based on evidence? - ryan_j_naughton
http://review.chicagobooth.edu/accounting/2018/article/want-more-policies-based-evidence
======
ejlangev
This idea always struck me as fairly misguided. It of course makes sense to
try to muster evidence and data when making decisions but it's not some sort
of panacea. The book The Tyranny of Metrics ([https://www.amazon.com/Tyranny-
Metrics-Jerry-Z-Muller/dp/069...](https://www.amazon.com/Tyranny-Metrics-
Jerry-Z-Muller/dp/0691174954)) goes into the successes and failures of various
attempts to use data for decision making in some detail. Found it to be an
interesting read.

In terms of public policy you have to decide what you're optimizing for and
that decision can't be made with data alone because it does not help resolve
questions of value and fairness.

~~~
narag
The underlying problem is politization. You can assume that if some political
party wants some something and another political party wants the opposite
something, you could find a set of impartial experts that would provide hard
data and solve the question. In the real world, there are two sets of experts,
holding opposite views and providing contradictory data. Everybody will make a
big noise and eventually nobody knows what happened, just that the question
got muddied and you aren't so sure about anything anymore.

~~~
TheOtherHobbes
The problem underlying politicisation is _confidence._ Science isn't binary,
it's a set of circles of decreasing confidence that spreads out from a core of
propositions that we're very confident about - more or less what you'll learn
on an undergrad physics course - to a set of increasingly tentative
hypotheses.

A lot of arguments about science are really arguments about confidence. E.g.
most climate change scientists are fairly sure about their models, but the
lack of absolutely certainty makes it possible for deniers to cherry pick a
tiny collection of outlier scientists who will argue in public that it's all
nonsense.

Policy makers and the media are some combination of corrupt and clueless, so
they're happy to go with the false equivalence this creates.

One way to depoliticise science would be to have an international science
foundation, which was funded independently of any individual government.

Of course there would be squeals of disapproval from vested interests, but
that would simply highlight the problem - the vested interests don't want
independent criticism or oversight. Their entire MO is based on regulatory
capture which gives them the freedom (for themselves only) to operate as they
want with no personal or financial consequences.

Scientific accountability would set them on the path to democratic
accountability, which is the last thing they'll accept.

~~~
chriswarbo
> A lot of arguments about science are really arguments about confidence. E.g.
> most climate change scientists are fairly sure about their models, but the
> lack of absolutely certainty makes it possible for deniers to cherry pick a
> tiny collection of outlier scientists who will argue in public that it's all
> nonsense.

I think scale/proportion is also a problem. Humans seem to place a lot of
value in narratives/stories but we aren't so good with quantities (e.g.
[https://en.wikipedia.org/wiki/Conjunction_fallacy](https://en.wikipedia.org/wiki/Conjunction_fallacy)
). Pretty much everything (economics, climate, etc.) has factors pushing it in
different directions, so we can always find a counterargument to any position
(e.g. we can rebuff climate change by pointing to solar cycles, CO2 causing
extra plant growth, etc.); that's fine, but some factors are overwhelmingly
more important than others, whilst we seem to cling on to these
stories/narratives and give them more equal weighting than we should.

As a concrete example, a family member used to leave their lights on
overnight, claiming that "they use more energy than normal when they're first
switched on". Whilst true, the saving is cancelled out after _seconds_ ( e.g.
[https://www.energy.gov/energysaver/when-turn-your-
lights](https://www.energy.gov/energysaver/when-turn-your-lights) )

------
lacker
I didn't understand the article's point at first because to me, medicine is
not a great example of evidence-driven policies. The American health insurance
system does not appear to be optimized by any sort of metric, much less to be
driven by evidence. Many key decisions, like how many doctors should be given
medical degrees each year, are not subject to any evidence at all. And just at
a high level in terms of performance for cost something is fundamentally "not
working" in medicine over the past couple decades.

It's true that for the micro decisions, like does drug X treat condition Y,
medicine appears to be much more evidence-based than, say, sociology. But the
overall dysfunction of medical policymaking makes me dubious of using medicine
as a model for other fields of decisionmaking.

~~~
rayiner
Even at the micro level: many childbearing related dictates (don’t drink, do
breastfeed) are based on extremly weak evidence. Emotion trumps data in many
cases even in medicine.

~~~
chaosite
What do you mean by "don't drink" being based on extremely weak evidence? Do
you mean after the pregnancy, or while trying to conceive?

Because the harms of drinking during pregnancy are well known. Fetal Alcohol
Syndrome is a real thing, as are other Fetal Alcohol Spectrum disorders.

~~~
winstonewert
I don't know if this is what the OP was referring to but to quote the
wikipedia article:

> Evidence of harm from less than two drinks per day or 10 drinks per week is
> not clear

It seems there is strong evidence that "heavy" drinking is bad, the evidence
that you shouldn't drink at all is much less clear.

------
ada1981
I’m concerned the medical field is what is being aspired too.

The number of examples of where standard of Care deviates from empirical
research is staggering.

We still lose hundreds of thousands of lives a year in the US to preventable
medical errors.

Many hospitals still refuse to use things like simple checklists that are
proven to save lives.

~~~
avgDev
So this is an area of my interest. I actually hope to one day be able to
improve diagnosis of certain diseases through apps.

I have widespread tendinitis caused by a reaction to a medication. I have seen
many physicians and ALL OF THEM are just terrible with dealing with stuff they
don't have expertise in.

Also, my cousin recently got diagnosed with EDS she suffered for over 20 years
but they just made the diagnosis now.

I cannot comprehend why doctors aren't just open when they have not diagnosed
a disease with some symptoms and ask to schedule another appointment. They go
type some terms into google and at least come up with a checklist or a plan to
test for diseases. The worst thing is when you have done some ACTUAL RESEARCH
and can even provide certain studies but they straight up don't have the will
to look at it. I think many doctors are just getting burned out by the current
state of the medical system in the US, which is definitely not helping.

As a SE I often say, I have never done that before but I'm sure with a little
time I can get it done. Medicine is supposed to be based on science but some
doctors do not trust drug companies and studies anymore, as that is another
shit show. There is pressure on researchers to obtain positive results.

------
aidenn0
Too many people know that evidence will indicate that a policy that they don't
want changed should be changed. This makes evidence gathering much, much
harder.

------
fein
> Who wouldn’t want science and empirical evidence to guide policy decisions.

One look at crime stats and this will not be a favorable idea.

~~~
rpedela
Care to elaborate?

~~~
Forge36
I think what OP alluded too is not understanding preexisting bias. "Where
should police patrol" ->where crime is highest. Where is crime highest
->existing reports point to X neighborhood having the most arrests.

Without asking "why" it may ignore That neighborhood is black, so historically
racist decisions lead to extra patrolling (but no evidence of increased crime
rates) which led to more arrests.

~~~
xamuel
Your logic could be misleading if not parsed carefully. It's not that police
shouldn't patrol where crime is highest--of course they should patrol there,
if they know where that is. Rather, the point is that "most arrests" is not
necessarily an accurate way to detect where crime is highest.

~~~
Forge36
Precisely. Thank you for clarifying!

------
nonbel
"Evidence-based" seems to end up meaning practices like NHST where there _is_
evidence involved, but the analysis is so messed up you can conclude anything
you want from it.

And its really funny they use "evidence-based medicine" as a positive example.
Basically this is what we should expect to happen:

> _' “Evidence-based medicine has drifted in recent years from investigating
> and managing established disease to detecting and intervening in non-
> diseases,” Greenhalgh and colleagues wrote in BMJ in June.

Furthermore, the “evidence” favoring various treatments typically comes from
trials in which companies decide which drugs to test, at what doses, on how
many people. Often the experimental statistical “evidence” in such studies
establishes a benefit for a drug that is of little practical value; the
supposed benefit, while perhaps real in a mathematical sense, is so slight as
to be meaningless for real patients. In other cases, perfectly sound evidence
regarding a particular disease is rendered irrelevant in patients afflicted
with more than one disorder (patients who are commonly seen in medical
practice, but typically excluded from the trials that produced the evidence).

On top of all that, Greenhalgh and colleagues point out, the sheer volume of
medical evidence makes assessing it all intelligently essentially impossible.
Even the guidelines summarizing the evidence are too voluminous to be useful
to doctors.

“The number of [evidence based] clinical guidelines is now both unmanageable
and unfathomable,” Greenhalgh and coauthors note. In one 24-hour period in
2005, for instance, a hospital in the United Kingdom admitted 18 patients with
44 diagnoses. The relevant U.K. national guidelines for those patients totaled
3,679 pages. Estimated reading time: 122 hours.'_"
[https://www.sciencenews.org/blog/context/evidence-based-
medi...](https://www.sciencenews.org/blog/context/evidence-based-medicine-
actually-isnt)

~~~
08-15
Medicine and life sciences adopted bad statistics. Every single paper claims
something along the lines of "what we observed is _so incredibly unlikely_
(<5%) assuming complete randomness, it _must_ mean our theory is true, our new
drug works, the bad pollutant causes cancer, etc." Papers that should claim
"what we observed is completely ordinary and there is nothing to see here"
never get published.

Even if this was done correctly (it isn't, as John Ioannidis keeps pointing
out), you'd get lots of false positive results (Ioannides again). The moment
you look at a collection of statistics, the only available conclusion is "They
all contradict each other!!1"

Until they ask physicists for help, medicine, life sciences, and many other
fields will not be evidence based. We can call them "argument based", but
that's not the same...

------
mbrodersen
Politics is mostly about power and ideology. Not truth/evidence/practicality.
Even if people agree on the evidence they will still disagree on what to do
about it.

------
Dowwie
Government tends to conduct studies prior to engaging in an effort but it's
questionable how much evidence from the studies actually gets used in decision
making.

------
luckydata
Coming from the Chicago School of Economics, specialized in voodoo theories
that don't pan out in the real world this is pretty funny.

~~~
rsj_hn
Pretty much. The Chicago school was the one pushing for financial
deregulation, IIRC.

I think the problem here is that we elect governments to solve collective
problems, not personal problems. Collective problems arise from the
interactions of large groups of people and they are generally intractable.
Even if an economist tries to accurately model human behavior, it's really
hard to do as you start adding more rational agents, and then you need a
theory of production and a theory of expectations formation and all of a
sudden what you are doing is just a toy to be tractable.

So the zone where there is clear unambiguous science is very small, and it's
generally not the zone of contention in politics. Stuff like "don't swallow
arsenic" \-- it doesn't really help us to govern or solve any of our political
questions.

So if we can't use science to show us how to organize ourselves, the debate
turns to meta-approaches which are basically philosophies of how we approach
problems.

For example, a liberal might start with a Rousseau-type blank slate view of
human nature and a various declaration of rights, both of which are basically
statements of faith, and then try to think of some way that government can
secure those rights through a period of debate, and then adopt whatever seems
right to the people doing the debating, even if it means overturning long
established customs.

A conservative might go -- woah, there is hidden information in customs and
traditions, which we may not appreciate (it's hidden) but, as an article of
faith, the fact that these customs have lasted so long means that they are
probably useful for _something_, and I doubt your ability to think of every
unintended consequence of your policy, so I'm going to oppose it, and instead
suggest you do something much smaller, as a pilot, over here for a generation,
and then see what happens.

And a libertarian might be led to say, "Your proposal infringes on my personal
liberties, and thus, as an article of faith, must lead to bad outcomes."

Etc.

So the real debates we have are actually methodological debates given that we
are working under imperfect information and limited knowledge about how
complex interacting social systems work. Saying "we should use more science"
is vacuous B.S. What we need is a better functioning public policy discourse
that supports more course corrections, allows more people to air their views,
and eliminates certain known distortions, like the influence of big money
groups on campaigns. That's a hell of a lot more useful than pretending that
scientists can be politicians or that politicians should be scientists.

~~~
raihansaputra
I don't know whether you've also read these books or not, but for others who
are really interested in this, I can't recommend Anti-Fragile and Skin In The
Game by Taleb that explores this view and topic. I just finished them both and
I feel the book can explain it in a reasonable way, but don't get too serious
on his name-calling. It gets a bit funny when you know where he comes from.

~~~
rsj_hn
I know of Taleb, but haven't read him. I've read some of his essays, and he
has good points to make. The name calling doesn't bother me (since he hasn't
gored one of my sacred cows, not because I am especially enlightened)

