
Investigating a ten-year-old estimate that “most social programmes don’t work” - NadinevdW
https://80000hours.org/articles/effective-social-program/
======
schneems
My S/O used to work at a company AuntBertha. They're a "search engine for
social services" but they are a for-profit company (a B-Corp).

Their biggest clients are hospitals and insurers. There are people called
"frequent flyers" who come to the emergency room frequently and while they
might only be 2% of clients they account for a huge share of costs. It's
actually cheaper to hire social workers to provide these people with the
things they actually need (i.e. treating the causes) instead of the symptoms.

Here's an example. Guy is low income. Cannot afford food & insulin for
diabetes. So he chooses to not buy insulin and the food he does buy is high in
sugar (think cheap whitebread). Goes into diabetic shock and gets sent to the
emergency room. Instead it's much cheaper for everyone involved if a social
worker finds out the problem, helps him research some foodbanks to cover his
nutritional needs or gets him on a discount insulin program. It's not free,
but it's much cheaper than trying to clean up a mess after the fact.

~~~
auggierose
Let's just say if you live in a country where you cannot get insulin if you
need it, that is a pretty fucked up country.

~~~
zzzcpan
Yeah, it's shocking how inhumane this is. And how casually the op mentions it,
like it's not a big deal.

~~~
schneems
Am I "op" here? I live in America so it's all I've ever known.

Another example involving insulin: The client got free insulin but did not
pick it up. It was only after a social worker dug into the issue found out the
client did not have a fridge and so when picking up the insulin the pharmacist
said "do you have a fridge", they said "no" and walked away. The social worker
found a program with home depot to get a free mini fridge.

------
protomyth
I participated in a social program funded by the gov in the 90's as the data
person. It was a combined early childhood education / social service program.
I won't discuss what I thought of the outcomes or the cost / benefits, but I
think there is a piece a lot of people miss.

There are consultants in DC that take all the data from a program and
summarize it. They have a contract and aren't always the folks who have the
assistance contract on the grant. People are sadly, people, and sometimes the
summarizing folks have some pretty powerful incentives to put a spin on the
numbers, drop certain grants, and well, other things. Life is incentive based
after all.

When I look at the outcomes, I often wish there was a way to get some
anonymized raw take because I am a bit mistrustful of third-party analysis
particularly when they aren't up on simple things like geographical and
economic differences. Plus the tendency to roll all the results in one set of
numbers ignores how big the USA is and that San Francisco is a bit different
than Salt Lake City or Rosebud.

PS: one thing, it is amazing how many children start life with hearing
problems - fix that early and save the rest of us a lot of money. Plus,
fatalogic has it right in my book, life is complicated, one axis of action
doesn't really cut it.

~~~
jansho
I'm glad that you mentioned hearing problems (ear infections specifically).
They have been shown to affect child development in ways that aren't
immediately intuitive, like hyperactivity and their ability to pick up phonics
[1]. That obviously has an impact on their early learning, and consequently
the rest of their lives.

[1]
[https://www.hslda.org/strugglinglearner/CraftDocs/EarInfect....](https://www.hslda.org/strugglinglearner/CraftDocs/EarInfect.asp)

~~~
protomyth
Yeah, new parent really need to make sure they get the screenings and be very
aware of their child's hearing. This can slow your child's development in the
critical 0-3 years. This is not a grow out of it thing, quite the opposite in
fact.

------
fatalogic
As a former social worker I think most social programs don't work because they
try to deal with multifaceted problems from a single point of action. It's
like trying to fix a suspension bridge by replacing one cable while all the
other cables are broken.

~~~
forgotmysn
do you think a UBI would be a more effective way of helping people in more
complex situations?

~~~
fatalogic
It would help some people but for most of the people I worked with money was
more a symptom of their issues than a root cause.

Lets say we have a child who is in 5th grade who reads at a 1st grade level.
Most programs will say oh well he/she just needs more education so they create
after school tutoring. That's great but there are so many things that can be
hindering that child's ability to read. The real root may be a medical issue,
family issue, cultural issue, structural or a combination of one or more.

People aren't equations, you can't just input x and expect y.

~~~
jimmaswell
>People aren't equations, you can't just input x and expect y.

People react in mostly predictable ways to certain environments and
situations. We're largely deterministic state machines. This kind of mystical
thinking devalues the importance of a scientific, objective, data based
approach to solving human problems. We aren't an exception to science.

~~~
PeanutCurry
I'd like objective data that sustains your argument then. I think approaching
problems, including social problems, as rationally as possible is important.
I, similarly, believe that leveraging all available data is important.
However, as far as social issues are concerned, American psychology and
western psychology in general have shown themselves to be inadequate when
addressing the universal set of humans and not the western set. We've seen
evidence of this not only in western vs. east-asian psychologies but also in
the psychological treatment of refugees from areas like Sudan. I appreciate
that there might come a day when human psychology is a solved problem, but
unless that problem is currently solved then it seems like leaving room for
mysticism is more pragmatic because it permits us to identify, and adapt
around, areas of uncertainty while still providing some amount of care.

~~~
jimmaswell
Admitting uncertainty or having probabilities isn't unscientific, but saying
something is outside the realm of normal physics with no evidence is. Sure,
psychology as we know it may be flawed, but retail, advertisers, etc. have a
pretty good track record of systematically achieving results based on
predictable human behaviors.

------
pillowkusis
An anecdotal but interesting illustration of the suprising finding that
"social programs don't work": [http://freakonomics.com/podcast/when-helping-
hurts/](http://freakonomics.com/podcast/when-helping-hurts/)

The tl;dr being in the mid 1900s when somebody actually DID try to measure the
results of a mentor program for at-risk youth, they found the impact was
slightly negative.

Actually fixing societal problems is a really hard problem and what actually
works is often very counterintuitive. I think the whole field could do to let
the data guide them better (also difficult because so many outcomes are "make
people feel better", and most people are pretty bad at actually knowing how
they are or will be feeling).

~~~
xxSparkleSxx
I've long had this thought about charity itself, it seems that a lot of
charity is propping up bits and pieces of capitalism so we don't look at the
root causes of why we "need" charity in the first place.

On one hand it's great that children around the US can have life saving
surgery because so many donated to their cause, on the other hand would we all
be better off donating to a lobbying cause for single-payer healthcare?

If we could get that passed, every child in the US could have life saving
surgery if needed and it wouldn't be a matter of whether or not their
community is charitable enough.

------
maerF0x0
People are failing to recognize that it doesnt matter what percentage of
programs fail/succeed. The metric that matters is the ROI on programs. If 25%
of programs succeed, giving a 100x ROI, then they more than make up for the
failed programs. Unless we know a better way (UBI seems promising) to help
people, then programs seem to be the best bet.

The problem is that people hear "75% of programs fail" and think "Dont start
programs". This is like saying "75% of businesses fail therefore do not start
a business" .

~~~
kerkeslager
Most startups fail. We'd better shut down YCombinator!

------
twotwotwo
I appreciate the last section's perspective: that giving to the causes best
supported by evidence, though very helpful, does not guarantee 10x (social)
returns since measurement is imperfect and some big returns are likely from
things that aren't fully studied yet. Besides how that makes sense in its own
right, it sort of pushes folks to recognize that that the many useful
strategies the effective altruism community has come up with still aren't a
silver bullet for aid effectiveness.

Recognizing there's no silver bullet seems especially relevant to me because
there's a folk interpretation of "most social programs don't work" that's more
like "we spend tens of billions of dollars trying to alleviate poverty; why
isn't it gone now?". And _that_ question only seems to make sense because
people misjudge the size of the challenge relative to the amount spent on it
and the returns you can expect from that money.

The U.S.'s yearly global health budget, if spent entirely on sub-Saharan
Africa, would be under $20/person, and you don't, say, raise average income
$10k with $20, no matter how wisely or creatively you invest it. Demanding
that kind of return from charity is like expecting every investment to perform
like early money in Google, except, perhaps, that the impact of your demand is
less on the investor and more on the global poor.

Separately, it seems like a shame to me that in fields where we do have strong
signs of a high multiplier, like bednets or deworming, the opportunities
haven't already been fully exploited by governments and billionaires. Some of
it is that large spenders are making their choices through a process much
different from someone like GiveWell's, but some is also that governments, for
example, spend a lot less of their budgets on non-strategic global-health aid
than most think (<1% for the US). More about aid spending here:
[http://www.npr.org/sections/goatsandsoda/2015/02/10/38387558...](http://www.npr.org/sections/goatsandsoda/2015/02/10/383875581/guess-
how-much-of-uncle-sams-money-goes-to-foreign-aid-guess-again)

------
chrismealy
This is silly. The biggest social programs in America are Social Security
(basically a UBI for old folks), Medicare (health care), and the #1 economic
development success of all time, public education. The kind of social programs
they're talking about in this article are so unambitious it's hardly
surprising their results aren't measurable.

------
BartSaM
I would love to see more work-for-benefits programs. Where you are required to
do some labour work for the community (cleaning wild areas, swiping side
walks, planting trees etc.) in exchange for some better than average benefits.
IMO everyone should be entitled to such a work. This way we would eliminate
involuntary extreme poverty.

~~~
schneems
Many people who are currently homeless are not the bums you see on the side of
the road. Many have assets (computers, clothes, maybe even cars) but have been
temporarily displaced from a home. Usually due to unforeseen financial
hardship like an unexpected medical bill, or an auto accident etc. These
people don't have a safety net of family or friends they could borrow money
from so they get evicted. These people may currently have a job but no place
to stay when they leave form work.

Many of the successful homeless programs are "housing first" programs. Search
for "colorado homes first program" to read more about it. In many cases it's
also the cheapest way to do it as well. If you want to organize a group of
currently homeless people to do some task, you have to coordinate travel, pick
them up, have a van or a truck, hire a coordinator to plan the events and
oversee them, and then at then on top of it pay for housing. It's actually
cheaper just to provide housing.

Anyway, just wanted to say the issue goes deeper than "why won't these
lazybones work".

~~~
BartSaM
You are going extreme on me now.

I did not say "lazy guys won't work".

I said a lot of people would love to work for their benefits. I have been in
this situation and hated. And I know a lot of pride people, that would not
take free money.

I have not said this is a solution to everything.

I said, this would be a good thing and additionally should be put in the
Constitution ("right to work").

I think there should be multiple social programs and people should be able to
choose ones.

Programs that are performing best should get more funds, the ones that are the
worst should be removed.

~~~
shubb
Such a programme was recently tried here in the UK.

I don't know the details but friends effected mentioned:

* Being required to attend an office full time where they were monitored job searching and required to do soft skills courses, which they perceived as not useful.

* After an extended period of joblessness, being required to work for free (while recieving normal out of work benefit), for instance at a supermarket

This was critisised because:

* 'Jobless' people working for 1/4 the minimum wage displaced normal workers

* The programme encouraged people to make a high volume of inappropriate job applications. These make it harder for employers to sift candidates who actually want a job (I notice - do you want 100 angry highschool dropouts applying for your developer roles?), and reduce time and energy available to chase real leads.

* People who landed interviews for real jobs risked catastrophic penalties if they attended the interview, because the bureaucracy often failed to accept that this is what they were doing.

You could argue that this was a badly executed programme. Or that it was more
focused on reducing public spending by discouraging people using their social
insurance than on rehabilitating anyone. Or you could argue that aspects were
good and got a bad rap.

For me, I think the nanny state can be terrifying, and that that no workplace
needs people who'd rather be at home.

~~~
BartSaM
This is another contradictory to what I said.

I said working for public good - not supermarkets etc. Working in public
spaces for community benefit. That is a huge difference. Why should someone on
benefits work for a private business or even public offices? That is
ridiculous! They are receiving public help, they should help the public.

There is absolutely no place for the nanny state. Try not to look at things
black/white.

~~~
prawn
I'm interested in seeing something as you've described too. All the little
tasks that would make a community look really attractive, but that are ignored
by councils. The sort of things you see industrious, elderly (often migrant)
men do around their own gardens when they have endless spare time. Not back-
breaking labour or anything especially unenjoyable, but tasks for people who
have pride in having things look presentable.

~~~
shubb
There is though a workforce of gardeners employed by local governments and
individuals that you would be displacing. Perhaps from a good job onto the
work programme.

~~~
BartSaM
I do not see it would happen on a large scale. Those work-to-benefit could
work on things, that local government cannot afford or are not planning to do.
Even cleaning wild areas which are a serious problem in some parts of woods
for example. Or cleaning beaches, or planting trees in remote areas. There are
those things that simply are hard to achieve when you are tight on a budget
already.

------
robertwiblin
I'm one of the authors of this post - look forward to hearing you comments!

~~~
sharemywin
So, I couldn't get a good feel on how much of the issue is small sample size?

~~~
BenjaminTodd
Some of it is that, but much less so for the meta-analyses. We have a bunch
more comments about this in the doc.

------
nonbel
The way of telling if it works is: statistical significance.

This "approach" to research was debunked as pseudoscience long ago (eg here is
one from 1967[1], and that wasn't the first)... So the answer is: no one knows
what works or doesn't, (edit: at least not based on the usual claimed method
of drawing conclusions from the data).

[1]
[http://www.fisme.science.uu.nl/staff/christianb/downloads/me...](http://www.fisme.science.uu.nl/staff/christianb/downloads/meehl1967.pdf)

~~~
cuchoi
It might not be perfect, but it is the best way we have come across.

~~~
nonbel
What about actual science? Where you figure out how to get reproducible
results, come up with theories to explain why those results are the way they
are, then make predictions with the theories to test on new data?

~~~
BeetleB
>come up with theories to explain why those results are the way they are, then
make predictions with the theories to test on new data?

Probably never will happen in social science, and in much of medicine. Too
many confounding variables.

I'm willing to bet that if we threw away all the "knowledge" we obtained from
p-tests, and stuck only to the methodology you espouse, we would know far less
true stuff than we do currently (albeit we would not know a lot of false
stuff).

Simply put: Even though there is a lot of noise, we do know more using these
methods. And the nice thing about bad results due to misuse of statistics:
Much of it is not reproducible.

Ultimately, the problem isn't the use of these statistical methods. The
problem is the lack of emphasis on reproducibility.

~~~
nonbel
>"Probably never will happen in social science, and in much of medicine. Too
many confounding variables."

This is some kind of talking point. I'm not sure who is behind it, but it is
totally false. There are _tons_ of examples of this, especially from the pre-
NHST era:

[https://en.wikipedia.org/wiki/Kleiber%27s_law](https://en.wikipedia.org/wiki/Kleiber%27s_law)

[https://en.wikipedia.org/wiki/Law_of_mass_action](https://en.wikipedia.org/wiki/Law_of_mass_action)

[http://www.ncbi.nlm.nih.gov/pubmed/17845298](http://www.ncbi.nlm.nih.gov/pubmed/17845298)

[https://en.wikipedia.org/wiki/Law_of_effect](https://en.wikipedia.org/wiki/Law_of_effect)

[http://www.tandfonline.com/doi/pdf/10.1080/00221309.1934.991...](http://www.tandfonline.com/doi/pdf/10.1080/00221309.1934.9917847)

[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2916857/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2916857/)

[https://en.wikipedia.org/wiki/Compartmental_models_in_epidem...](https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology)

[http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2007940/pdf/brjc...](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2007940/pdf/brjcancer00386-0010.pdf)

[https://en.wikipedia.org/wiki/Cardiac_output](https://en.wikipedia.org/wiki/Cardiac_output)

------
zebraflask
I would imagine most of them can't, it amounts to trying to bail out the
ocean, one bucket at a time.

In less hyperbolic terms, many of these programs are attempting to tackle
intractable societal problems when - even for ones that are relatively well-
funded - the cost of organizational overhead, limited ability to cause lasting
impact outside of their specific area of influence, etc., reduce what they do
to a lot of good intentions and few meaningful results.

Depending on the "clientele," it amounts to attempting to provide assistance
to people who really can't or don't want to be helped (hard drugs and the
chronic homeless, for example).

------
VikingCoder
"Is it fair to say that it's hard to create good social programs? And we
should all remember that some social programs work far better than most people
realize."

Something like that might not give the wrong impression...?

------
fweespeech
The big secret is social programs are largely to promote stability and prevent
civil unrest. The effectiveness at the stated intention is a secondary
concern.

We probably are better off with a basic income style system where people can
opt-into public (or private) education, public (or private) medical services,
and the balance is paid out to them in cash every month. The cost being
recouped via cutting existing programs and raising tax revenues to recapture
100% of the value once you have a 6 figure household income.

------
omegaworks
From the article:

"So is it fair to say 'most social programmes don’t work?'

I think this is a little ambiguous and potentially misleading. Individual
projects mostly don’t work, but whole areas often do have a positive impact.
So, if you pick an intervention at random, then on average your impact will be
positive, because there’s a small but important chance of you picking one of
the good ones."

Can we fix this to the proper, non clickbait title of "What fraction of social
programmes don’t work?"

~~~
robertwiblin
"4 Reasons Why The Answer To This Complicated Social Science Question is More
Subtle Than You Think - The Conceptual Clarification in 3 Will Shock You"

~~~
sly010
"4 Reasons Why The Answer To This Complicated Social Science Question is More
Subtle Than You Think You Never Thought Existed"

------
schneems
Aren't posts supposed to have the same title as the article? Should be: "What
fraction of social programmes don’t work?"

------
BeStaXP
It might be true that "most government programs don't work" (at least to
achieve their stated purpose), with social programs being a subset of that...

~~~
AnimalMuppet
I suppose I should be flattered that you liked what I said. I don't approve of
you presenting my exact words as your own, though...

------
AnimalMuppet
It might be true that "most government programs don't work" (at least to
achieve their stated purpose), with social programs being a subset of that...

