

The trouble with non-profits - wglb
http://www.aaronsw.com/weblog/measurenonprofit

======
daydream
"But it does mean that if you’re involved in nonprofits (or predictions), you
need to be much more careful about making sure you’re doing a good job.
Unfortunately, few nonprofits do that."

Oh really? So he can cite research that says, for example, that only x% (where
x < 50%) of nonprofits with an annual budget of $500k have ever done any sort
of evaluation, either comprehensive or more basic?

His assertion may be true. But just throwing this out there with absolutely
nothing to back it up, in a blog post titled "The Trouble With Nonprofits", is
frankly insulting to me.

I spent > 4 years working in a technical role for a nonprofit, and I can
assure people that measurement and evaluation were very much on the minds of
staffers. Everyone (funders, management) was becoming more data-driven, so our
organization got a grant to conduct a multi-year evaluation. It was a very
positive and necessary process, and the results were very positive. Like
Aaron, I definitely recommend some sort of measurable goals for most every
nonprofit out there.

I don't take issue with his opinion. What I take issue with is the tone of the
article.

For example: "...but an actual attempt to measure how much they’re improving
people’s lives. For most nonprofits, I expect these numbers will be
depressingly small. "

It can be easy, as young paragons of technology and entrepreneurship, to look
over to the nonprofit world and disparage it, to think of it and it's
practitioners as "lesser than". I should know; it's the attitude I had before
I started working at a nonprofit.

Truth is, running a nonprofit can be a hell of a lot harder than running a
for-profit. At any stage, you have all the issues that for-profits have -
hiring, firing, managing employees, management issues, getting money in the
door, technological problems - PLUS a whole other layer.

A lot of this other layer comes because of the weak link between financial
health and results. Am I accomplishing my mission? How do I measure success?
How do I maximize success based on dollar spent? How do I judge where to
allocate capital or resources - when maximizing revenue isn't a goal?

Why am I wasting my time writing this? Because this article uses a lot of
conjecture and one very poor example to make the entire nonprofit sector seem
like a bunch of bumbling idiots who don't know how to use spreadsheets.
They're not, and it's not fair for this community to see such a distorted
view.

~~~
neeson
I've worked at two social ventures that had non-profits as clients (Web
Networks, and most recently Urbantastic.com).

There emphatically is something wrong with non-profits. Just because there are
some excellent ones (Sturgeon's law applies) doesn't mean that the field at
large isn't ill.

An anecdote: we had a meeting with a guy who worked at a large company with a
philanthropic streak. They provided, free of charge, a "dashboard" for
planning, along with free training and tech support to any non-profit that
wanted it (it cost considerable money otherwise). It was simple but very
powerful, allowing you to do sensitivity analyses on projections to determine
which courses of action would result in the best returns - both monetary and
any other metric you wanted to put in (i.e. the social goals of the project).

It was like he was selling popsicles at an ice rink.

We ran into a very similar issue with Urbantastic. We had a web service that
was very effective at getting small things done for non-profits: it's called
micro-volunteering. There's no supervision required, you just write down what
you want and someone always did it.

When we founded Urbantastic we thought the challenge was going to be in
getting skilled people to help out for free. That was not a problem. The
problem was that we'd go up to non-profits and say: here's a machine where all
you have to do is post your todo list, and people will cross off items for
you. For free. And we never got uptake.

People lined out the door offering to help, and almost no one took them up on
it. Despite our offers to walk non-profits through it, and despite the recent
funding cuts which meant that they needed all the help they could get.

I'm not saying that Urbantastic is perfect, but in our pilot cities this was
overwhelmingly the result, and I still can't think of any incremental change
that we could have done to get the non-profits engaged. A big part of it was
that most of them were still uncertain about the whole email thing. It's not
an exaggeration to say that some of them didn't know how to use a spreadsheet.

With our fancy web 2.0 ajaxy website, we felt like we were giving out iTunes
gift cards to the lost tribes of Papua New Guinea. They didn't value it
because they weren't even close to being at the point where they could use it.

I care about social good - I wouldn't have founded a social venture if not.
And twice now I've been floored by the state of the non-profit world. Again,
10% of them are brilliant, but the other 90%... It's a big problem, and it's
not going to get better by pretending that it's not there. That world is
amazingly backwards. And it's saddening to see such important mandates being
handled by such ineffective organizations.

My advice to anyone here planning on doing a social venture: drop backwards
compatibility with the non-profit world. (one giant caveat: working
exclusively with one or two of the few clued-in orgs might work).

~~~
djm
A lot of what you are saying here resonates with me. I work part-time for a
local charity in my town and have been involved in various others at times.
There are a number of problems charities normally face which may explain your
findings to some extent, including:

\- well meaning but incompetent staff. People don't seem to understand that
wanting to do good doesn't naturally translate into the ability to do it.

\- no real management. I suppose this may be local phenomenon, but where I am
(UK), there seems to be a tendency for small charities to be managed by
wealthy housewives as a kind of hobby job. There is very definitely a social
network of them that dominate the charities in my town. Often this means the
management will be consistently absent from the organization. You can also see
my previous point.

\- no real check on the management. Charity trustees are often people who just
want their name to be associated with a charitable organization. I can think
of several trustees of the charity I work for who probably have only a vague
idea of what we actually do.

\- funding constraints. Charities in the UK who receive funding from
government departments are usually monitored in terms of the volume of work
they do rather than the outcomes they produce (which are much harder to
quantify). Thus the incentive is often to not try to improve efficiency as
reducing the work load may negatively impact funding.

Obviously my examples are to some extent just personal experiences that may
not be universal. However I would guess that most non-profits face similar
problems and that these problems may help explain why you got such a negative
reception with your projects.

With regards to volunteers though, what did you to do assess their competency
to perform the tasks they were willing to help out with? I ask because part of
my job is to manage my organization's volunteers (which mostly come to us on
work experience placements from the job centre). They frequently start working
for us claiming to have various skills and then we discover that they often
don't. For this reason I am usually quite unhappy about accepting new
volunteers - when I have to get rid of them is is much harder (emotionally,
for me) than sacking an incompetent employee. This might also help explain
some of the resistance you encountered to what you were doing.

------
Dove
> This isn't to say that we should have companies replace nonprofits

Yes we should.

Charities I donate to are selling me a product just as much as companies I buy
from. That the product is a better world--by some measure that seems
reasonable to me, rather than a simple benefit for myself, matters not a whit
to my decision making process.

I don't give without a very concrete understanding of what the money will be
used for, any more than I buy stock without a concrete understanding of what
I'm buying and for how much.

By all means, sell me peace, charity, and goodwill! Perhaps that will motivate
you to learn how to produce it, and to convince me that you have done so.

~~~
rapind
For-profit companies selling you goods and services often spend enormous
amounts on marketing, advertising, and branding in order to convince you to
part with your hard earned money. This doesn't mean their product is any
better than their competitors though. It just means they've spun you into
believing it is.

While I think it's a good idea to research your charity or donation to a non-
profit, I definitely do not think they should operate like a for-profit
business.

I think this is a great article, and there are some really pointed areas of
improvement for non-profits. However, let's not turn to the free market gods
for a solution please.

~~~
cwan
I think it really depends on the specific area/charitable cause - but where it
comes to economic development, markets are a far better solution than
charities and aid which have often done far more harm than good (e.g. food
aid). In general, the potential for inefficiency and even fraud is arguably
even higher given the lack of transparency and often weak governance).

Anyone who donates money to a charity should at least spend as much time
researching what they're "selling" as they would a comparable product/service.
The tax returns are available but some of these firms spend as much as 80% of
their funds raised in administration and overhead - which is often far worse
than most for profits. Yes, not for profits also "often spend enormous amounts
on marketing, advertising, and branding in order to convince you to part with
your hard earned money".

Fortunately at least anyone can look up any charity's IRS Form 990 that lists
such things as salaries (which are sometimes as much if not higher than
private market salaries). I usually start here: <http://www.guidestar.org/>
(using bugmenot for login/password)

------
tokenadult
As a board member of two very active nonprofit organizations, and former board
member of another, I really appreciate the suggestion to devote attention to
demonstrable results. I'll bring that up at our next board meeting for each
organization. It's easy to track membership, and reasonably routine to track
member satisfaction in membership organizations, and some organizations set
policy goals (within the scope of the IRS rules about political activity by
some kinds of nonprofits) that are verifiable. But, yes, organizations doing
work far from most supporters (e.g., foreign relief organizations or
missionary organizations) often can gain funds for years without having to
show results. It's regrettable when a willing donor isn't giving money to best
effect.

~~~
jseliger
"But, yes, organizations doing work far from most supporters (e.g., foreign
relief organizations or missionary organizations) often can gain funds for
years without having to show results. It's regrettable when a willing donor
isn't giving money to best effect."

Not to be a shill for my own blog again, but the reason so few organizations
run real evaluations is that it's really, really hard, expensive, and time-
consuming to do. Consequently, funders by and large don't offer real money or
incentives for it, and if they don't, there's not much of an impetus to
improve, and I describe this basic idea in much greater detail here:
[http://blog.seliger.com/2008/04/24/studying-programs-is-
hard...](http://blog.seliger.com/2008/04/24/studying-programs-is-hard-to-do-
why-its-hard-to-write-a-compelling-evaluation)

Furthermore, some funders don't want real evaluations, even if they say
otherwise; for example, we wrote a bunch of Community-Based Abstinence
Education proposals for various clients, where the main purpose of the
evaluation was to try and demonstrate that abstinence-only education works,
and I described the problems with that in this post:
[http://blog.seliger.com/2008/10/12/what-to-do-when-
research-...](http://blog.seliger.com/2008/10/12/what-to-do-when-research-
indicates-your-approach-is-unlikely-to-succeed-part-i-of-a-case-study-on-the-
community-based-abstinence-education-program-rfp)

~~~
cturner
The pool of funds is often not fixed. A non-profit can make themselves more
competitive for available funds by providing metrics.

~~~
jseliger
That's true in theory but not in practice because most funders don't care,
which is a point I describe further in the posts linked to above.

------
russell
I take his essay to mean that, if you want results, measure results and hold
them to it. Chess experts are experts because they are measured by their
results. Political pundits are hit hit or miss because they are not held
accountable for their results. Non-profits should be measured by their
results, not their process, PR, or grants. Makes sense to me, but quantifiable
results are often hard to get, witness the discussions here about how to to
tell goo programmers from bad.

~~~
fatdog789
How do you measure the "results" of a non-profit, when most of them work in
non-quantifiable fields?

You end up with the quantity versus quality argument, favoring large,
superficial efforts over narrowly tailored, substantive efforts.

~~~
patio11
_How do you measure the "results" of a non-profit, when most of them work in
non-quantifiable fields?_

If McDonalds can work out how to quantify feeding hungry people then any
charity working in Africa should be able to quantify feeding hungry people. If
an insurance company can tell me the average lifespan of their clients to six
decimal places and what exactly the risk factors are among that pool, then
health-focused charities should be able to do it as well. etc, etc

(I have some familiarity with the literature and know the excuses for why
charities operating in Africa with mobile phones and computers can't be
expected to have as good a handle on their numbers as a British bank circa
1780. Yeah yeah, it must be difficult when you rely on globally distributed
workers, many of whom are not literate. Oh wait, so does McDonalds.)

------
joanna
Another important thing to remember when comparing non-profits and companies
is the power of the customer. Companies go bankrupt because the customer has
the power to discontinue their patronage if they deliver a product or service
poorly (unless there are anticompetitive practices in play - ahem comcast).
However, in the non-profit world the recipient of the service or products has
very little power to motivate a market. Those who contribute money to non-
profits could theoretically be this force, but does someone who wants to
invest in reducing malaria really have to invest in understanding the health
metrics and do this assessment with each investment? I don't think that is a
permanent solution either. No doubt feedback and accountability are vital to
making non-profts more successful, but let's challenge ourselves to think of a
more systematic way to implement this type of change rather than summing with
measurement is needed.

~~~
stcredzero
_Companies go bankrupt because the customer has the power to discontinue their
patronage if they deliver a product or service poorly_

This definitely happens to many kinds of non-profit!

 _in the non-profit world the recipient of the service or products has very
little power to motivate a market._

For 501-c3 organizations that provide classes, lectures, and performances,
often the _opposite_ is the case. Many non-profits have to be run much like a
business. There can be something like a marketplace where the "customer" pays
not with currency, but with their time. (And sometimes, both money and time
are involved.) Also, very often, the "recipient" of services can easily
directly involve themselves with running the organization. (KPFT radio in
Houston is an example of this.)

~~~
anamax
>> Companies go bankrupt because the customer has the power to discontinue
their patronage if they deliver a product or service poorly

>This definitely happens to many kinds of non-profit!

Unless I missed the sarcasm, it doesn't happen with charitable non-profits.
They die when funding dries up, and since their customers don't provide
funding, their customers have little/no effect on their survival.

Yes, a charity can continue to exist long after folks stop showing up to
receive services.

~~~
dagw
_since their customers don't provide funding_

That depends on your point of view. I'd argue that the people supplying the
funding are the customers and that the work they do and people they help is
the service they provide. So if their customers aren't happy with the quality
of the service they'll take their money somewhere else, exactly like in the
for profit case.

~~~
yummyfajitas
The problem is the customers of nonprofits often cannot measure the results
themselves.

If McDonald's serves me a burger that is inedible, I know about it immediately
and can stop patronizing McD's.

If Burgers4Africa distributes inedible burgers in Somalia, how do I know about
it?

~~~
stcredzero
My point is that often the customers _can_ measure the results. The article is
worded as if this applies to _all_ non-profits.

------
enra
This is why I don't donate. Charities usually don't publish exact reports
what's the money is used on and what have they achieved.

Building schools doesn't help if you don't hire teachers and get kids attend
school and money used on clever guilt marketing doesn't solve ecological
problems.

------
netsp
_"This isn’t to say that we should have companies replace non-profits..."_

Actually, that is not necessarily a good thing to just sidestep. A lot of
things that non profits work on might work better if they were worked on by
companies. A lot of things that companies work on might have to be filled by
non profits if companies weren't doing it.

Think of second hand stores. They are often run not-for-profit in order to
provide cheap stuff to poor people. These compete directly with commercial low
cost stores and other second hand stores. If these beat the non profits, there
is no societal loss. One of the worlds most celebrated not for profits, the
Grameen Bank provides small loans to poor people. It seems very possible that
this NGO (and its many imitators) will be put out of business by commercial
banks.

The UN has started a free online University to help the world's poor. I would
take an either-or bet on for profit variants making a lot of progress on this
issue commercially.

A great way of making non profits more like chess masters is making non
profits into businesses. You don't always have to. But if it is feasible, its
worht considering.

~~~
carbon8
_"Grameen Bank provides small loans to poor people. It seems very possible
that this NGO (and its many imitators) will be put out of business by
commercial banks."_

Traditional, for-profit banking simply can't, which is the entire point. Even
when microfinance institutions lean further toward for-profit, they end up
more and more like traditional banks making traditional big loans.

And that's the way it actually moves, with microfinance institutions edging
toward becoming more commercialized, and when they drift too far away from
poorer clients, new micro institutions will likely come in to take their
place. In other words, the shift naturally moves in the opposite direction.

~~~
skybrian
Compartamos in Mexico is a for-profit bank making microloans. There's some
controversy about that:
[http://www.nytimes.com/2008/04/05/business/worldbusiness/05m...](http://www.nytimes.com/2008/04/05/business/worldbusiness/05micro.html)

~~~
carbon8
Indeed, it's former NGO that aggressively went for-profit. That's a good
article and sums up some of the major debates around latin american
microfinance.

The problem with for-profit microfinance is that, as a general rule, financial
performance increases with loan size, and that's why commercialization is a
cause for concern, along with a lot of the other baggage that comes with
commercialization like pressure to increase profits and employee incentive
programs that negatively impact service toward small clients.

Compartamos accounts for the problems of the small loan market (notably the
inability to determine which borrows are good or not) the same way
moneylenders needed to before them, by charging very high interest rates. But
this ends up screening out lower end borrowers, hence the discussion of
"mission drift."

Commercial banks have long dabbled in elements of microfinance, but there are
fairly concrete obstacles, mostly operational and information-related issues,
that make it tough to be profitable, and even tougher to approach it as
strictly for-profit while continuing to be actual microfinance.

------
wglb
This is also interesting in how it relates to the study of airplane crashes
and the strong, relatively immediate feedback that these provide. It seems
that auto safety was much more slow to catch on, and required odd forces to
bring the topic to the forefront.

This also has an interesting interaction with a small startup essentially
searching for the important metric for its survival.

~~~
cwan
For profits have the benefit of having profit/positive cash flow as a guide
that not for profits do not. These are probably much better metrics for small
startups given that success in any stated altruistic endeavours are ultimately
going to show up in these two metrics anyway (and must in order for a startup
to be sustainable).

------
baran
This is _one_ of the reasons why non-profit companies are flawed. There are no
incentives in place for monetary gain. Non-profits are good at raising
awareness, not being a sustainable entity.

If someone wants to make a difference and change world (especially developing
countries), I believe there has to be financial incentives : _social
entrepreneurship_.

~~~
billswift
If you can't make money doing something, is there any good evidence that it is
worth doing at all? Leaving aside goods with "free" competition from the gov't
(schools, roads, libraries, etc). EDIT: Also leaving aside things you do for
your more direct benefit, like growing your own food or contributing to open
source projects you use.

~~~
fhars
Did you really ask "is there any value in leading a moral life if it doesn't
pay off financially?"

Non-profits do not actually aim at financial gain (duh!), so measuring their
success in monetary term is an error of categories. Of course it is difficult
to assess success of an endeavour like that, and Aaron is right in raising
awarness for these difficulties, but forcing an inappropriate kind of
measurement (did your non-profit make enough profit?) on them doesn't help
solving this problem.

The correct question to ask are "what are our goals?" "How do we go about
reaching our goals?" "Did our past activities in fact further our goals?" and
so on. "how much more money did we raise and distribute" may factor in the
analysis, but on its own it is pretty meaningless.

~~~
billswift
There are many reasons to do things other than money, but when you are dealing
with people that you don't personally or directly know the fact that they are
willing to pay for your services is basically the only real signal you have as
to whether you are benefiting them. And if you are not making more doing this
(for a combination of monetary and non-monetary benefits) than you could
something else, then you need to switch to a better use of your time.

Of course, the contributors benefit from donating money, etc by signalling
effects and warm fuzzies, but that doesn't mean the recipients are benefiting
nearly as much as the contributors are spending.

~~~
billswift
As Friedman put it in "Machinery of Freedom", there are basically only three
ways to work with others - love or shared goals, free exchange, or coercion.

------
chaostheory
"Making a bad prediction isn’t like that... it’s months or years before your
prediction is proven wrong. And then, you make yourself feel better by coming
up with some explanation for why you were wrong: ...And so you keep on making
predictions in the same way — which means you never get good at it."

I think this also applies to many macro-economists

------
fhars
There is of coure a very important problem charities trying to do this kind of
measurments of efficiency face. They are in a market for public perceptiom to
get donations, and (if we take the numbers the article cites for Oxfam), can
you imagine the loss of donations and goodwill after a few newspapers and TV
programs run the headline "Charity X spends half their donations on statistics
instead of helping people!"? Even if the costs could be amortized over a few
years, the charity would be dead by then.

Perverse incentive factors, again. (And the news might be right, collecting in
depth statistics about the help program might just be a pretext to give a
lucrative consulting contract to the director's best friend).

------
chasingsparks
I would argue that there is a second, equally-important, albeit interrelated,
problem with non-profits in general: they are highly risk averse. Many non-
profits still operate under a create an endowment and dispense funds to proven
-- or at least perceptive -- good projects model. Consequently, there is a "if
it ain't broke, don't fix it" mentality, which is to say weak experimentation.

I should mention that most of my non-profit experience deals with medical
research foundations. Political think tanks are limited by the general
impossibility of proving all but trivial problems.

------
netsp
Does anyone have a recommendation for a charity with demonstrable results in
improving the lives of people in a measurable way?

~~~
aaronsw
<http://givewell.net/> investigates charities to make sure they achieve
demonstrable results, they have a list of those that do.

There are also meta-charities like GiveWell and the MIT Poverty Action Lab
that investigate other charitable attempts.

~~~
blasdel
_Fuck that noise_ \-- if you'd have even bothered to google their name, you'd
have found that Givewell are total scumbags:

[http://metatalk.metafilter.com/15547/GiveWell-or-Give-em-
Hel...](http://metatalk.metafilter.com/15547/GiveWell-or-Give-em-Hell)

Summaries: <http://mssv.net/wiki/index.php/Givewell>

They got caught out because they were blatantly astroturfing for themselves
and slagging 'competing' charities all over the web. Then it turns out that
even by their own generous accounting, more than half of their funds goes
directly to their own overhead. On top of that, their 'research' is at best
based on getting charities to rigorously assess themselves.

Their rubberstamp board publicly (in the NYT) pretended to fire one of the
founders, but less than a year later he was quietly back at his old position.

~~~
netsp
You would expect that any organisation doing this kind of work would attract
criticism. There is a taboo against harming a charities ability to raise
funds.

Do you know of any organisation doing similar but better?

~~~
req2
There's <http://www.charitynavigator.org/>, and the arguments against:
[http://tacticalphilanthropy.com/2007/11/charity-
navigator%E2...](http://tacticalphilanthropy.com/2007/11/charity-
navigator%E2%80%99s-vital-mission-hides-flawed-rankings)

------
riffer
This is extensible well beyond non-profits. The general form is still rough in
my mind but it is something like "avoid situations where accountability and
feedback mechanisms are not essential parts of the environment"

------
yankeeracer73
A good book just came out about this very issue:
[http://www.amazon.com/gp/product/0470454679/ref=ox_ya_os_pro...](http://www.amazon.com/gp/product/0470454679/ref=ox_ya_os_product)

------
MaysonL
Of course, if one looks at the profit-seeking world, one sees multiple
examples of organizations blind to feedback: GM springs to mind these days.

~~~
billswift
True, but responding to feedback is a lot harder than being aware of it. And
especially so in a larger organization. I'm not so sure that GM's problems
stemmed from not being aware of the problems as not being able to respond for
various structural, legal, and social reasons.

------
req2
Somewhat related, on charity: <http://news.ycombinator.com/item?id=800663>

