

Those Who Care Not For Trees - glaugh
http://blog.statwing.com/those-who-care-not-for-trees/

======
nkurz
I realize this is post is intended as a light-hearted sales pitch rather than
a deep analysis, but I think the conclusions are misguided.

The overriding problem is that you are trying to presenting conclusions based
on 300 responses out of 10,000 people surveyed, but never mention the
possibility of measurement error. Even ignoring intentionally incorrect
responses, considering that these respondents appear "cranky", how many might
inverted the 1 to 4 scale? How many might have been one row off?

To go further and break down this 3% of respondents into age, region, and
income is very prone to overfitting. Are these conclusions consistent across
the multiple years of the survey? How many "over 74" tree-haters were there
each year? Are they more or less likely to miskey an answer?

 _How do you feel about the current number of trees in your city? Too many?
About right? Not enough? You probably didn’t answer “Too many."_

Why presume that everyone wants more trees? Is it hard to believe that there
might be a point where a neighborhood could have too many, and that different
people might have different thresholds for this? Your rhetorical question
presumes a strange form of cultural diversity bounded by moral certitude.

I've met many Americans raised in the rural West who feel hemmed in by trees.
I've met Australian ranchers who consider them to be weeds that steal ground
water that could otherwise grow grass. There are numerous accidents caused by
obscured street signs, and cities remove trees all the time. I know allergy
sufferers everywhere who have very strong opinions on which trees they like
and which they despise. And while I love fruit trees, many see them mostly as
food for rats.

Going one step further, even if you think the study is completely accurate,
and the demographic numbers large enough to be significant, the post misses
what to me seems like the obvious question: do the responses correlate to the
number of trees in each area? Do the areas with fewer trees have more people
who want more? Do the areas with the most have the most who are satisfied? If
not, why not?

Sorry for the vehemence. I'm reading lots of articles lately that misuse
statistics, and I don't think this post puts your company in a good light.

~~~
glaugh
OP here. Thanks for the comments.

For readability's sake we did not present the statistical side of the results,
but they were unambiguous for everything we mentioned in the post. Perhaps in
the future we should present those results in footnotes to assure inquiring
minds that we addressed valid concerns like yours (e.g., overfitting).

Love the examples of valid reasons folks don't like trees. Almost any of them
seem like viable hypotheses to explore for why folks fall in the 3% (though
obviously we'd need very different data from that which we have handy).

To address the specific comment about trees per neighborhood: unfortunately we
didn't have granular enough data about the locations of respondents to analyze
that; agreed that there's likely something there.

Thanks again for the comments.

Late followup edit: Thanks to your suggestions, we added some clarifying
footnotes to the post.

. Phone survey results were same as written survey results, so the issue isn't
measurement error.

. All findings were backed by statistical testing, the p-values for which were
almost always below 0.0001.

. Tests were replicated year over year; didn't add an additional note about
overfitting but doesn't seem like a big concern with 300 datapoints and just a
few variables of particular interest.

It wasn't that hard to stick this into the post as footnotes; in the future
we'll do better with that. Thanks for the help.

~~~
nkurz
Thanks, I appreciate the updates and thoughts. I did dive a little into the
data, and yes, there does seem to be something real going on. Interestingly,
there are quite a few people who answered that their neighborhood had too many
trees, but the city as a whole had about the right number. Makes me wonder if
this is speaking about "fairness" rather than actual tree preferences.

The statistical testing is tricky, though. Did you go in with the thesis that
there might be a correlation between age and ideal number of trees, or did you
first pin down "too many trees" and then search the data for correlation? Or
even more problematically, did you search for cross-correlations on all
columns and then discover the age-tree link? If either of the second, your
definition of "significance" needs to be custom.

~~~
glaugh
Went in there with the aim of exploring the trees question, and hypothesized
there might be relationships between that and obvious demographic questions.
The "crankiness" bit was much more driven by exploratory analysis.

We take a fairly pragmatic approach to the multiple comparisons problem you're
highlighting. If p-values are < 0.00001 we don't really worry unless we're
doing a crazy amount of analyses. And when we do get p-values that are
borderline (like .01) and we're doing multiple comparisons we'll mention that
there may or may not be something actually happening there (as we did in our
post about Baltimore parking tickets: [http://blog.statwing.com/baltimore-
parking-tickets-revisited...](http://blog.statwing.com/baltimore-parking-
tickets-revisited/)).

For what it's worth, I think ceph_'s comment and link below
(<http://online.wsj.com/article/SB116165781554501615.html>) about the
relationship between trees and gentrification is the best available guess as
to what's ultimately driving these attitudes. But yeah, it's probably a
combination of a lot of things, many of which aren't actually about trees per
se.

------
scrumper
Interesting enough. Perhaps 3.9% of the respondents didn't like having their
time wasted with a survey, so decided to be deliberately awkward in every
response they made. This would seem to fit in with the 'lumberjacks' being
older and having a lower income than other respondents: both characteristics
can be associated with a kind of hard-nosed practicality and a disdain for
treehuggers and silly surveys.

I'm sure there's a law somewhere that says that no matter how completely
indefensible a position, there is some contrary-minded group who will loudly
proclaim allegiance to it. And if there isn't, there is now.

(Edit: OP, not suggesting your survey was silly, just that the 3.9% might have
thought so.)

~~~
hermannj314
The respondents in general came from one geographical part of the city and
also grade the quality of service of the city more harshly than others.

I interpret this that the parks department might be doing a really bad job in
certain parts of the city. The trees are not kept well and trimmed,
obstructing views and making driving unsafe, branches in the street after
heavy winds, something like that. Now if I already think the city is doing a
piss poor job, I'm sure not going to give them the justification they need to
raise my taxes just so they can plant more trees (when they can't be trusted
to take care of the ones that already exist).

In other words, the respondent's answer depends on the who they think is
asking the question and how they think their answer might be used.

~~~
scrumper
Very good point about this specifically. They live in a poorer part of the
city. It shouldn't be that way, but it may well be that their neighbourhoods
aren't as well tended as some of the richer parts of San Francisco.

> In other words, the respondent's answer depends on the who they think is
> asking the question and how they think their answer might be used.

Also, a good point about surveys in general.

------
tzs
I'd expect that many of the responses from people who would prefer less trees
are from people who have specific trees that they are annoyed with. For
instance, if there are some city-owned trees that make it so you have to clean
your gutters often, or that are blocking what would otherwise be a spectacular
(and property value boosting) view, I can see thinking that your city has too
many trees.

~~~
charonn0
View-blocking is a major problem in a beautiful city with lots of trees.

------
duck
I think calling these people lumberjacks is a bit off, since lumberjacks
actually love trees and want to see more of them as well.

~~~
glaugh
Fair point. Any suggestions for a sorta silly label for a group that genuinely
doesn't like trees? Went with lumberjacks to avoid saying anything overly
disparaging, but it's not great.

~~~
sunspeck
Treeshruggers

~~~
callum85
This is the one

~~~
glaugh
Agreed. Sorry we didn't act on this sooner.

------
ceph_
Trees are seen as a step towards gentrification to some. Efforts to plant more
trees in the tenderloin have been actively rallied against.[1] As they
believed the increase in property values they would no longer be able to live
there.

[1] <http://online.wsj.com/article/SB116165781554501615.html>

~~~
glaugh
Good find. That makes a lot of sense. Added a note about this at the bottom of
the blog post.

------
gyardley
Small percentages like these aren't necessarily worth reading much into, and
they don't necessarily have anything to do with the question being asked.

It could very well be that lower-income, older people (who _would_
disproportionately live in the southwest) are disproportionately likely to
make mistakes when filling out surveys.

Use of a letter-grade system also introduces error, since the value of a
particular letter grade is subjective and has varied over time. It could very
well be that older people, who received their own grades before the grade
inflation of the last generation, have stricter standards when it comes to
assigning letter grades to things.

~~~
glaugh
Didn't dive into it in the post, but we settled the survey-entry-mistake
hypothesis to our satisfaction by looking only at the responses that were
gathered via phone (about 1/4 of the dataset) and finding that the same trends
still held true.

Also, I really wanted the grade inflation hypothesis to be true, but I just
took another look at the data and age turns out not to be consistently
indicative of lower or higher grades for city services.

Thanks for the comment. Both really valid questions/comments.

------
zalew
off-topic: if you link to footnotes, provide a link back to the place I was
reading.

~~~
devb
Does any website do that? Does the back button not work well enough?

~~~
TeMPOraL
Wikipedia does. Unfortunately pg's essays don't, which is a shame, because it
would help the reading experience much.

------
goggles99
Aside from the fact that this survey is worthless because of a gross lack of
data - 3% of people surveyed probably don't travel into the city much and live
in areas with dense tre growth. If you lived their life you would probably
agree with them, instead you question if they are just grumpy.

------
bravoyankee
Are these the same people who litter, or would walk past litter even if it was
right in front of them?

