
Why the snow forecast for New York City was so bad - cryptoz
http://www.washingtonpost.com/blogs/capital-weather-gang/wp/2015/01/27/why-the-snow-forecast-for-new-york-city-was-so-bad-and-what-should-be-done/
======
acheron
More to the point, who cares? The storm was still pretty bad: this isn't a
case of a "bust" forecast. Long Island got hit, as well as much of New
England.

So the line on the map of where the heaviest snow would be was a bit off: the
only reason this is "news" at all is because that line happened to affect New
York City, and there are a lot of journalists who live there whose view of the
world matches that New Yorker cover[1]. If the line had been anywhere else we
wouldn't have heard about it at all.

[1]
[https://en.wikipedia.org/wiki/View_of_the_World_from_9th_Ave...](https://en.wikipedia.org/wiki/View_of_the_World_from_9th_Avenue)

~~~
DontBeADick
> More to the point, who cares?

Did you forget about the entire city of Philadelphia? 14 inches predicted, < 1
inch actual snowfall.

The city is practically a ghost town today thanks to a botched weather
forecast. Tons of businesses and schools closed preemptively because they were
predicting massive snowfall totals with what seemed like absolute certainty.
According to meteorologists here, it wasn't about whether or not we were
getting snow, it was about whether we were getting 1 foot or 2 feet.

~~~
forgottenpass
_Did you forget about the entire city of Philadelphia? 14 inches predicted, <
1 inch actual snowfall._

OK, but so what? This is about unpredictability, not over-estimation. If the
unpredictability was better conveyed the best case scenario is still everyone
preparing like they did for a storm that didn't hit. Worst case is suffering
hardships due to underestimating the storm.

I'm a bit further northeast and it's coming down pretty hard outside my window
right now. The preemptive cancellations really did avoid a lot of unsafe
travel here. We have a bit more familiarity with the unpredictability of
snowfall predictions, but it usually manifests as people saying: "It won't be
_that_ bad" and getting into trouble the few times it really is that bad.

It didn't hit the cities that aren't used to big snowfall. But those cities
house a lot of national media, so now we get to listen their scorn for the
next week.

------
cryptoz
While this article mentions that the GFS model was more accurate than the
others (ECMWF & NAM), it neglects to give the backstory as to why it was more
accurate and less trusted. For years the GFS has been made fun of because of
its poor performance when compared to the Euro - and it just got the major
upgrades it needed this month! New supercomputers, new high-res models, all
went online just before the storm arrived. So, very few trusted the model even
though it was in the best spot to get everything right.

In an unexpected move, The Weather Channel was much more accurate than the US
NWS / AccuWeather / etc because of their use of the GFS as a trusted model.
More info in this article: [http://www.alternet.org/environment/why-almost-
everyone-got-...](http://www.alternet.org/environment/why-almost-everyone-got-
snowpocalypse-wrong)

Edit: And of course, it's my opinion that all the models are currently
suffering from a lack of input data. The "...and what should be done" part of
the headline, in my opinion, should be answered with, "everyone download
PressureNet and contribute your phone's sensor data to weather models", so
that we can better predict difficult quantities like heavy snowfall and severe
thunderstorms:
[https://play.google.com/store/apps/details?id=ca.cumulonimbu...](https://play.google.com/store/apps/details?id=ca.cumulonimbus.barometernetwork)
;)

~~~
potash
Do you know of any research about the accuracy of forecasts as a function of
sensor data input? I.e. what kind of an improvement will we see if some number
of Android users install PressureNet?

~~~
cryptoz
Here is a recently published paper by Cliff Mass that is the first attempt at
forecast experiments with smartphone pressure data:
[http://journals.ametsoc.org/doi/full/10.1175/BAMS-D-13-00188...](http://journals.ametsoc.org/doi/full/10.1175/BAMS-D-13-00188.1)

The early results are good. There will be an improvement [1]. How much of an
improvement? We don't know yet. Cliff thinks it could be a revolution for some
types of forecasts, but we don't have the density of sensors yet to know for
sure.

For some comparisons, we sent cliff about 20,000 measurements per hour for his
experiment in that paper. We're now delivering about 200,000 per hour to
researchers, and that's not nearly enough. Our aim is for 2M per hour and I
hope to reach that in the next 2-3 months. Around 1-2M per hour is probably
sufficient to provide the "revolution" in accuracy that Cliff predicting.

I should also note that 1-2M per hour is small. We should be able to get
closer to 1B per hour, but it'll take a while to ramp up to that kind of scale
(that'll be like, every smartphone + watch + car that has a barometer).

The improvements will be slow and steady until we get massive scale and are
able to run our models in real time. Until then, it's tough to guess how good
the improvements will be.

[1] In the linked paper above, I believe the results were a reduction in root
mean square error of about 1deg C for a 3-hour temperature forecast in the
Pacific Northwest.

------
jeffreyrogers
Alternative hypothesis that also fits the evidence: You look bad if reality
turns out worse than your forecast. You look less bad if reality turns out
better than your forecast, thus there is a built in incentive to make
forecasts more pessimistic than is justified.

Imagine if 10in were forecast and then NYC actually got 2 feet. Then everyone
would be complaining that the city wasn't adequately prepared for the 14 extra
inches they received and would blame the forecasters for all the economic
consequences of not being prepared.

~~~
spikels
If I'm understanding the article correctly, this seems correct. Of the many
models almost everyone chose to emphasized the models with the more extreme
output. Different parties had different motivations: selling news, avoiding
political blame...

Interesting things happen when science meets human nature. Science rarely wins
especially if there is uncertainty.

~~~
jeffreyrogers
> Science rarely wins especially if there is uncertainty.

I agree 100%. One of the most interesting books I've read lately has been The
Black Swan by Nassim Taleb, which deals with this at length (ostensibly it is
about unpredictability in financial markets, but the implications are far
broader than that).

------
dougk16
After watching the same ridiculous news reporting touting "historic", "storm
of the century", "unprecedented", etc., year after year, the more cynical side
of me suggests that there's a small supplemental reason besides flawed models:
there's probably good money to be made from overhyping storms. At the very
least news stations must get great ratings the day or two beforehand. Note
that this doesn't preclude accurate estimating...you just have to pull a bunch
of cheap psychological tricks to get people to focus on the worst case
scenario. The end result of people freaking out is still largely the same.

One reason this comes to mind is that I can't ever remember a storm reaching
its worst-case snowfall predictions, and usually they barely make the
minimums. A rule of thumb I use is to treat the minimum prediction as the
actual maximum and zero as the actual minimum. Anecdotal, but seems to work
well.

~~~
icehawk219
The "play up the fear" stuff around weather reporting in recent years is why
I've given up even attempting to use weather forecasts from anywhere but
weather.gov which is sourced directly from The National Weather Service and
NOAA. It's just a straight up information dump, no annoying ads or fear
mongering for clicks. When I heard they were naming blizzards last year, and
this year now, I was only momentarily shocked because it was the next logical
step.

This has far more severe consequences then just clickbaiting though. When
Hurricane Irene was moving up the coast a few years ago they did the typical
"storm of the century, omg a billion people might die!!!" routine and we got
light wind and rain. The following year they did it again with Hurricane Sandy
(I really hate when people call it Superstorm Sandy ... it was a small
hurricane! We were just ill-prepared for it) and a whole lot of people didn't
buy into it and paid the price. I'm only 27 but I have trouble remembering a
time where every thunderstorm we got wasn't played up to be an extinction
level event. At this point I have my small cache of supplies and otherwise
completely ignore storm predictions for the most part.

~~~
testguy34
It _was_ the NWS in this case who was playing up the fear. The useual suspect,
The Weather Channel, is the only one who got it right.

~~~
icehawk219
I didn't realize that. My only exposure to weather reports at this point is
actually just going to weather.gov and saying "ok, might get snow, I'll go the
store tonight instead of tomorrow". And stuff like this is precisely why.

------
jacques_chester
Several years ago in Australia, a blogger called Possum Comitatus, aka Scott
Steel, used to mock journalists for breathlessly reporting every twitch of the
polls, even though most such movements were within the margin of error.

So successful was this mockery that the way polls are reported actually
_changed_. It become normal to report a change in polls with a remark about
whether the change was within the margin. Political race-calling has moved
onto less abuses of statistics.

Better reporting can be done, and done with surprising ease, using a simple
tool.

Mockery of journalists.

To gain status and prestige over their peers, some will adopt the better
method. In a few months they'll all do it.

------
basseq
Three problems:

1\. Uncertainty isn't actionable. A statement like, "There's an 85% chance of
snow with accumulation of 1–24" with peak probabilities of 28% and 17% at 2"
and 18", respectively" is nonsense to most people. And moreover, it doesn't
mean anything to me. Are we going to get snow that will affect my morning
commute? Are we going to get so much snow that I need to take emergency
action? A probability distribution doesn't answer those questions, even if
that's the best that exists.

2\. Weather is news. The cable news channels were dominated by snow
predictions yesterday, and the Weather Channel exists as an "entertainment"
venue, not a source of scientific information. If it doesn't fit into a
soundbite, the public can't absorb it. And "Weather is hard to predict"
doesn't garner viewership.

3\. Prepare for the worst. While small changes in the environment can lead to
large variability in snow accumulation, the fact of the matter is that there
was a not-unlikely chance (according to the models) that NYC, Boston, and
other cities could have seen crippling snowfall. They were able to prepare for
that eventuality. That it didn't come to pass is almost a non-issue. Can you
imagine the outcry—not to mention the impact—if the situation was reversed
(i.e., surprise 3' of snow)? Fairfax County Public Schools outside of DC got
_slammed_ in the media for not closing a few weeks back.

The real question is: at what point of probability do you prepare for a
potential outcome, and are you prepared to back that up (either way)? Do we
prepare for record snowfall on 80% likelihood? 50%? 10%? (And as a broader
point of discussion, this same calculation applies to other areas: TSA spend
vs. terrorism; flood insurance cost vs. coverage; etc.)

~~~
Goronmon
_the fact of the matter is that there was a not-unlikely chance (according to
the models) that NYC, Boston, and other cities could have seen crippling
snowfall. They were able to prepare for that eventuality. That it didn 't come
to pass is almost a non-issue._

The thing is, for the Boston area at least, the crippling snowfall totals came
true. Areas west of Boston were reporting well over two feet of snow earlier
in the day.

~~~
emhart
Yeah, southwest of Boston we are getting wrecked. Still coming down hard,
hoping it finishes up soon. I'm glad we had the warnings, and I'll be
particularly glad if we lose power, which seems pretty likely at this point,
since the warning gave me the chance to stock up, charge battery backups, etc.

------
joshdick
People don't like it when 14 inches is forecast and the actual snowfall is 2
inches, but they aren't going to like hearing forecasts of 2-16 inches.

That sort of wide confidence interval is more accurate, but it's also
basically useless. How does a 2-16 inch forecast help a parent or school
district or mayor plan for anything?

~~~
saalweachter
A single 9x% confidence interval like 2-16 inches gives the impression of a
continuous bell-curve distribution, when you might have a multiple peaks based
on a couple of simple factors. I wonder if it would be useful to give out a
couple of probability buckets. If people heard "20% chance of 10 or more
inches, 60% chance of less than 2 inches", would then know to be ready for a
lot of snow, but not be surprised if it doesn't happen, or would they ignore
the considerable chance of a heavy snowfall?

------
scott_s
We should always strive to improve our predictions, but I see a fundamental
dichotomy: do you prefer false-positives, or false-negatives? We don't get to
pick no wrong results.

When it comes to predictions like snow storms, hurricanes and tornadoes, you
obviously want to favor false-positives. It's better to predict something bad
will happen, pay the preparation costs and have nothing happen, than to have
something bad happen when you're unprepared.

So, yes, we should try to improve our predictions. And perhaps we should be
better at communicating our uncertainty. But I don't blame the meteorologists,
or the people in government who shut things down and told everyone to prepare.

------
Sharlin
I was delighted to notice a while ago that the Finnish Meteorological
Institute had added 50% and 80% confidence intervals to their 10-day online
forecasts, for instance:
[http://en.ilmatieteenlaitos.fi/weather/turku?forecast=long](http://en.ilmatieteenlaitos.fi/weather/turku?forecast=long)

------
mikeash
Note that this article is written by the Capital Weather Gang, a group of DC-
area meteorologists who consistently put out accurate and nuanced short-range
forecasts of upcoming weather in the area. They give out great info and they
aren't afraid to state their uncertainties up front, nor are they afraid to
admit it when they're wrong (which is not too frequent).

They give an implicit answer to the question of "how could everybody do
better?" which is "be more like us." And I think that would be great.

------
fideloper
I was curious about what kind of bump weather sites get in ad revenue as a
result of storm hype.

That's an interesting angle when wondering where the hype is driven from ...
if you find the mental image of Murdoch twirling his mustache (this is my
imagination, so why not add a mustache?) while making quiet phone calls from a
back room a correct version of reality.

------
JackFr
How is it possible that millions of New Yorkers accurately discounted the
probability of it being a 500-year storm and yet the mayor didn't? The
responsibility must lie with either the Mayor or the meteorologists. And there
is something to own up to -- shutting down the city is not without real cost
to real people. The mayor and governor will stay it was for our own good and
better safe than sorry. But put a few million people under house arrest (no
using roads -- under threat of arrest, no public transport) for basically no
reason.

But a few things to consider: 1) The meteorologists with the most dire
predictions are the one who are going to be picked up by the media -- the
media needs to sell ads. 2) The bulk of the population accurately discounted
the most dire predictions, because of well-calibrated but subconscious
Bayesian calculation. 3) Bureaucrats, and to some extent politicians of both
parties, have a well-intentioned, but costly and wrong, need to subvert the
judgement of millions and replace it with that of a handful of experts. When
the judgement proves wrong, they rest on their good intentions without
assuming responsibility. 4) In subverting these judgements bureaucrats and
politicians can and do rely on the states monopoly on force.

------
mathattack
It seems like the costs are asymmetric on misses. If a bunch of people die,
the mayor gets voted out of office. If they prematurely close the city for a
day, the mayor just apologizes afterwards.

------
toephu2
We are in the year of 2015, and yet we still cannot accurately predict the
weather...

