
NOAA upgrades the U.S. global weather forecast model - mehrdadn
https://www.noaa.gov/media-release/noaa-upgrades-us-global-weather-forecast-model
======
peterlk
If anyone is interested in doing your own thing with weather data, check out
MADIS [0]. There are various levels of access, some of which require NOAA
approval. But if you're serious about making weather predictions, it's a good
thread to pull on. I once set up a MADIS node, and our server was shut down
very quickly by Amazon for "suspicious traffic", so beware of that - there's a
lot of data that gets pushed through the system. If I remember correctly, it
was kind of a pain in the ass to get set up/configured, but it was pretty
cool.

[0] [https://madis.noaa.gov/index.shtml](https://madis.noaa.gov/index.shtml)

~~~
Something1234
Can you go in to more details about how to get this setup?

~~~
peterlk
I honestly don't remember any more. At the time, we were working with NOAA,
and I remember a problem that was solved by talking to an admin at NOAA (our
IP needed to be on some official whitelist or something), but that may have
been for a restricted data set. We didn't end up using it for long because the
client said so.

But I dug around for some information to maybe get you started.

Installation:
[https://madis.ncep.noaa.gov/doc/INSTALL.unix](https://madis.ncep.noaa.gov/doc/INSTALL.unix)

API:
[https://madis.ncep.noaa.gov/madis_api.shtml](https://madis.ncep.noaa.gov/madis_api.shtml)

Data restrictions:
[https://madis.ncep.noaa.gov/madis_restrictions.shtml](https://madis.ncep.noaa.gov/madis_restrictions.shtml)

Another resource that may help: [https://press3.mcs.anl.gov/forest/regional-
models/global-dat...](https://press3.mcs.anl.gov/forest/regional-
models/global-data-acquisition/)

When I was working on this stuff, I found that a DFS on various government
subdomains (like MADIS) was the best way to find information. It was tedious,
but it worked.

It's also helpful to put on your fortran hat. For example, I once attended a
Haskell meetup where someone wrote a parser to deal with parsing binary files
from NOAA. I also was in a meeting (with some NOAA folks) once where I was
asked if I "would prefer an ASCII file, or a binary one". This is not a world
that operates on JSON or XML. Expect binary blobs with flags (bits) that
change the meaning of other flags in fun and exotic ways. The binary nature of
the data can help with data throughput limits, but boy is it a pain to deal
with.

~~~
jcadam
> This is not a world that operates on JSON or XML. Expect binary blobs with
> flags (bits) that change the meaning of other flags in fun and exotic ways.

That bring back memories... As a government contractor I've had to work with
sensor data (seismic, radar, etc) in various formats that were developed well
before the rise of XML and JSON :(

My favorite was a mixed ASCII and binary format, where each data record in a
file had an ASCII header that described the format of the following block of
binary, and pretty much anything could be different between records, even
within the same data file (Time units? Integers? Floats? 16 bit integers? 64
bit? Big/Little Endian?).

I had to write a parser for that :'(

~~~
moftz
The most "fun" I've ever had was decoding command and telemetry from piece of
equipment for a ground station. The box would spit out this massive frame of
data. It was a very long ASCII string that you would turn into binary to break
into 6bit BCD values (no clue why they didn't use 4bit...). There were random
flags of odd bit lengths (sometimes just a single bit, sometimes 5bits) thrown
in between numbers for arbitrary reasons rather than just having all the
binary flags up front. My python script was this ugly mess of slicing up the
frame to turn it all into a very nice struct I could pass to the rest of the
system. The manual with this piece of hardware was some old scan that must
have been xeroxed a million times over so some portions of the document were
just unreadable and you had to guess what those bits did. Other parts of the
frame were just undocumented. Commands were send one by one as single letter
with the actual ASCII representation of the numerical command parameter.

When I started the project, I looked online to see if anyone had done any
previous work on this thing. A vendor was selling a GUI for the thing for
$2000, I scoffed at the price and started working it myself. By the time I was
done, it had probably cost my employer more than that but at least we had our
own code that could connect to whatever you wanted rather than a GUI with a no
API.

~~~
kaybe
Did you try to sell it too?

~~~
moftz
It was an internal project for a large company so no.

------
quico
The fact sheet showcases the improvements in the forecast:
[https://www.noaa.gov/sites/default/files/atoms/files/DOCUMEN...](https://www.noaa.gov/sites/default/files/atoms/files/DOCUMENT%20-%20Global%20forecast%20system%20fact%20sheet.pdf)

------
chiph
This is probably in response to the Euro weather model, which has been
producing better forecasts. Especially for named storms.

~~~
exhilaration
The Europeans are putting big money into weather forecasting, "The goal is to
be able to provide, by 2025, reliable forecasts up to two weeks in advance."

[https://www.zdnet.com/article/europes-big-weather-
supercompu...](https://www.zdnet.com/article/europes-big-weather-
supercomputer-data-center-is-about-to-leave-uk/)

As far as I know other than this model upgrade, there are no major investment
being made in American weather forecasting.

~~~
ekianjo
How far from that goal are they at the moment?

~~~
foepys
Judging by last week's forecasts for my region: pretty far off. Sudden rain in
the regions of 10mm/m² and a lot of clouds when sunny weather was predicted.
The exact opposite happened as well. Temperatures were off by 5-8°C, too.

All of this for same day forecasts, not even 2 days in advance.

~~~
ekianjo
rain is pretty hard to predict though. I guess what matters the most is if you
can reliably predict major events like heavy storms rather than if you will
get a bit wet later today.

------
esaym
Somewhat related, what are some good weather sites for storm monitoring? I've
been using ventusky[0] for rain forecasts and mrms[1] for storm and hail
conditions. Is there anything better?

[0][https://www.ventusky.com/](https://www.ventusky.com/)

[1][https://mrms.nssl.noaa.gov/qvs/product_viewer/](https://mrms.nssl.noaa.gov/qvs/product_viewer/)

~~~
timdorr
I really like [https://www.windy.com/](https://www.windy.com/)

~~~
blackRust
Windy.com allows you to compare models, which varies based on the region you
have zoomed in to.

This is really handy as national models can be much better for short-term and
higher resolution predictions.

It has the widest range of models freely available that I'm aware of,
including the commercial ECMWF model.

------
davidw
I just wish we had a weather radar here in central Oregon

[https://cliffmass.blogspot.com/2014/11/the-other-radar-
gap-e...](https://cliffmass.blogspot.com/2014/11/the-other-radar-gap-eastern-
slopes-of.html)

~~~
toomuchtodo
How interested are you in getting this done? Below is a link to a phased array
demonstrator (SPY-1A) that was dismantled and replaced with a newer version in
2016. Might find out where SPY-1A is sitting (the phased array may have been
returned to the US Navy), and since it'll perform both weather and aircraft
surveillance, might be easier to sell to stakeholders for the coverage gap.

[https://www.nssl.noaa.gov/tools/radar/mpar/](https://www.nssl.noaa.gov/tools/radar/mpar/)

Alternatively, Roberts Field appears to be a major commercial air hub in
central Oregon. You might argue from a safety perspective to your
Congressional representatives (perhaps in concert with local air carriers and
AOPA) that the airport needs a TDWR station (cost will be ~$4MM-8MM), which
could also provide NOAA with the necessary weather surveillance data.
Thunderstorms aren’t common on the West Coast though, hence the lack of TDWR
stations in West Coast states. If you pursue this route, you'd want to get
funds for this into some sort of federal transportation bill, as part of
enhancing the safety of the air transportation system.

[https://en.wikipedia.org/wiki/Terminal_Doppler_Weather_Radar](https://en.wikipedia.org/wiki/Terminal_Doppler_Weather_Radar)

~~~
davidw
I've mentioned it to our representative, Greg Walden, in the past, but he
doesn't seem interested, which is a pity, because this is the kind of non-
partisan stuff that they ought to be getting done for their constituents.

------
justinph
Some meteorologists are not in love with the new model. My local
forecaster/weather blogger suggests that the v3 model tends to overestimate
cold snaps and move storms too fast in the mid-latitudes:
[https://blogs.mprnews.org/updraft/2019/06/milder-with-
spotty...](https://blogs.mprnews.org/updraft/2019/06/milder-with-spotty-
showers-friday-is-noaas-gfs-upgrade-any-better/)

------
plantain
And it's extremely dubious whether there is actually any improvement...
[https://cliffmass.blogspot.com/2019/04/us-numerical-
weather-...](https://cliffmass.blogspot.com/2019/04/us-numerical-weather-
prediction-darkest.html)

------
steve19
I was researching weather prediction not long ago. From my nieve perspective
it seems that dispute all the increased gpu computational power and advances
in machine learning, there have not been any great advances in weather
prediction. Is this true?

Edit: Downvotes for simply asking a question. sigh.

~~~
dls2016
I worked as a forecaster for a bit but never made it to the research world
(studied theoretical pde instead of computational)... however at the time
_huge_ gains had been made in data assimilation. One fact that has stuck with
me was that ~1/3 of the computation time for the UK Met global model run was
consumed by data assimilation. I don't remember statistics anymore but data
assimilation schemes were a big driver of improved forecast skill.

I also recall the ECMWF had surprisingly accurate long range forecasts based
on ensembles. It could predict 500mb heights out two weeks, no sweat.

Re: your comments... My guess is that a gpu isn't suited for use in an
operational model due to data access patterns (and possibly not even helpful
with the solver). But again, I'm not a computational pde guy. Also, perhaps
machine learning would be useful but that would be post-processing or perhaps
parameterizing sub-grid phenomenon. There's already a process called model
output statistics (MOS) for adjusting raw fields from a weather model.

~~~
labster
The physics is pretty well known at this point, and there's only so much you
can gain by increasing from second to third order approximation. The errors in
the initial conditions are just larger. Most of the action has been on data
assimilation and better parameterizations because of that.

I've been out of the field for ten years now, but it's really nice to see
improvements to the core physics to this degree.

I'm still skeptical of your supposed two week 500 heights forecast from the
ECMWF model. I live near the western Pacific (i.e. the data hole) and it's
really easy to find crazy model solutions after 7 days. And I'm pretty sure
you weren't looking at the Southern Hemisphere.

~~~
dls2016
> I'm still skeptical of your supposed two week 500 heights forecast from the
> ECMWF model.

You're probably right to be skeptical, for the record I was only a forecaster
for a short period of time over ten years ago... didn't even serve my full
four year commitment as I volunteered to get out under the Air Force "force
shaping" at the time. I was stationed near Rammstein and we created forecasts
for Europe. I was referring to the ECMWF _ensemble_ products, specifically.

------
Tempest1981
A while back, I found a NOAA search that showed historic chance of rain by
month:

[https://www.wrh.noaa.gov/images/mtr/sjc_pcpn_prob.gif](https://www.wrh.noaa.gov/images/mtr/sjc_pcpn_prob.gif)

Since then, I spent hours trying to find this page again, unsuccessfully. All
I have is this URL for San Jose. (Replacing "sjc" with other airport codes
doesn't always work, since "mtr" is a region code?)

Anyone know where this precipitation data lives?

~~~
akie
I made this: [https://www.wmo.int/cpdb/united-states-of-
america](https://www.wmo.int/cpdb/united-states-of-america) \- click on
“climate normals”

~~~
dymk
Clicking on the "SEATTLE" link,

    
    
        Missing Controller
        Error: Climate.DashboardController could not be found.

~~~
akie
That sucks, I’m sorry. I made this about 5 years ago and am no longer involved
in the project.

~~~
Tempest1981
Thanks for the reply!

Was it accessing/presenting raw NOAA data? Or a different source? I notice the
data is thru 1990; different from NOAA.

Is the php source available for reverse engineering?

~~~
akie
The data comes from weather stations all across the US, which I assume are
managed or operated by NOAA. This project is at the World Meteorological
Organization, an international organization of weather organizations - of
which the NOAA is a member. Presumably this means it's official NOAA data.

With some minor effort you could extract the raw data, in JSON form, from
[https://www.wmo.int/cpdb/climate/climate_normal/per_country/...](https://www.wmo.int/cpdb/climate/climate_normal/per_country/USA/source:dashboard)

If I remember correctly the data came from a CD-ROM with historical data. When
we put it online the data was already 20 to 25 years old. It was nevertheless
the most recent data that we had available, I don't remember the reason why we
didn't have anything more recent. The PHP source is (or was) a CakePHP
application, and honestly isn't that interesting. There was not more data in
the PHP application than what is presented here.

EDIT: It was already a messy application when I got there, I cleaned it up as
well as I could, but after I left it seems to have gone downhill again. Ah
well. Not my problem anymore.

~~~
Tempest1981
Thanks again -- interesting to learn about this space! I wasn't aware of WMO.

------
neuronexmachina
This site has a technical overview of some of the models and algorithms used
by the new FV3 system:
[https://www.gfdl.noaa.gov/fv3/](https://www.gfdl.noaa.gov/fv3/)

------
westurner
> _Working with other scientists, Lin developed a model to represent how
> flowing air carries these substances. The new model divided the atmosphere
> into cells or boxes and used computer code based on the laws of physics to
> simulate how air and chemical substances move through each cell and around
> the globe._

> _The model paid close attention to conserving energy, mass and momentum in
> the atmosphere in each box. This precision resulted in dramatic improvements
> in the accuracy and realism of the atmospheric chemistry._

Global Forecast System > Future
[https://en.wikipedia.org/wiki/Global_Forecast_System#Future](https://en.wikipedia.org/wiki/Global_Forecast_System#Future)

------
sebnukem2
Wouldn't this kind of problem be a perfect match for machine learning? It
would have a huge dataset to learn from. Why isn't it happening or what
prevents AI tech from forecasting the weather?

~~~
stankypickle
It is because there is an understanding, from first principles, of the
dynamics that drive weather (e.g. conservation of mass, momentum and energy).
The current models are build upon these principles to make predictions, and
conform to expectations of how physics operates. The method that these models
are based on (finite volume) is efficient and adaptable if modifications need
to be made.

Using AI and ML to make predictions about weather will likely not account for
the conservation principles and might lead to ridiculous results (in some
sense). Creating an accurate AI/ML model of a complex and chaotic system might
lead to wrong predictions under extreme circumstances (e.g. predicting the
weather >5 days out for an extreme hurricane) or under conditions where some
implicit assumption has changed. One can at-least attempt to grapple with
these issues when using finite volume. Under AI/ML you just have to hope your
model is properly trained.

------
tumetab1
I like the cautious approach to this stuff

> The retiring version of the model will no longer be used in operations but
> will continue to run in parallel through September 2019 to provide model
> users with data access and additional time to compare performance.

------
microcolonel
This is huge. Good work to the folks at NOAA and the American taxpayer for
footing the bill.

~~~
Buttons840
I'm sure it wasn't the biggest part of our "bill". :)

------
koboll
Man, that sidebar is one hell of an aggravating UI element.

------
yufeng66
what kind of hardware is the GFS model run on?

------
wavesplash
As far as I understand, these are still hand designed algorithms using a tiny
fraction of possible weather data. Impressive for old school methods. Would be
even more awesome to see how far ML could take the state of the art.

~~~
elil17
Weather data is a system where we have a really good understanding of the
underlying physics but can’t do enough computation to simulate them in a way
that’s detailed enough to make truly accurate predictions.

Machine learning is all about finding an unknown function that underlies known
data. This is sort of the opposite issue: we know the underlying function but
can’t compute it.

That said, ML models are being applied to situations where the whether data
doesn’t translate directly into known physical quantities, like satellite
images (see [https://developmentseed.org/projects/hurricane-
intensity/](https://developmentseed.org/projects/hurricane-intensity/))

~~~
nestorD
You can also use ML to learn a metamodel : a model trained on an accurate (but
too costly to be run in real time) simulation.

~~~
SubiculumCode
Almost like super resolution techniques?

~~~
namibj
Pretty much. Maybe OpenAI will try to throw transformers at it.

