Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Get 70 Years of Historical Weather Data via API (oikolab.com)
24 points by a_square_peg 3 months ago | hide | past | favorite | 7 comments

There was a post last week about a weather API, so I thought I'd share mine as we had some major updates.

This is a historical reanalysis data (ERA5 from ECMWF) post-processed for fast time-series access. The primary dataset is a gridded (0.25 x 0.25 lat/lon) dataset covering the entire world so the data can be extracted for any arbitrary location. Later this year, a higher resolution dataset (0.1 x 0.1 lat/lon, about 9km x 9km) will also be added.

I've been working on this because I believe having access to historical weather is important for us to understand how climate is changing where we live. I'm not a programmer by training so it's a little rough around the edges but I believe it's the largest repository of weather data outside the primary sources, with 50+ weather parameters covering 70+ years with hourly resolution. It should be about 10~100x faster than getting the data directly from the source, and without having to deal with NetCDF/GRIB formats.

All in all, it's about 100~200TB of data that's been post-processed/chunked to Zarr format, with about 1TB/day on-going. Current users are mostly analysts who occasionally need large volume of data and don't really have time to go to the primary sources. We've just rolled out a feature to request multiple locations also request regional data (returns the data in NetCDF format) although most users seem to be happy with querying for single locations at a time.

Any comments would be greatly appreciated.

The post from last week is presumably about https://open-meteo.com ¹, for others(like me) who want to check that one out too.

Colour me genuinely shocked that this much weather data is available to the public, fantastic project!

¹ https://news.ycombinator.com/item?id=28499910

Thanks! Yes, that's the post I was referring to. :)

Most weather data what we encounter from apps are actually public data - they're just cumbersome to work with. There is so much there but not as widely used as they should be.

Do you have forecasting data as well? Could we use that to figure out how accurate forecasting is from various agencies?

If you mean historical forecast data, we do archive it but only for seven days at the time. There are quite a bit of information comparing different model performance that you can look up (i.e. https://apps.ecmwf.int/wmolcdnv/) on the other hand. In general, the European model (from ECMWF) is considered to ahead of other agencies' models in terms of performance.

Sounds a bit like what Automobilista 2 is using for their real historical weather data.

From the sounds of it, it does seem like they might be using the same dataset. :)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact