
Ask HN: What are some good sources of climate data to analyze? - rpeden
I&#x27;ve noticed that articles mentioning climate and climate change that make it to the front page end up being flagged.<p>I think this is <i>probably</i> fair, because discussions about such articles invariably end up being political and don&#x27;t end up contributing very positively to HN.<p>I certainly <i>have</i> thoughts and opinions on this topic, but upon reflection I realize they&#x27;re not especially well informed. They&#x27;re mainly based on what I&#x27;ve read in popular media, and I&#x27;m hoping I can do better than that.<p>So I&#x27;d like to approach it from another direction. If I want to get some raw climate data for myself, and analyze it using R or Python, what would be a good starting point?<p>Are there specific data sets that I should focus on? Are there any well known papers that I should read?
======
titojankowski
"No One Gives a Fuck About Climate Change" made the front page of Hacker News
in June. This inspired a team to created the first Open API for Climate Data.
It's a repository of carbon dioxide data as measured on Mauna Loa by NOAA.

The prototype is up and running with data from 1958 - 2017!
[http://api.carbondoomsday.com](http://api.carbondoomsday.com) prototype
frontend: [http://carbondoomsday.com](http://carbondoomsday.com).

The project is open source on GitHub: [https://github.com/giving-a-fuck-about-
climate-change](https://github.com/giving-a-fuck-about-climate-change)

The original blog post that inspired the project: No One Gives a Fuck About
Climate Change ([http://titojankowski.com/no-one-gives-a-fck-about-climate-
ch...](http://titojankowski.com/no-one-gives-a-fck-about-climate-change/))

Would love to get your perspective and feedback! Maybe you want to follow the
project on Github?

~~~
titojankowski
Also, continue this discussion with other like-minded hackers!

Join the "Giving Fucks about Climate Change" Google Group:
[https://groups.google.com/d/forum/giving-fucks-about-
climate...](https://groups.google.com/d/forum/giving-fucks-about-climate-
change)

------
sword_smith
I have a Master's degree in physics from the University of Copenhagen and I
have taught together with one of the leading climate scientists from that
university Bo Møllesøe Vinther. He claimed that the best dataset on global
average temperatures is satellite based and is the University of Alabama in
Huntington dataset (UAH) which goes back to 1979.

You can find it here:
[https://www.nsstc.uah.edu/climate/](https://www.nsstc.uah.edu/climate/)

Direct link for monthly temperature averages:
[http://www.nsstc.uah.edu/data/msu/v6.0beta/tlt/uahncdc_lt_6....](http://www.nsstc.uah.edu/data/msu/v6.0beta/tlt/uahncdc_lt_6.0beta5.txt)

~~~
prewett
Wouldn't you need data going back for thousands of years? 40 years of data
won't tell you anything about the cause of the temperatures in this data set.
You'll need to eliminate the possibility that we are in a Little Heat Wave
cycle; if you got forty years of temperature data during the beginning of the
Little Ice Age, you won't have a representative sample, for instance.

~~~
mapster
Its a unique dataset. it does not portend to be a multi-source long history
dataset. you raise a question re: long term trends which is another topic.

------
lisper
[http://berkeleyearth.org](http://berkeleyearth.org)

Not just data, but analysis tools as well. Run by a Berkeley physicist who was
originally a climate change skeptic and changed his mind after doing this
analysis.

~~~
nl
I looked into this a while back and came to the conclusion that this is the
best source.

To combat the "oh no they are 'corrected'" claims, they host the uncorrected,
completely raw files too[1]. Then you can have the enjoyable task of trying to
work out how to correct for sensors moving, cities growing, etc. Look for the
"raw" datasets, as opposed to the Breakpoint Adjusted files.

[1] [http://berkeleyearth.org/data/](http://berkeleyearth.org/data/)

------
mindcrime
It's slightly tangential, but you might find this project, and the surrounding
community, to be of interest.

[https://climate.apache.org/](https://climate.apache.org/)

Also, in addition to historical data that's already collected and available
online, an interesting side note is that it's possible to directly receive
some satellite data yourself using a relatively inexpensive SDR (Software
Defined Radio) device. This article deals specifically with receiving images
from NOAA satellites, but there are probably other signals floating around out
there that would be useful to you.

[https://www.rtl-sdr.com/rtl-sdr-tutorial-receiving-noaa-
weat...](https://www.rtl-sdr.com/rtl-sdr-tutorial-receiving-noaa-weather-
satellite-images/)

~~~
mturmon
The Apache project you link is not tangential at all -- it's spot on! See the
examples at:

[https://github.com/apache/climate/tree/master/examples](https://github.com/apache/climate/tree/master/examples)

The people advising the project have significant publications and research
experience. I believe some of their examples are replicating peer-reviewed
work with that software.

~~~
mindcrime
Fair enough. I just meant tangential in the sense that the OP was asking about
data, not software specifically. But yeah, it is very much about working with
climate models. Unfortunately I can't claim any experience working with the
project myself... I'm only familiar with it in passing.

~~~
mturmon
The authors inhabit the same floor of the lab where I work, so I may have a
touch of bias. ;-) I think some data sources are linked with the examples, but
I have not checked myself.

------
godshatter
I follow some websites casually, usually daily, to get an idea of what the
numbers are showing me. It's just a ritual I go through in the mornings before
going to work.

When I want to see where we are temperature-wise, I go to
[http://www.drroyspencer.com/latest-global-
temperatures/](http://www.drroyspencer.com/latest-global-temperatures/) for
the latest satellite monthly temperatures.

The What's Up with That? website (which is strongly biased towards the
"denier" side of the debate) has a great collection of graphs related to
arctic and antarctic sea ice: [http://wattsupwiththat.com/reference-pages/sea-
ice-page/](http://wattsupwiththat.com/reference-pages/sea-ice-page/)

I go to [http://spaceweather.com/](http://spaceweather.com/) to watch the
sunspot numbers (just for curiosity sake),
[http://sealevel.colorado.edu/](http://sealevel.colorado.edu/) for sea level
information, and
[http://www.cpc.noaa.gov/products/analysis_monitoring/ensostu...](http://www.cpc.noaa.gov/products/analysis_monitoring/ensostuff/ensoyears.shtml)
to see if we're in an El Nino or a La Nina (or somewhere in the middle).

------
matt4077
Google's Bigquery has the following huge set of weather measurements, and you
can use both the initial trial credit, as well as a substantial free quota
every month to run analysis on it right within Bigquery:
[https://bigquery.cloud.google.com/dataset/bigquery-public-
da...](https://bigquery.cloud.google.com/dataset/bigquery-public-
data:noaa_gsod)

Quote: "This public dataset was created by the National Oceanic and
Atmospheric Administration (NOAA) and includes global data obtained from the
USAF Climatology Center. This dataset covers GSOD data between 1929 and
present, collected from over 9000 stations.

Dataset Source: NOAA

Category: Weather

Use: This dataset is publicly available for anyone to use under the following
terms provided by the Dataset Source — [http://www.data.gov/privacy-
policy#data_policy](http://www.data.gov/privacy-policy#data_policy) — and is
provided "AS IS" without any warranty, express or implied, from Google. Google
disclaims all liability for any damages, direct or indirect, resulting from
the use of the dataset.

Update Frequency: daily"

------
sasheets
Here's a compilation of links to get you started:
[https://tamino.wordpress.com/climate-data-
links/](https://tamino.wordpress.com/climate-data-links/)

Climate science is a big subject and you should start by understanding the
basics of the subject. Spencer Weart's history of climate science is a good
place to start:
[https://history.aip.org/climate/index.htm](https://history.aip.org/climate/index.htm)

Also, I see many people with strong statistics/engineering backgrounds
analyzing climate data and coming to wrong conclusions because they never
bothered to learn the physics. Don't be those people.

~~~
rpeden
Thanks for the suggestions.

And thanks for the heads up regarding coming to wrong incorrect conclusions.
That's important and useful advice.

------
pjc50
The IPCC full reports are probably the place to start.

However, before embarking on your own analysis, you should consider what
questions you're trying to answer and how to avoid misleading yourself. There
are two very important and extremely complex areas: proxies and forcing.

Proxies are the use of data other than temperature measurements to infer
temperature, such as tree rings and ice cores.

Forcing is the second-order effects of more CO2 such as causing more water
evaporation - whic is itself a greenhouse gas.

~~~
pale-hands
I think you've identified the two most important topics. A nitpick, though: I
disagree with your definition of forcing, it's the same as my definition of
feedback.

A forcing is anything that moves the temperature away from the equilibrium
between radiative losses and insolation, whether higher or lower. Additional
CO2 is itself a forcing.

------
padraic7a
I believe that the dataset published by ICARUS in Maynooth University might be
more complete and up to date than some of the others mentioned here. It is a
combination of a large number of datasets, is global in scope and runs from
1900 to 2012.

[http://maynoothdigitalcollections.ie/catalog/bg257f05g](http://maynoothdigitalcollections.ie/catalog/bg257f05g)

It is the basis for the two following articles:

[http://onlinelibrary.wiley.com/doi/10.1002/2015JD024583/abst...](http://onlinelibrary.wiley.com/doi/10.1002/2015JD024583/abstract)

[http://onlinelibrary.wiley.com/doi/10.1002/2015JD024584/abst...](http://onlinelibrary.wiley.com/doi/10.1002/2015JD024584/abstract)

IIRC the largest files might not download correctly from this repository
linked to above. If that it the case you should contact the researcher Peter
Thorne:

[https://www.maynoothuniversity.ie/people/peter-
thorne](https://www.maynoothuniversity.ie/people/peter-thorne)

------
newforice
I like this data:

[https://gcmd.nasa.gov/KeywordSearch/Metadata.do?Portal=amd&K...](https://gcmd.nasa.gov/KeywordSearch/Metadata.do?Portal=amd&KeywordPath=%5BKeyword%3D%26%23039%3BCH4%26%23039%3B%5D&OrigMetadataNode=GCMD&EntryId=EARTH_LAND_NGDC_PALEO_VOSTOK1&MetadataView=Full&MetadataType=0&lbnode=mdlb5)

And from there:

[https://www.ncdc.noaa.gov/data-access/paleoclimatology-
data/...](https://www.ncdc.noaa.gov/data-access/paleoclimatology-
data/datasets/ice-core)

And a Wiki article discussing it:

[https://en.wikipedia.org/wiki/File_talk:Vostok_Petit_data.sv...](https://en.wikipedia.org/wiki/File_talk:Vostok_Petit_data.svg)

Where this chart is interesting:

[https://en.wikipedia.org/wiki/File_talk:Vostok_Petit_data.sv...](https://en.wikipedia.org/wiki/File_talk:Vostok_Petit_data.svg#/media/File:Vostok_Petit_data.svg)

------
stared
Right now I am using this data:
[https://data.giss.nasa.gov/gistemp/station_data/](https://data.giss.nasa.gov/gistemp/station_data/)

While my motivation is to demonstrate matrix decomposition (see:
[https://twitter.com/pmigdal/status/902563470125789188](https://twitter.com/pmigdal/status/902563470125789188)),
I wanted to use a credible source. See also a few threads on Open Data Stack
Exchange (e.g. [https://opendata.stackexchange.com/questions/1546/how-
might-...](https://opendata.stackexchange.com/questions/1546/how-might-i-go-
about-visualising-historical-temperature-csv-data)).

~~~
mindcrime
Since you mentioned the Open Data Stack Exchange, it might also be worth
pointing out a couple of sub-reddits that might also be of use to the OP (or
anybody):

[https://www.reddit.com/r/datasets/](https://www.reddit.com/r/datasets/)

[https://www.reddit.com/r/opendata/](https://www.reddit.com/r/opendata/)

------
loudmax
If you're looking for raw data, you can find measurements from NASA's earth
observation missions here:
[https://earthdata.nasa.gov/](https://earthdata.nasa.gov/)

NASA's Worldview application presents the data as overlays on maps of the
Earth and it lets you scroll backward in time from the start of the missions
(around 2000 or so) up to near real time (a few hours old). If you want to
make sense of the scientific data, you'll still need to put some work into
understanding what the instruments are measuring:
[https://worldview.earthdata.nasa.gov/](https://worldview.earthdata.nasa.gov/)

------
lordCarbonFiber
The National Centers for Environmental Information are probably a good place
to start. Collection of links to data can be found here:
[https://www.ncdc.noaa.gov/data-access/quick-
links](https://www.ncdc.noaa.gov/data-access/quick-links). Alternatively there
are some collected sets available here (which may be a little cleaner/ have
some metatdata annotation):
[https://public.enigma.com/search/climate](https://public.enigma.com/search/climate).

------
me_bx
Data from European Union's satellites (Copernicus Program) is available for
free.

Copernicus Climate Change Service is under development, some data is already
available:

[http://climate.copernicus.eu/](http://climate.copernicus.eu/)

Some other Copernicus services may also contain relevant information related
to climate change: atmosphere, land, marine..

[http://www.copernicus.eu/main/services](http://www.copernicus.eu/main/services)

------
devty
Oregon University has precip/temperature data for CONUS freely available!

[http://prism.oregonstate.edu/](http://prism.oregonstate.edu/)

------
marcus_holmes
[http://www.woodfortrees.org/](http://www.woodfortrees.org/) \- excellent
stuff here

There's an interesting wrinkle going on in Australia at the moment. The Bureau
of Meteorology routinely adjusts the "raw" data (for entirely sensible
statistical reasons). However, the net effect of these adjustments is
apparently to amplify the warming trend. There's controversy even in the raw
data - any analysis produced from it can be criticised. I don't think going to
the source is going to resolve the politics.

------
kadal
[https://www.climatedataguide.ucar.edu/](https://www.climatedataguide.ucar.edu/)

Quoting from the website "Search and access 194 data sets covering the
Atmosphere, Ocean, Land and more. Explore climate indices, reanalyses and
satellite data and understand their application to climate model metrics. This
is the only data portal that combines data discovery, metadata, figures and
world-class expertise on the strengths, limitations and applications of
climate data."

------
rpeden
There's a reply that's dead, but I'd like to address it. It asked:

"We have 25 years or so invested in the work. Why should I make the data
available to you, when your aim is to try and find something wrong with it?"

That's a fair question, but my aim definitely isn't to try to find something
wrong with the data. My aim is only to explore the data and find out what's
right and what's wrong about my current understanding of the situation. I
don't plan to blog about or even discuss the work I do.

~~~
godshatter
Also, isn't publishing something in order for others to "find something wrong
with it" a core component of the scientific method?

~~~
airza
sure, but statistically someone who knows how to correctly interpret this data
probably already has the academic communication channels to solicit it
directly, so someone randomly asking for it online probably has an axe to
grind

~~~
rpeden
Right, which is why I thought it was a fair point for the original poster of
the message to bring up. And it seems like there are lots of people with an
axe to grind on this topic.

I'm just looking to learn more for myself, and I decided to ask HN because I
was having trouble deciding where to start. I realize it's tough to look at
data on a complex topic without a thorough academic background in it. But I
figure I'll never learn more if I don't at least get started.

~~~
drewmol
Thanks for explaining this, I would have thought the same: I'm a novice,
looking for good climate data sets to analyze... asking HN could be a good
place to start. Based on the feedback/links provided so far, I think it is!

------
c0nducktr
[https://data.giss.nasa.gov/](https://data.giss.nasa.gov/)

------
mhh__
I've never used it myself, but Richard Muller's Berkely Earth project publish
their datasets online.

------
geekamongus
[http://www.ncdc.noaa.gov](http://www.ncdc.noaa.gov)

