Hacker News new | comments | show | ask | jobs | submit login
Global warming? Check for yourself (nfshost.com)
107 points by KazimirMajorinc on Mar 4, 2010 | hide | past | web | favorite | 54 comments

Data from any one weather station can neither prove nor disprove – well – I would guess pretty much anything climate related.

The author doesn’t claim any such thing and the article is actually a quite nice read. The headline here stretches the hyperbole of the original headline quite a bit too much, though.


I liked the article. He used an emotionally-charged topic as a hook, didn't take sides, and instead taught me something about a cool language.

Great HN post.

After reading about the climatology controversies, and not knowing who to believe, I did my own investigation of the climate data using the GHCN data, which was the most comprehensive publicly available set I could find.


This is written in C++, and you can plot the data in numerous different ways, including individual weather stations if you wish.

Very interesting work.

High "average maximum temperature" -- what kind of statistic is that? Why should he use that instead of the average temperature? Randomly picking a 1-day outlier from each year is useless.

Meanwhile, from the adults who use crazy concepts like "sattelite data" and "arithmetic mean": http://af.reuters.com/article/topNews/idAFJOE61O02C20100225

Last month was the hottest january on record.

EDIT: Not to take anything away from the guy -- cool code, cool app. But to those who are trumpeting this -- maybe you should take a look at that "hottest january on record" article and take a look at the methods that were used to arrive at that conclusion.

Funny.. a bunch of people upvote what is clearly labelled by its own author as amateur and naive number extracting, then downvote a link to rigorous statistical analysis saying that last month was the hottest on record.

It's not about the methods, it's about the conclusions? If they agree with me, the methods are ok, if they don't, they're a biased scientist conspiracy?

What the hell.. if someone was linking to an amateur analysis saying that log n was more expensive than n^2, would we be getting all of this philosophy about how everyone's perspective is valuable?

I think the voting indicates:

  data, code => cool => upvote

  statement by some organization with no data, no analysis => boring => downvote

And given the recent breaches of trust amongst the climatologists, this is exactly what is appropriate.

Building up open and transparent datasets and models that anyone can work with (or lacking the domain expertise and technical skill, at least be comfortable that somebody can watch over the scientists' shoulders) is exactly what's needed.

Another article that trumpets talking points without mentioning all of the caveats is NOT needed. And the fact that it yet again implies a need for action, while entirely ignoring any assessment the costs of those effects, together with a weighing against the costs of remediation, puts the Reuters article squarely into the fearmongering side of the equation, 180 degrees off of rational debate.

You imply that by being a scientist, you are untrustworthy, and that being a layperson without expertise makes you trustworthy. This idea seems quite backwards. A lay person can make any sort of corrupt claim without any repercussions. Meanwhile, if a scientist falsifies data and gets caught, their entire career can be lost.

Not to mention that scientists generally have no vested monetary interest in the results, despite what people here seem to think. Grant money can only be used for research, not for personal gain. By going into the sciences, instead of industry, you are willingly taking a massive loss in pay. To claim that these people are corrupt is both an insult and naive. Like any area, there will be a few bad apples.

You imply that by being a scientist, you are untrustworthy, and that being a layperson without expertise makes you trustworthy.

Not at all. Being open with your data and methods make you (more) trustworthy. Stonewalling and avoiding disclosure makes you untrustworthy. It has nothing to do with your role, but with how you play it.

if a scientist falsifies data and gets caught, their entire career can be lost.

Which of the East Anglia scientists lost their careers, or even their job? (I understand that there's no real evidence that they falsified data, just some implication. But they were badly unethical.)

Not to mention that scientists generally have no vested monetary interest in the results, despite what people here seem to think. Grant money can only be used for research, not for personal gain.

The continuation of their research is personal gain. A conclusion of "nothing interesting going on here" means they've got to find a new area of research, find new grant sources, etc. If they happen to find evidence of something scary, that gives them job security. I have no proof that they followed this line of thought, but it is definitely a conflict of interest.

That said, virtually anyone involved in the field is there because they've got some passion for it, so probably everyone has some degree of conflict.

The fact that someone made a choice that leads to a lower pay scale does not make them above corruption. You may as well ask me to believe that the NEA's sole concern is teaching kids, rather than preserving their members' benefits.

"And given the recent breaches of trust amongst the climatologists"

Or maybe you have fallen for the propaganda of the denialists? Anyway, I think usually the actual scientific papers come with data references (otherwise they are indeed useless), but there is also a place for articles that normal people can understand. Not everybody wants or can code up their own verification program for every scientific article they come across.

Denialist? You sound like a member of the Spanish Inquisition. Are we talking science (where constant questioning is [or is that just silly me?] was always supposed to be intrinsic to the investigative process) or religion? The latter, it seems, as far as you're concerned.

Has it dawned on you that use of the word is a kind of own goal? "Hey folks, we believe in the true religion and must rout those non-believers!"

Well I was responding to "breaches of trust", which seems like propaganda given that all that can be pointed to is some minor errors that are bound to happen in a large scale undertaking. I don't think calling those "breaches of trust" is very scientific. Maybe the one about the Himalya glaciers could be called like that, but I don't think it was the actual scientists being sloppy in that case.

I was only using the word "denialist" to shorten the sentence - what would have been an appropriate word, then?

all that can be pointed to is some minor errors that are bound to happen in a large scale undertaking

Ahem. Conspiring to thwart freedom of information laws is something that is bound to happen in large scale undertakings? Either you're propagandizing yourself, or you're someone that I really don't want working for my company.

The glacier thing was characterized as a "typographical error", which itself is clearly intended to minimize the ethical breach. A "typographical error" is one related to the typography -- maybe transposing digits, or accidently chopping off the bottom of a page or something. It clearly does not apply to errors of judgment, including non-reviewed sources as if they're factual.

None of this refutes AGW, which is actually my point. It makes us all wary. And the cure for that is openness and transparency. A battle of ad hominem attacks doesn't do anything for either side of the debate. But in the long run, building a case that we can all trust because we watched it being erected is to the benefit of all -- tree huggers and denialists alike.

"Conspiring to thwart freedom of information laws"

When did that happen? If you are referring to the leaked emails, I don't think the case is so clear. Unless you want to believe it to be so.

The glaciers: don't remember the exact details, but wasn't it some relatively informal publication where the boss of the organization slipped something in without verifying with his scientists? Sounds more like sloppiness - although they should have uncovered the error sooner. But who knows, they might have had other things on their mind. I don't know how many scientists actually think about Himalayan glaciers on a daily basis (do you?). Except for the ones living in the Himalayan, one of was asked over the phone and apparently misinterpreted.

In any case, I don't think that error was made in an actual scientific paper, it was more part of general propaganda and politics. Not that I like it at all, but I don't see how it discredits the science as a whole.

In fact, I personally think dwelling on minor details and trying to blow them up to be elephants is a major indicator of revisionism.

I linked this downthread, you should give it a look : http://www.xkcd.com/675/

Data are out there and you can do little stuff with it if you want and if you have a precise question in mind.

This is what I did some time ago: http://www.gilestro.tk/2009/lots-of-smoke-hardly-any-gun-do-...

A recurring argument among denialist is that data are someone tricked to show things that aren't true. Some of those claims can actually be tested, but one needs to know what they are talking about.

Maybe you didn't finish the article: he's performing an incorrect analysis on purpose!

"I've made a number of small mistakes and inappropriate design decisions in this post (some deliberate, or at least, some I'm aware of, others are accidental). But, given the published and freely downloadable weather data, the code listed here, and - of course - the excellent free and open source newLISP language, it should be possible for anyone to retrace my steps, find my mistakes, and present a more credible or compelling view of the same dataset."

Little disappointed so many commenters seemed to miss that. I thought that paragraph was the entire lesson, with the preceding material there so this point would sink in.

Since I'm interested in information design (especially right now) I was quite disappointed with the graph he ultimately produced. I'm hoping someone with a bit more time than me takes up his challenge and goes back to correct some of his mistakes (in both analysis and storytelling). I'd rather like to do it myself but I doubt that's going to happen anytime soon.

Ultimately, a compelling story about how we as humans require stories for understanding, and how important it is to verify the stories that we choose to believe.

I cheated and just asked him. :-D

If you're interested, his response is here:


Ha! Well, that explains things rather neatly then. If he was really building only up to that final paragraph, it would have been shaped differently. Instead it seems that was just one conclusion he drew out of a project started for different reasons. Thanks for sharing this.

Offtopic: that syntax highlighting upon mouseover is a great idea. Does the fact that I haven't seen this before mean I need to get a new editor?

Yes, it's fun. PLT Scheme editor supports similar "block style" parentheses highlighting, and it is useful for reading large data sets or machine generated code.


Armagh itself is a smallish town in Northern Ireland, less than 1000 miles south of the Arctic Circle although bathed in the warm currents of the Gulf stream.

I wonder how stable the Gulf stream is over time.

This graph actually carries a lot of intellectual weight for 2 reasons:

1) it is an apparently unbiased inquiry, which means a lot. The allegations against leading climate change scientists, whether they are valid or not, highlight an inescapable reality about the world of academic geologists (no matter how "inconvenient" of a truth it is): like many scientists, these people need to raise large amounts of funding for relatively boring projects (i.e. ice samples anyone?). But if they tie in their research to a hot (no pun), visible issue (as rendered by the media and politicians, regardless of its scientific merit), then BAM! It's a lot easier to get funding. Hence, climate scientists have a huge incentive to paint global warming as massive, urgent big problem, whether or not it actually is. Seeing data crunched by an amateur, who has the skills to process it, but not the incentives to skew it, means a lot.

2) Even though it only shows weather data from a single region, "global warming" would presumably be a global phenomenon that one could detect from basically any location (and if this is not the case, then the press needs to revise its doomsday scenarios about global warming necessarily flooding New York City and Shanghai, because if it's only occurring in some places, then we shouldn't use data gathered in Canada to predict catastrophe in East Asia.)

I'm not claiming to know the scientific truth about this issue one way or the other, but I do know that there are a lot of forces at play here besides science (funding incentives, politics, news media sensationalism, green energy business interests, etc.), and I think a return to the data is a great place to start unraveling this issue that is far more complex than Al Gore or his opposite Dick Cheney would have you believe.

That particular visualization wouldn't show an average global temperature increase until after the artic was quite dry.

As the author says, a picture is a story. His story is "at an extremely high level we have seasonal temperature fluctuation."

Your entire second point shows a complete misunderstanding of well, everything.

Item 2 is just wrong. Even something as (comparatively) well understood as an El Niño event causes wilding differing effects from region to region. Every time a scientist goes on TV to explain this stuff, they have to explain this very point: local measurements don't cut it, you have to look at and correlate very complicated global data sets. Apparently you didn't listen becuase of...

Item 1, which seems to me to be a completely unsubstantiated ad hominem. You could use that same argument to reject any science you don't want to hear.

You're incorrectly using the term "ad hominem." An ad hominem argument is when you try to invalidate an idea with an irrelevant personal attack. The problematic incentives facing climate change scientists are certainly relevant to their research. I'm not saying we should dismiss their data necessarily, but I am calling into question their motivations. Would we be skeptical of public health research on smoking funded by a tobacco company? I hope so. But just because they would have strong incentives to produce a particular outcome doesn't mean their science is wrong, but it could very well be biased. Same here.

The scientific truth of the matter is that last month was the hottest on record: http://af.reuters.com/article/topNews/idAFJOE61O02C20100225

There is room for scientific doubt about whether this is caused by CO2 -- it's a greenhouse gas but correlation != causation and all that.

But if you conclude that the earth isn't actually warming, you're doing it wrong -- there's no doubt about that.

1. You assume a (fairly) anonymous, amateur source based on a single data point is more unbiased than someone who has devoted most of their lives to collecting data? I totally agree with the movement to make scientific data more open, but I think you are really underestimating the motivations of most scientists.

2. Extrapolating from a single data set leads to huge biases in interpretation. There are too many uncontrolled variables to say anything meaningful beyond what the temperature in this city has averaged. I think the rate of change that people are worried about in overall temps would probably not even register on this graph clearly.

If you are going to criticize how science is done, you should probably learn more about how it is done.

And some basic statistics.

While the graph is nice and this was a great visualization example in newLisp, drawing conclusions from the graph alone is not really possible without some amount of statistical analysis (e.g., de-meaning or accounting for seasonality somehow).

And to the author's credit, the observations made are not conclusions in themselves.

This type of analysis and visualization is best done in Mathematica, which incidentally also supports both functional (a la Lisp) and declarative (a la Prolog) programming. It costs money, but it's the right tool for the job.

Nice chart. Looks like there is global warming once a year or so! Conspiracy!

I am not sure what the point of this is, but it's an interesting demo of newLisp... but just use R next time.

This is Hacker News.

What did you want to say?

HN is an open prairie where conspiracy theories roam wild, free and careless.

People here don't respect science and think they're smarter than they really are.

Honestly, I wasn't thinking about the 'truth' of the claims, I'm FAR from qualified to judge. But rather his approach is something that would be interesting to hackers. Was I wrong?

No, but it seems your comment added nothing to the discussion. (And it was short enough to be misunderstood.)

Touche. Mea Culpa.

What's the point of this?

All it shows is that someone without scientific training is incapable of drawing conclusions from data out of context; they can't "check for yourself". I tought the only people that don't know this are journalists, bloggers and politicians.

I was thinking about this particular type of ad hominem argument the other day and decided I loathe it. Imagine if people told Plato he couldn't write about philosophy because he didn't go to harvard and get a degree in philosophy. Or tell galileo that he was wrong because he didn't have a degree in astronomy.

Valid scientific investigations don't require scientific training. This person, who wrote some code and plotted some numbers on a graph may know more about math and programming and data visualization than many of those running weather stations, so it isn't productive to say this person has nothing to contribute because he's someone without scientific training.

You don't even know that statement is true. Regardless of its truth, the numbers, the code, and the result should speak for themselves. It doesn't matter whether the investigator is trained in science or not.

As a "trained" computer engineer, I am personally capable of taking a raw data set and plotting their numbers on a chart and drawing conclusions from them.

Of course there are additional steps one could take with this data to help the human mind comprehend it. I'll tell you one thing for sure, the chart looks nothing like this scary one http://en.wikipedia.org/wiki/File:Instrumental_Temperature_R...

Why? If you have scientific training, refute the facts, write another set of code, analyze the data and see what you get. Is it different? Can you replicate the results?

Or can you just say this person doesn't have any scientific training and write him off? Personally I think it's awesome that someone is taking the initiative to test a hypothesis. Really, there is nothing more scientific than that. It takes work and I commend this person.

I agree with you that excluding someone from a scientific debate because he has no academic credentials or even just not the right kind of academic credentials out of hand is wrong. Certainly wrong in this case, because the linked article doesn’t even make any grandiose claims.

Valid scientific investigation doesn’t require scientific training. That’s certainly correct. That doesn’t mean, though, that every investigation has the same weight. I have no expert knowledge. How am I supposed to know whether an amateur is brilliant or crazy? Academic credentials are more often than not a useful filter to figure just that out. Not perfect, but better than giving everything equal weight.

(Cue conspiracy theories about the academic establishment and generous East Anglia email quoting – NOW! :)

> Imagine if people told Plato he couldn't write about philosophy because he didn't go to harvard and get a degree in philosophy. Or tell galileo that he was wrong because he didn't have a degree in astronomy.

Very interestingly, your examples only add to my point. Plato didn't had a degree in philosophy because philosophy was just being "invented"; Socrates (his master, btw) had just brought rigor to the sophists blabering. And Galileo didn't get a degree in astronomy mostly because it didn't exist by then. He did however get one in mathematics and tought astronomy in a university before he made his observations to support Copernicus' model. You see, both men where very far from amateurs giving opinions in a complex field.

> Valid scientific investigations don't require scientific training.

Bulshit. Most of times it does. Particularly in complex matters. That's why the peer review process matters. Numbers don't "speak for themselves"; that's why we talk about "lies, damned lies and statistics".

As a pratical exercise, I'd suggest you, as a "trained computer engineer", to let your projects to be designed and implemented by a non-trained computer engineer. You'd know what I mean.

Seems to me an a guy who couldn't get hired as a physicist had a rather large effect on physics in 1905. The hiring process is a form of 'peer review'.

I really don't think you should take your understanding of Einstein's early career from Yahoo Serious movies.

from Wikipedia "After graduating, Einstein spent almost two frustrating years searching for a teaching post,"

Which is about 1/10th of what your post is implying. You might want to read something deeper than a wikipedia summary.

Peer review is neither necessary nor sufficient for science.

And Einstein would possibly have agreed with that to some extent, as evidenced by his reply in 1936 to the rejection of the only paper of his ever even subject to peer review, even then a new concept in fields unrelated to medicine. This has literally nothing to do with his inability to find work after he graduated university or the hiring process of science institutions in general. This emotional argument doesn't just not reflect Einstein on any level deeper than a Wikipedia summary of a bio, but lacks any historical perspective as well.

Einstein's 1905 works, specifically the Annus Mirabilis papers, lacked the formal review process we understand today, but were certainly reviewed by the two Nobel prize-winning physicists who selected them for publication in their journal. The formal review panel concept simply did not exist at the time outside of the medical fields, but that is not to say there was no stringent editorial control or gatekeeping in physics journals, and certainly not in Annalen der Physik. If anything, Einstein was subject to a far less fair and inclusive process.

I suspect you completely missed the point of the parent.

The "Check for yourself" part is only in the editorialized headline, implying that the READER of the article is capable of drawing his own conclusions. This is obviously dishonest, because you can't draw your own conclusions unless you do the research yourself (like the independent researcher who wrote the article did).

The No Spin Zone (O'Reilly) uses the "We present the facts, YOU DECIDE" mantra. So do most conspiracy theory movies: "We'll just show you MONEY IS DEBT, and AMERICA is due for RUIN. But is this bad? YOU DECIDE!"

And so on.

But for any scientific work this is unacceptable, because the way in which you present the results matter. Pick one scale on a graph and people will assume the problem is huge, pick a logarithmic scale and people will conclude there is no problem. It's the responsibility of the researcher to both present the facts and to explain what they mean.

The you-decide tactic is manipulative, and it doesn't belong in HN titles.

- I should note that I don't have any issue with the article itself. I liked it, and I commend independent research.

Nothing against amateur scientists. However, it says "At Armagh Observatory", so it is hardly global warming - it's probably Armagh warming? Haven't read all the details, so I can't say. But there seem to be other data sets out there that show rising temperature? So why should this one data point be more significant than others?

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact