

The Rank Amateur - jgrahamc
http://www.jgc.org/blog/2010/02/rank-amateur.html

======
randomwalker
Disclaimer: I'm a scientist, so maybe I'm biased. But then again, I'm often a
vocal critic of the system myself, so maybe not.

Science is actually admirably egalitarian. That's not one of the things I
think is wrong with it. The heavily politicized branches might not be
egalitarian; I have no experience with them. But in general, you don't need to
be a member of any professional organizations or have any other credentials to
submit your work to journals and conferences (which is how we transact our
business.) Many journals employ double-blind reviewing, in which case even
unintentional discrimination based on credentials is unlikely or impossible.

Nevertheless, scientists might often give the impression of not wanting to let
amateurs into the 'club'. Why is this? It is simply an issue of bandwidth.
Think about this: for each amateur, like the author, who had something to
contribute, how many do you think _thought_ they had something to contribute
but were mistaken? If you guessed a hundred thousand you'd be in the right
ballpark. I've seen it in many different areas.

For example, the number of people trying to submit "proofs" of P != NP, (or
worse, P = NP) is just ridiculous. Some are well-intentioned although
ignorant, and others are just cranks. Some fields of research attract more
amateur claims than others, but whichever field you look at, the amateurs
greatly outnumber scientists, and the vast majority of them are mistaken.

So what do we do? We use a simple filter. If you speak our language, we'll
listen to you. This is what _appears_ to the lay public as anti-amateur. But
it's not, really. All you have to do is to learn some simple definitions and
terminology set forth in a straightforward way in previous papers, to prove
that you've done your homework. Then write up what you have to say and we'll
be happy to give it a read. Sure, like every heuristic, it's not perfect;
sometimes there are false negatives. But there really isn't an alternative.
Without a filter, all we'd ever be doing is debunking crackpot theories.

I acknowledge that much of this probably doesn't apply to climate change
"science," which is a special case. But there's been a lot of criticism of
science in general, and I wanted to set the record straight. It's important to
distinguish between what's really broken and what appears broken, otherwise we
risk throwing the baby out with the bathwater.

~~~
ahk
Your criticism of current scientific practice does not go far enough. Recent
issues like the refusal to grant access to climate data cannot be so easily
brushed off with a "not enough bandwidth" complaint.

We have the internet now. There is no reason to not put up a site with data &
code when you publish your paper and just point all requests to go to the
site. Scientists need to start using these tools like the rest of us and stop
hiding behind such false excuses.

~~~
hga
His disclaimer that starts with " _The heavily politicized branches might not
be egalitarian..._ " since he has no experience with them covers the
particular exception you point out.

------
patio11
Amateurs are a useful check on the system because they will never submit a
grant proposal to the committee chaired by the co-investigator of the paper
they're reviewing, and hence can make aggressive, confrontational demands of
his code like running it.

~~~
JoachimSchipper
Which is why almost every scientific publication does not tell the peer
reviewers who the author is, and vice versa.

~~~
yummyfajitas
In my experience (math, physics, CS), most journals are only single blind. The
referee is anonymous, the author is not.

Additionally, referees can sometimes give themselves away.

 _The authors also fail to cite [Irrelevant Paper, Irrelevant Paper 2,
Peripherally related paper 3]. These works should be cited and compared to the
work being reviewed._

"Irrelevant Paper", John Smith. Journal of Trivialities, 1995.

"Irrelevant Paper 2", by John Smith and Steve Jones. Journal of followups to
irrelevant papers, 1998.

"Peripherally related paper 3", by John Smith and Phil Johnson. Journal of
Blah Blah, 2002.

~~~
scott_s
A minority of CS conferences are double-blind - I've only submitted to one
that was. But it's difficult to maintain true author-blindness. If your work
builds on previous work at all, it's difficult to maintain that. Further, I
once reviewed a paper that was double-blind, and before I made it through the
abstract I knew immediately where the paper came from. (Not the specific
authors, but I knew it had to be from a particular company.)

------
m_eiman
_"The most effective people at finding errors in scientific reasearch are
scientists: it was professional glaciologists, after all, who exposed the
error in the IPCC 2007 case study of Himalayan glaciers." To exclude the
amateur is to deny a large part of the history of science._

Well, he's not saying that a scientist is always right and an amateur is
always wrong. He's just saying that a scientist is more likely to find errors
in research material, which I think sounds reasonable since they're supposedly
trained to do it. Then there's e.g. the good ol' "seeing what you know is
there instead of what's actually there" effect which works against those
who've spent a lot of time with something.

Good find on the bad data, btw! Real science is in the numbers, not in the
propaganda surrounding them.

~~~
jgrahamc
I do agree that that quote doesn't tell the whole story, if you read the
professor's entire letter you'll see what I'm talking about.

------
pbiggar
I'm pretty sure Einstein had a PhD (I believe it was on Brownian motion).
Hardly a rank amateur. Patent offices don't hire Joe Soap to review patent
application.

~~~
david927
This is an important distinction, and the other amateurs always came early in
the field's history when it was a simpler science.

Climate science is not in its infancy. It's broad, deep and complex.

I desperately hope you're not implying that we should give credence to the din
of crackpot circus clowns who think dramatic climate change isn't really
happening and that Al Gore is personally placing space heaters on the glaciers
and polar icecaps.

~~~
david927
I'll take those downvotes as a "Yes, Al Gore is personally placing space
heaters on glaciers."

~~~
david927
_You are being downmodded because you are nonsensically comparing them to
deranged conspiracy theorists. You do so without evidence, most likely for
political purposes_

I have no political purpose -- and I'm not sure how politics play in watching
shipping lanes open up on the north pole.

I'm glad if they find errors. But correct me if I'm wrong, it hasn't change
_anything_. And this is what concerns me: that it raises a hand for debate.
The debate is over. We need to act and we need to have started yesterday. I
feel like I'm watching a woman get raped, but instead of people stopping it,
they want to talk about if it's really "rape" if the man wears a condom.

We are at a tipping point and we can, and should, cut carbon emissions. Moving
to alternate energy sources is an important step that we will make eventually
anyway, but to do it sooner means that the poorest of the world will not get
quite as screwed in the end.

~~~
jacoblyles
Most of the flaws that Steve found are in studies that claim to show that
current climate warming is unprecedented in human history. These studies are
important for drumming up the panic that you so ably display. It ought to be
of interest to any person that we get these studies right before we squander
trillions of dollars on useless environmental projects in a world where half
of humanity is still desperately poor.

But it seems that many people are more interested in having their own pet
ecopocalypse to feel self-righteous about than actually getting the science
correct.

~~~
david927
So you're saying the science is "wildly" off its claim? The north polar ice
cap isn't already half gone? No. You want to nitpick exact numbers because
that will stall any process, and hey, you have nothing to lose.

You're not poor. You're not at risk. It's the water supplies of millions of
poor that are vanishing, not yours. It's the possible forced migration of
millions that's pending, not yours.

These are not "useless environmental projects". They are changes we will make
regardless. But doing them now will save lives.

------
glymor
Amateurs scientists are still scientists, using similar methods and acting in
a constructive way. If nothing else many eyes effect makes them useful -
particularly in astronomy.

Journalists attempt to generate controversy to increase page views. Myles
Allen was correct that corrections to climate models have come from scientists
amateur or otherwise not journalists writing copy.

~~~
jgrahamc
I agree about the journalists bit, but when he says in his letter includes the
following:

 _John Christy took a lot of heat over the satellite data, but nothing
remotely like what is being turned on Phil Jones. It would have been romantic
if John's error had been uncovered by journalists combing through stolen
emails, or members of the public issuing freedom of information requests. But
it wasn't. It was found by the US government funding a painstaking independent
analysis of the satellite record, with John's co-operation, just as Phil has
said he would be happy to co-operate with an impartial and scientific re-
analysis of the surface temperature record, if anyone wants to fund such a
thing._

The implication is that none of these 'members of the public' are any good,
the only thing worth considering is funded research.

~~~
glymor
I agree he's too broad (and in other places as well). I had assumed it was
just the heat of the argument.

I also wonder about him invoking his professional capacity. It implicitly
makes him a representative of scientists; in acting so he really should have
been more nuanced.

------
araneae
Well, it used to be that being a scientist wasn't a profession at all. Most
science was done by the independently wealthy.

In many fields, today, to be an amateur you need to be independently wealthy
as well. Most science requires a lot of expensive equipment. The "amateurs" he
spoke of were using techniques that required little money (data analysis, pen
and paper math)

------
lazyant
I'm getting tired of this "Einstein also failed a test in university" meme
(usually suggesting in was in physics or math).

From what I remember from a biography he always did excellent in physics and
math, and in that famous entrance exam (that I think he took a year earlier
than usual) he only failed in French, which he didn't care much for.

~~~
jgrahamc
Did I claim that? It's entirely factual to say that he didn't get into ETH
Zurich, had a hard time getting a research job and did his best work while
employed by the patent office.

~~~
lazyant
Note that I agree with the general thesis of your article.

While you posted a fact (the failed admission test), that fact is a cherry-
picked outlier that gives the opposite impression of Einsteins actual academic
achievements (especially more so if you omit the rest of the circumstances).

It's like saying the fact that I won my Physics School chess championship in
1996 and leaving it there (true) without mentioning that I'm actually a
mediocre player that won because many of my opponents didn't show up.

My rant was more general; I'd swear I've seen in a couple of movies the
"Einstein flunked a physics test".

------
ZeroGravitas
I'm not sure the amateur-professional dichotomy is useful.

They put the data out there and someone found an error. Someone else might
spot an anomaly that leads to some important discovery.

Whether that someone is a professional scientist, a professional programmer, a
statistics student or a crank is mostly irrelevant. Putting stuff out for re-
use can (if planned for) be cheap and easy. There's no reason in this day and
age, not to do it.

That's what's important here and pitting professional against amateur doesn't
really move you forward on that issue.

~~~
arethuza
I have met some people with Chartered Engineer/PE status who program - not
many though. They are the only people I would call "professional programmers".

------
edw519
The most important lesson my mentor ever taught me:

We went to a client to work on Problem X. He quickly determined that solving
Problem X would achieve nothing. Problem Y was the real problem, but was way
outside our areas of expertise.

So what did he do? He slept 4 hours a night for the next 2 weeks studying
everything he could find about Problem Y. He reviewed reports, industry
literature, called experts, and talked to as many people in the company who
knew anything about that subject area. Within 2 weeks, he presented a
brilliant solution that no one had ever considered but was instantly
understandable by their experts. (That solution included work done by us and
we had a great client relationship for years.)

Later, I asked him why he tried to accomplish something so difficult with such
a seemingly tiny possibility of success. I'll never forget his reply...

"I didn't know that I couldn't do it, so I did it."

~~~
neilk
It's a charming anecdote, but don't take this as a story of individual
brilliance. Read it as faith and perseverance triumphing over doubt, and
conquering problems with appropriately applied work.

Note the crucial elements.

He talked to a lot of experts in field Y. This is not a "lone genius"!

He had the freedom and incentives to work on this almost uninterrupted. He
would have spent at least 100 hours, which is worth at least as much as two
undergraduate courses. Also, he was an expert in related field X, which must
have helped.

Furthermore, he because he knew the practical problem, he could cut down the
search space. He didn't have to investigate _everything_ about field Y, just
stuff applicable to him.

If you consider all that, the solution doesn't look so miraculous. What's more
interesting to me is that I think even ordinary people could be achieving that
kind of result _all the time_ if they had the right background, incentives,
and support.

------
nico
Here's another example, James Ellis, who came up with the idea of public key
cryptography: <http://www.wired.com/wired/archive/7.04/crypto_pr.html>

(HN submission: <http://news.ycombinator.com/item?id=1064931>)

~~~
jgrahamc
I'm not sure he counts as an amateur. He was employed to do this sort of
thinking, it was just done in secret.

------
DanielBMarkham
Never underestimate what a fresh set of eyeballs can do for a problem, even a
highly sophisticated and nuanced one.

Good hackers work in all sorts of fields they are not experts in. I can't tell
you the number of times I've seen "breakthroughs" with technical problems
simply because the smart group with all the jargon had to explain themselves
and their situation to the outside hackers just trying to make a computer
program help them out. It's extremely easy for groupthink to take over small
groups of specialists, and it happens most of the time without any of them
realizing it. By the way, just so we don't feel so cocky, this also happens
all the time in groups of programmers in a project who take hidden assumptions
and make them into unwritten law without anybody questioning them either.

The amateur, outsider, or programmer has a vital role in technical groups and
efforts of all stripes.

