
Grad Student Who Took Down Reinhart and Rogoff Explains Why They're Wrong - BruceM
http://www.businessinsider.com/herndon-responds-to-reinhart-rogoff-2013-4
======
jgrahamc
I have argued in the past that data and code need to be submitted with
academic papers: <http://blog.jgc.org/2013/04/the-importance-of-open-
code.html> This is just the latest example of why, IMHO, code needs to be
open.

~~~
jurassic
Yes! We need to see more material under Matt Might's CRAPL license. I feel a
strong urge to write off any result that doesn't have accompanying code with
it.

<http://matt.might.net/articles/crapl/>

~~~
anigbrowl
Great idea, awful awful name - CRAP license, as in 'Community
Research/Academic Programming' = CRAP. Why would your make your concept the
butt of jokes from the outset?

~~~
rflrob
It's right there on the page: "It's not the kind of code one is proud of." By
releasing it as CRAP, you're acknowledging that the code is imperfect,
reducing the perceived barrier. If academics feel like you can only release
code if it's pristine, it'll never get released, because for the most part,
the incentives aren't there to make clean code in an academic setting.

~~~
anigbrowl
This is to badly miss the point. What matters in this context is accuracy, not
elegance.

If I brute-force detection of the first 10000 prime numbers (for some non-CS
context) by testing their divisibility from _n_ to 1 it's horribly inefficient
and programmers may laugh, but it's accurate, simple, and easily reproducible.
Given that a definition of a prime number as _divisible only by itself and 1_
, it may be better to present a brute-force algorithm than to get into a
sideline discussion about validating some new-fangled method like the Sieve of
Erasthostenes, that upstart.

Science aims at proof rather than efficiency. What matters is the quality of
the result, and its reproducibility. If you can get to the same result in a
much more timely fashion that's awesome, but that's an engineering achievement
rather than a scientific one. I don't think that anyone wants to put out code
labeled as CRAP in an attempt to mollify engineers.

~~~
pjscott
Have you seen typical research code? It's crap. If you looked at it, your
first reaction would probably be "Wow, this is crap." It's created in a hurry,
usually grows by accretion, and the people who write it are usually pretty
half-assed at programming, having made the solidly practical decision to focus
more on the other aspects of their field. The commenting is usually lousy-to-
nonexistent, the indentation is frequently screwy, and everything about it
just screams "I am a temporary hack, written only to get publishable results."

And that's to be expected, because of how most scientific code gets written,
and what the incentives are. And releasing this code would still be strictly
better than _not_ releasing it. We just need to keep the expectations of code
quality low, which the CRAPL license does.

~~~
anigbrowl
Only programmers have high expectations of code quality. If I'm a biologist,
my target audience is other biologists, not programmers.

Your comment reads to me like you want non-programmers to ritually humiliate
themselves by labeling their stuff as CRAP before you will deign to parse it
for correctness. Patronizing people is not a good way to get them on-side.

 _We just need to keep the expectations of code quality low, which the CRAPL
license does._

The purpose of the CRAPL is to make code accessible, not to lower
expectations. It's _not about_ what you think of the code, it's about whether
the code yields correct results.

------
davesims
For me this was the 'tl;dr' quote:

"It would be absurd to think that governments never have to worry about their
level of indebtedness. The aim of our paper was much more narrowly focused. We
show that, contrary to R&R, there is no definitive threshold for the public
debt/GDP ratio, beyond which countries will invariably suffer a major decline
in GDP growth."

~~~
SilasX
R&R weren't trying to claim that, were they?

~~~
bhb916
I think they were. Speaking as someone who is generally pro-austerity, I think
R&R concluded that after 90% debt/GDP there was major drop-off in GDP growth.
The corrections all but destroy this assertion. That's not to say that debt
and GDP growth aren't inversely proportional/correlated -- just that there is
no magic level that changes the relationship substantially.

Honestly, there are so many variables and dimensions in this system and so
little data I'm not sure we should ever be drawing sweeping conclusions like
this.

~~~
SilasX
>I think they were. Speaking as someone who is generally pro-austerity, I
think R&R concluded that after 90% debt/GDP there was major drop-off in GDP
growth.

That doesn't mean they were asserting what the grad students claim, that the
falloff in growth is _unavoidable_ , which sounds like an extreme strawman.

~~~
yyqux
R&R's defense strategy seems currently to pretend that they weren't claiming
anything in particular by the paper originally: "hey man, I was just putting
it out there. I didn't, like, mean anything by it."

But their own public pronouncements, and the way they responded to other's
interpretations of the results, pretty much indicated that they though they'd
found firm evidence that there was some big non-linearity at 90%.

~~~
SilasX
I'm not claiming R/R are saints. I'm just objecting to ridiculous strawman
implied by:

"We show that, contrary to R&R, there is no definitive threshold for the
public debt/GDP ratio, beyond which countries will invariably suffer a major
decline in GDP growth."

RR's claim is entirely statistical, so of course they're not going to claim
that exceeding the ratio _always_ ("invariably") leads to a major decline in
growth.

RR's being wrong doesn't mean it's okay to insinuate they hold ridiculous
positions that they really don't. That's just compounding public
misconceptions.

~~~
pseut
Have you been following RR's op-eds, etc over the last two years? I have the
impression that they've been less circumspect than you claim.

~~~
SilasX
"X carries too much/undesirable risk of Y" != "X unavoidably leads to Y"

The former is what RR said, the latter is the strawman being attributed to
them.

------
harshpotatoes
ErSo, there are a few things I don't quite understand. In the original study,
Renhart had erroneously averaged 7 numbers together to get a slightly negative
growth. The 'correct' result is obtained from averaging eight numbers
together, resulting in slightly positive growth. What is unclear to me, is how
are either of these numbers considered to be very significant? With so few
samples in the average, I would draw the conclusion that both measurements
agree with one another and the error bars for their measured result are quite
a bit larger than they seem to be suggesting. Is this common in economics?

~~~
jessriedel
Macroecon has the simultaneous hindrances of (1) not being able to run a
controlled experiment and (2) a dearth of data. It doesn't excuse people for
being far too confident in the conclusions they draw, but it does explain why
such large differences of opinions can persist (especially on such a
politically charged subject). People can't be pulled away from their priors.

~~~
dragonwriter
> Macroecon has the simultaneous hindrances of (1) not being able to run a
> controlled experiment and (2) a dearth of data.

Well, except #1 should be "not being able to run experiments in laboratory
conditions". Statistical controls in an experiment are still controls, and it
is still possible though (because of the point made briefly by #2, to run
statistically-controlled experiments) sometimes of limited utility; the dearth
of data (specifically, the small number of data points compared to the number
of independent variables that need to be controlled for) is what limits the
utility of many experiments with statistical controls in the field.

~~~
jacques_chester
Even if you use rigorous statistical techniques -- and economists do under the
heading of econometrics, even inventing new techniques -- it's _still_ really
hard to draw meaningful conclusions.

There are just _so many variables_. And the more interconnected the world
becomes, the more variables there are. Pointing to any one correlation and
you'll find a hundred statistical fingers pointing at correlations with
totally different consequences.

~~~
dragonwriter
> Even if you use rigorous statistical techniques -- and economists do under
> the heading of econometrics, even inventing new techniques -- it's still
> really hard to draw meaningful conclusions.

> There are just _so many variables_.

Right. That's the problem with too few data points given the number of
independent variables that need to be controlled for. I addressed that
explicitly.

------
wpietri
Seems like a great argument for more transparency. This guy wasted a lot of
time trying to guess what they had done. Given that publishing all the
supporting materials is approximately free, perhaps journals could start
requiring a git link that contains data, code, and paper drafts.

~~~
niggler
"journals could start requiring a git link that contains data, code, and paper
drafts."

But when the goal is to create a result rather than report it, transparency is
the enemy

~~~
carbocation
> But when the goal is to create a result rather than report it, providing
> data, transparency is the enemy

I strongly believe that errors are a much more pervasive problem in science
and related fields than malice is.

~~~
niggler
"science and related fields "

But we are talking about economics, and republican groups like fox news eat it
up.

My favorite example is the 2011 chart distorting the display of the
unemployment rate:

[http://mediamatters.org/blog/2011/12/12/today-in-
dishonest-f...](http://mediamatters.org/blog/2011/12/12/today-in-dishonest-
fox-news-charts/185162)

~~~
carbocation
Fox News is not in the business of science. I used the term "related fields"
as a euphemism for Econ and Psych because I didn't want to get drawn into a
demarcation debate.

Wherever the line is drawn, the dishonesty of some random Fox News chart is
not related to the honesty or dishonesty of actual scientific research.

The much bigger problem _in science_ is error.

~~~
niggler
" didn't want to get drawn into a demarcation debate."

Econ and Psych really do lie in a gray area, and by avoid the demarcation
debate you completely miss the relevance of the issue at hand.

If policy makers weren't using this study to justify more austerity, then we
probably wouldn't have such a prolonged discussion.

------
tokenadult
From Jelte Wicherts writing in Frontiers of Computational Neuroscience (an
open-access journal) comes a set of general suggestions

Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting
the daylight in: reviewing the reviewers and other ways to maximize
transparency in science. Front. Comput. Neurosci., 03 April 2012 doi:
10.3389/fncom.2012.00020

[http://www.frontiersin.org/Computational_Neuroscience/10.338...](http://www.frontiersin.org/Computational_Neuroscience/10.3389/fncom.2012.00020/full)

on how to make the peer-review process in scientific publishing more reliable.
Wicherts does a lot of research on this issue to try to reduce the number of
dubious publications in his main discipline, the psychology of human
intelligence. It appears that the discipline of economics research needs help
with data openness too.

"With the emergence of online publishing, opportunities to maximize
transparency of scientific research have grown considerably. However, these
possibilities are still only marginally used. We argue for the implementation
of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and
(3) online data publication. First, peer-reviewed peer review entails a
community-wide review system in which reviews are published online and rated
by peers. This ensures accountability of reviewers, thereby increasing
academic quality of reviews. Second, reviewers who write many highly regarded
reviews may move to higher editorial positions. Third, online publication of
data ensures the possibility of independent verification of inferential claims
in published papers. This counters statistical errors and overly positive
reporting of statistical results. We illustrate the benefits of these
strategies by discussing an example in which the classical publication system
has gone awry, namely controversial IQ research. We argue that this case would
have likely been avoided using more transparent publication practices. We
argue that the proposed system leads to better reviews, meritocratic editorial
hierarchies, and a higher degree of replicability of statistical analyses."

~~~
MaysonL
Note that the Reinhart-Rogoff article was not peer-reviewed.

------
zaroth
It's not related to the article, but rather the site hosting it.

I noticed more than one fishy looking 3rd party domain loading while the page
downloaded, so went into Inspector to see what was up. There are one or more
resources loaded from each of the following domains, in many cases including
javascript...

2mdn.net, scorecardresearch.com, bizographics.com, tynt.com, optimizely.com,
google.com, sail-horizon.com, facebook.com, 247realmedia.com,akamaihd.net,
vizu.com, pubmatic.com, imrworldwide.com, advertising.com,
googlesyndication.com, doubleclick.net, chartbeat.com, sharethrough.com,
fbcdn.net, skimresources.com, gstatic.com, stumbleupon.com, tynt.com,
adadvisor.net, youtube.com, shareth.ru, agkn.com, yimg.com

'shareth.ru' seemed particularly suspect, until I realized it was probably
sharethrough.com trying to be cute.

There are so many domains being trusted here the drive-bys could have drive-
bys.

------
codeulike
Its fascinating that part of the problem in the Reinhart and Rogoff paper
comes down to an Excel formula error
[http://nymag.com/daily/intelligencer/2013/04/grad-student-
wh...](http://nymag.com/daily/intelligencer/2013/04/grad-student-who-shook-
global-austerity-movement.html)

~~~
ScottBurson
One columnist I read said something like "it will be interesting to see what
historians make of the fact that a global economic policy was driven in part
by an error in an Excel spreadsheet".

~~~
pseut
It's rare that we get such a clean natural experiment like this in economics;
maybe future economists will thank R&R for cleverly introducing this negative
exogenous policy shock (sarcasm).

------
justin_vanw
Journals shouldn't publish papers without all the supporting data and computer
code necessary to reproduce the result.

This seems obvious, but established academics have a vested interest in not
sharing. Sharing data and code opens their work up to impeachment (as is seen
here), and it gives others a jumping off point to extend their work.

Both of these side effects of sharing code and data are bad for the careers of
successful academics, but good for literally everyone else in the world
(including, ironically, these academics). It should be a no brainer, but then
again, who referees these papers? The very people who have the most to lose by
such a change.

It will take a lot of clamoring from the outside to bring about a world where
scientific work is considered not legitimate without full documentation of the
experiment performed.

~~~
YokoZar
I recall hearing somewhere that the original paper wasn't actually published
in a peer-reviewed journal, so requirements for data and code wouldn't really
have applied anyway.

Perhaps the greater lesson is that we should trust proper journals for a
reason.

------
driverdan
This was an interesting and well written followup but I'm disappointed in
Herndon's decision to publish it on Business Insider. BI is generally full of
blogspam and has interstitial ads.

~~~
cjensen
Agreed. BI is edited by Henry Blodget who was charged with securities fraud as
an analyst which lead to him paying millions in settlement and being
permanently banned from the securities industry. Really not where you want to
publish anything meant to be taken seriously.

<http://en.wikipedia.org/wiki/Henry_Blodget>

------
beefman
More debate over 20 data points with no explanatory power one way or the other
(as Yglesias pointed out). This is not uncommon in macro. Another recent case
also centered on work by R&R

[http://johnbtaylorsblog.blogspot.com/2012/10/simple-proof-
th...](http://johnbtaylorsblog.blogspot.com/2012/10/simple-proof-that-strong-
growth-has.html)

(one of a host of commentary pieces on both sides... but essentially none in
the middle)

------
205guy
I think this method of averaging should hereby be called the Reinhart-Rogoff
Average, RRA for short.

I should program it into my business intelligence software, it's gotta be
useful for something--like for cooking the books.

------
ikhare
This whole situation makes me think: is there a static analysis tool for
excel? Seems like missing a few rows in an excel calculation could be
flagable.

~~~
tonyarkles
There sort of is one built-in. It usually warns you if you're doing things on
a group of cells and you're missing one of them.

------
ttrreeww
That grad student won't be getting the big job offers in industry, great way
to destroy your career by biting the hand that feeds you.

(Sarcastic, but it's real life)

~~~
nazgulnarsil
Doesn't matter, he'll now have a lucrative career being a USG debt apologist.

