Reinhart & Rogoff – the debt cliff that wasn’t

Reinhart & Rogoff, Fig 2

Timely as ever, I thought I’d finally get around to writing something about this Reinhart & Rogoff business. If you’re reading this blog, there’s a good chance you’ll be familiar with the story – a while ago, two high profile economists, Carmen Reinhart and Ken Rogoff, announced a finding with big implications for the austerity debate. They claimed that there was a strong negative relationship between national debt and economic growth. And further, that there was a pronounced ‘cliff’ at a 90% debt-to-GDP ratio – if a country crossed over this debt threshold, its prospects for economic growth were suddenly pretty dismal.

This finding was obviously of great interest to policy makers. If a high debt-to-GDP ratio strangles growth, then this is a big strike against the case for borrowing to stimulate the economy. Reinhart and Rogoff were invited to discuss their results with senators and congressmen, and were cited approvingly by politicians in the US and Europe.

That was until recently, when the story broke that they’d made a mistake in their calculations. Thomas Herndon, a graduate student at the University of Massachusetts, Amherst, when attempting to replicate their results for a class assignment, found that he couldn’t. It turned out that Reinhart and Rogoff made a basic data error in calculating average debt and growth figures for the countries they were analysing. Instead of selecting 20 countries, they’d accidentally omitted the first five countries alphabetically. As well as this, Herndon and his professors questioned the way they’d weighted various country’s data in the analysis. Basically, in Reinhart and Rogoff’s original analysis, any occurrence of negative growth in a country counted for as much as any other. So an economic contraction that lasted one year counted for as much as one that lasted for ten or twenty.  An alternative weighting meant a much weaker relationship between debt and growth and, crucially, no 90% ‘cliff’.

After all this came out, a lot of stories focused on how Reinhart & Rogoff’s original paper was never actually published in a peer-reviewed journal. A recent story on the always excellent More or Less emphasised the fact that politicians shouldn’t rely on unpublished work that hasn’t had a chance to be subjected to proper academic scrutiny. This is certainly true, and it’s worrying (though not at all surprising) that politicians weren’t quite smart enough to ask how country’s data were weighed in this very simple analysis, but were easily morally bankrupt enough to characterize a single unpublished study as a “widely acknowledged” fact, as did (EU economics commissioner) Ollie Rehn. However, what also worries me is that ‘proper academic scrutiny’ might not have solved this problem either.

In a generous mood I’d grant that peer-review, or subsequent academic criticism might have caught the weighting issue. This is the kind of thing that tends to be noticed by academics who disagree with someone else’s conclusions. However, what Herndon did was something that almost no-one does any more – he tried to replicate an existing finding. Despite the fact that one of the basic principles of the scientific method is that your results should be replicable, it’s very rare that an academic will go to the trouble of trying to reproduce an existing result using exactly the same materials and methods. This is true even in economics which often relies on existing, publicly available data – which should make replication pretty straightforward.

The main reason for this parlous state of affairs is that, if you’re an academic, replicating other people’s work doesn’t get you anything. You’d struggle to get anyone to give you money to repeat something that’s already been done, and no peer-reviewed journal would publish your results even if you did. Career-wise, replicating an existing finding will almost always be a complete waste of time – time that could be spent doing your own original work.

A second, maybe slightly less important reason we don’t see more replication is that academics don’t share. At least they don’t share as much as they should. In most studies in economics (and other fields that rely on existing data, like sociology and epidemiology), all it would take to replicate someone else’s findings would be access to their original ‘spreadsheet’ (the original data, along with a list of what calculations they did). The only reason Herndon found what he found was that Reinhart and Rogoff were kind enough to send their spreadsheet to him. A lot of academics wouldn’t do that. Why send this data, data you’ve likely spent months compiling, to someone who can just use it for their own research? Who’s to say they won’t use it to beat you to some interesting finding, setting back your own research agenda and putting you that much closer to being out of a job? On top of that, why show them your workings as well? When handing in, say a finished report at work, would you want someone else to see every step you went through to get to that final report? What if there’s an embarrassing mistake in there?

Reinhart and Rogoff’s embarrassing mistake got caught (if you listen to them, it didn’t change their results much anyway). But hundreds of mistakes like this slip through the academic net every day. They make it through into papers that may be cited hundreds or thousands of times; sometimes by policy makers or doctors with the ability to do real harm to real people. So here’s hoping the Reinhart and Rogoff case prompts a new enthusiasm for replication. But I wouldn’t hold my breath.

About Robert de Vries

I'm an Early Career Research Fellow in the Sociology department at the University of Oxford. I'm mainly interested in how people are affected by concerns about their social status; how it colours the way they think, feel, and behave. I try and contribute here regularly, but my addiction to writing excessively long posts keeps getting in the way.
This entry was posted in Blog posts and tagged , , , , , . Bookmark the permalink.

One Response to Reinhart & Rogoff – the debt cliff that wasn’t

  1. Ben Baumberg says:

    Couldn’t agree with this more. Someone (I can’t find it now, but if I discover it again I’ll put the link back here) recently made a suggestion that we should have a journal where the first half of it is challenges to the current cosy consensus, and the second half is replications of existing claims – on the grounds that these are the two hardest things to get published in academic journals these days, and they make a nice pairing…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s