Timely as ever, I thought I’d finally get around to writing something about this Reinhart & Rogoff business. If you’re reading this blog, there’s a good chance you’ll be familiar with the story – a while ago, two high profile economists, Carmen Reinhart and Ken Rogoff, announced a finding with big implications for the austerity debate. They claimed that there was a strong negative relationship between national debt and economic growth. And further, that there was a pronounced ‘cliff’ at a 90% debt-to-GDP ratio – if a country crossed over this debt threshold, its prospects for economic growth were suddenly pretty dismal.
This finding was obviously of great interest to policy makers. If a high debt-to-GDP ratio strangles growth, then this is a big strike against the case for borrowing to stimulate the economy. Reinhart and Rogoff were invited to discuss their results with senators and congressmen, and were cited approvingly by politicians in the US and Europe.
That was until recently, when the story broke that they’d made a mistake in their calculations. Thomas Herndon, a graduate student at the University of Massachusetts, Amherst, when attempting to replicate their results for a class assignment, found that he couldn’t. It turned out that Reinhart and Rogoff made a basic data error in calculating average debt and growth figures for the countries they were analysing. Instead of selecting 20 countries, they’d accidentally omitted the first five countries alphabetically. As well as this, Herndon and his professors questioned the way they’d weighted various country’s data in the analysis. Basically, in Reinhart and Rogoff’s original analysis, any occurrence of negative growth in a country counted for as much as any other. So an economic contraction that lasted one year counted for as much as one that lasted for ten or twenty. An alternative weighting meant a much weaker relationship between debt and growth and, crucially, no 90% ‘cliff’.
After all this came out, a lot of stories focused on how Reinhart & Rogoff’s original paper was never actually published in a peer-reviewed journal. A recent story on the always excellent More or Less emphasised the fact that politicians shouldn’t rely on unpublished work that hasn’t had a chance to be subjected to proper academic scrutiny. This is certainly true, and it’s worrying (though not at all surprising) that politicians weren’t quite smart enough to ask how country’s data were weighed in this very simple analysis, but were easily morally bankrupt enough to characterize a single unpublished study as a “widely acknowledged” fact, as did (EU economics commissioner) Ollie Rehn. However, what also worries me is that ‘proper academic scrutiny’ might not have solved this problem either.
In a generous mood I’d grant that peer-review, or subsequent academic criticism might have caught the weighting issue. This is the kind of thing that tends to be noticed by academics who disagree with someone else’s conclusions. However, what Herndon did was something that almost no-one does any more – he tried to replicate an existing finding. Despite the fact that one of the basic principles of the scientific method is that your results should be replicable, it’s very rare that an academic will go to the trouble of trying to reproduce an existing result using exactly the same materials and methods. This is true even in economics which often relies on existing, publicly available data – which should make replication pretty straightforward.
The main reason for this parlous state of affairs is that, if you’re an academic, replicating other people’s work doesn’t get you anything. You’d struggle to get anyone to give you money to repeat something that’s already been done, and no peer-reviewed journal would publish your results even if you did. Career-wise, replicating an existing finding will almost always be a complete waste of time – time that could be spent doing your own original work.
A second, maybe slightly less important reason we don’t see more replication is that academics don’t share. At least they don’t share as much as they should. In most studies in economics (and other fields that rely on existing data, like sociology and epidemiology), all it would take to replicate someone else’s findings would be access to their original ‘spreadsheet’ (the original data, along with a list of what calculations they did). The only reason Herndon found what he found was that Reinhart and Rogoff were kind enough to send their spreadsheet to him. A lot of academics wouldn’t do that. Why send this data, data you’ve likely spent months compiling, to someone who can just use it for their own research? Who’s to say they won’t use it to beat you to some interesting finding, setting back your own research agenda and putting you that much closer to being out of a job? On top of that, why show them your workings as well? When handing in, say a finished report at work, would you want someone else to see every step you went through to get to that final report? What if there’s an embarrassing mistake in there?
Reinhart and Rogoff’s embarrassing mistake got caught (if you listen to them, it didn’t change their results much anyway). But hundreds of mistakes like this slip through the academic net every day. They make it through into papers that may be cited hundreds or thousands of times; sometimes by policy makers or doctors with the ability to do real harm to real people. So here’s hoping the Reinhart and Rogoff case prompts a new enthusiasm for replication. But I wouldn’t hold my breath.