Replication scandal: We might not need austerity measures after all

Economists Carmen Reinhart and Kenneth Rogoff claim in “Growth in a Time of Debt” that countries with debt-to-GDP ratios above 90 percent have a slightly negative average growth rate. Their paper is one of the most cited by governments to justify austerity measures – and it might be wrong, as a replication paper shows.

Claims in the original paper

The sharp run-up in public sector debt will likely prove one of the most enduring legacies of the 2007–2009 financial crises in the United States and elsewhere. We examine the experience of 44 countries spanning up to two centuries of data on central government debt, inflation and growth. Our main finding is that across both advanced countries and emerging markets, high debt/GDP levels (90 percent and above) are associated with notably lower growth outcomes. (…) Seldom do countries “grow” their way out of debts. The nonlinear response of growth to debt as debt grows towards historical boundaries is reminiscent of the “debt intolerance” phenomenon (…).

Claims in the replication paper

“[C]oding errors, selective exclusion of available data, and unconventional weighting of summary statistics lead to serious errors that inaccurately represent the relationship between public debt and GDP growth among 20 advanced economies in the post-war period. Our finding is that when properly calculated, the average real GDP growth rate for countries carrying a public-debt-to-GDP ratio of over 90 percent is actually 2.2 percent, not −0.1 percent as published in Reinhart and Rogoff. That is, contrary to RR, average GDP growth at public debt/GDP ratios over 90 percent is not dramatically different than when debt/GDP ratios are lower. We also show how the relationship between public debt and GDP growth varies significantly by time period and country. Overall, the evidence we review contradicts Reinhart and Rogoff’s claim to have identified an important stylized fact, that public debt loads greater than 90 percent of GDP consistently reduce GDP growth.”

Impact of the replication

In the past, policy makers have used the Reinhart-Rogoff results to show how debt and growth are connected, and to propose austerity measures. “If a country reached a debt burden of more than 90 percent of its annual economic output, the logic went, it would quickly fall into a debt trap that would leave it struggling to grow in the coming years. Prominent politicians — including Olli Rehn of the European Commission and Representative Paul D. Ryan, the chairman of the House Budget Committee — cited it as a reason to try to impose major budget cuts.” (New York Times) The newly published replication paper means that we might actually not be able to associate heavy debts with a ‘growth trap’. Austerity measures in the last years might be based on wrong results. Dean Baker, co-director of the Center for Economic and Policy Research in Washington, D.C., even states that “this should be the cause for a major reassessment of the deficit reduction policies being pursued in the United States and elsewhere.”

What went wrong?

The blogs simply statistics and nextnewdeal.net explain what the replication paper uncovered:

1. The original paper excluded important data from their final analysis
2. The original paper used inappropriate weighting
3. The original paper had an error in an Excel formula which dropped several countries from their final analysis

What will change?

Policy makers might not change their view of austerity measures, but they might simply have to cite another paper and answer criticism more carefully. But among researchers the value of replication will hopefully be recognized more than before. Without replication, economics, political science and policy makers might base their decisions and work on wrong results – even if it’s just because of an excel error.

Links

• Original paper: Reinhart, Carmen M., and Kenneth S. Rogoff. 2010. “Growth in a Time of Debt.” , American Economic Review Papers and Proceedings 100(2): 573-78. [pdf]
• Replication paper: Thomas Herndon, Michael Ash & Robert Pollin, Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff [pdf]
• New York Times: A Study That Set the Tone for Austerity Is Challenged
• New York Times: Response From Reinhart and Rogoff
• Simply Statistics: I wish economists made better plots
• Next New Deal: Researchers Finally Replicated Reinhart-Rogoff, and There Are Serious Problems
• Slate: Is The Reinhart-Rogoff Result Based on a Simple Spreadsheet Error?
• Slate: Reinhart and Rogoff Respond
• Paul Krugman in the New York Times: More On Reinhart-Rogoff
• Center for Economic and Policy Research: How Much Unemployment Was Caused by Reinhart and Rogoff’s Arithmetic Mistake?
• Andrew Gelman: Memo to Reinhart and Rogoff: I think it’s best to admit your errors and go on from there
• Retraction Watch: Influential Reinhart-Rogoff economics paper suffers spreadsheet error