Much to our surprise, spreadsheet formulae have hit the news! A student has exposed simple errors in a previously well-respected economic study.

Back in January 2010, Harvard professors Carmen Rienhart and Kenneth Rogoff released a paper entitled Growth in a time of debt1, which concluded:

"Our main finding is that across both advanced countries and emerging markets, high debt/GDP levels (90 percent and above) are associated with notably lower growth outcomes."2

This paper has received a huge amount of publicity and has been relied on by politicians across the globe as support for 'austerity' economic measures.3

It has made the news because a critique was published by graduate student Thomas Herndon and co-authors Professors Michael Ash, and Robert Pollin of the University of Massachusetts. The Hendon-Ash-Pollin paper outlined three central critiques:

  1. selective exclusions of data
  2. unconventional averaging
  3. a coding error

According to the paper, adjusting for these items means that the controversial conclusion that 90% debt/GDP levels are "associated with notably lower growth" cannot be made.

As we're not experts in economic theory, we aren't best placed to comment on the first two issues4. However it is the coding error that has led to so much media attention.

Reinhard and Rogoff's paper contained a simple calculation error in the Excel file that supported their paper - when calculating the average growth where debt levels were more than 90% of GDP, the formula calculating the average only included rows 30-44, rather than rows 30-49.5 The missing rows had the effect of excluding New Zealand from the results.

This kind of calculation error is what keeps us awake at night!

One of the challenges of being a forensic accountant is that our work is heavily scrutinised by others (and, as a matter of practice, ought to be first heavily scrutinised by ourselves). There are a large number of people including lawyers, barristers, judges and other forensic accountants whose job it is to find mistakes and flaws in any report we issue.

The perception of a simple error can also be much worse than the effect of the error itself. It was in fact the other two criticisms in Reinhart and Rogoff that made a larger difference to the results. The coding error itself didn't change the results dramatically. However, this didn't stop the media, whose coverage focussed on their perception that the study 'couldn't even add up the numbers correctly' rather than the validity of the overall academic research. Whilst Reinhart and Rogoff may well be correct in their arguments rebutting Hendon's claims, it may be hard for them to get over this perception issue.

In many aspects of life, the prospect of a simple error can be a significant cause of stress. Professional firms try to avoid such errors by quality assurance procedures such as peer reviews of reports and arithmetic checking. Whilst we try to be as efficient as possible and explain the time we've taken to do things, it is quite hard to explain the net benefits from such quality assurance procedures. There is often no tangible outcome beyond a lack of errors, something that our clients would expect anyway!

However, the experience of Rinhart and Rogoff has shown that a failure to pick up a simple error can undermine the overall effect of analysis. Even if the error itself doesn't make a significant difference, it becomes the natural focus of a reader (in our case an opposing barrister), causing the perception of inaccuracy to outweigh the actual error itself.

Footnotes

1http://www.nber.org/papers/w15639.pdf
2http://www.nber.org/papers/w15639.pdf, page 22
3See http://qz.com/75117/how-influential-was-the-study-warning-high-debt-kills-growth/#75117/how-influential-was-the-study-warning-high-debt-kills-growth/
4Reinhard and Rogoff have rejected them, although this is an ongoing debate. See: http://blogs.wsj.com/economics/2013/04/17/reinhart-rogoff-admit-excel-mistake-rebut-other-critiques/ and http://qz.com/75117/how-influential-was-the-study-warning-high-debt-kills-growth/#75439/reinhart-and-rogoffs-defense-is-misleading-and-heres-how/
5For the sceptics, a visual illustration of the file is on Mike Konczal's blog: http://www.nextnewdeal.net/rortybomb/researchers-finally-replicated-reinhart-rogoff-and-there-are-serious-problems

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.