Weak links

Earlier this year, economist Paul Collier visited the Netherlands and talked about the policy recommendations spelled out in his best-selling book The Bottom Billion. He flew in from Africa, spent a few hours in the Netherlands, and then was on his way to rescue the Middle East. Recently Jeffrey Sachs and William Easterly – equally high-powered economists – also honoured the Netherlands with a visit, preaching their own version of the economic gospel.

Sachs, Easterly and Collier disagree on many economic issues. But they have in common the fact that their academic careers and reputations as policy advisors are to some extent based on the econometric approach of cross-country regressions.

The idea is simple: you collect information on a vast number of variables for a wide variety of countries and ask a computer to search for causal relationships. With the extensive datasets that are available on the internet this is easy to do, and it is a fun way to pass a rainy afternoon. A routine application is to ‘regress’ economic growth on a bunch of variables, including measures of investment, school enrolment, the rule of law and foreign aid. The computer spits out statistical associations almost instantly.

Most economists are well aware of the potential pitfalls of such an approach and take the outcomes with more than a pinch of salt. But non-experts can be easily misled by impressive data that suggest ‘hard’ and precise results and convey the illusion of absolute truth. The influence that cross-country growth regressions have on policy debate and formulation bears no relation to the shaky foundations on which they are based.

What are the potential problems with cross-country regressions? Most studies are not rooted in theory and are ‘open-ended’ in terms of the variables that may be included. The ‘one-size-fits-all’ approach of linear models implies that the link between income (or growth) and the selected variables is the same for all countries. But is it realistic to think the effect of institutions (or an abundance of natural resources) on economic growth is the same in Sudan as it is in Canada or Peru?

Cross-country studies suffer from limited data availability, tend to ignore ‘within-country’ heterogeneity of variables and generally struggle with the thorny issue of establishing causation. Although documenting correlations between variables is easy enough, it is much more difficult to prove, for example, that good health or secure property rights contribute to high incomes rather than the other way around.

A recent examination of World Bank research found that many prominent papers simply reveal ‘correlations between endogenous variables’ without containing any evidence of causation. Such research is uninformative for policy makers. On a related note, it is difficult to tease practical policy advice from studies that lump all countries together without paying much attention to channels or mechanisms through which ‘deep factors’ are related to outcomes. For example, it is not helpful to advise African policy makers to improve their institutions to meet Scandinavian standards.

One of the most influential economics papers of the new millennium was written by Craig Burnside of Duke University and David Dollar of the World Bank. It tells the tale that development assistance only ‘works’ in countries with good institutions and policies. This message had intuitive appeal and fit the agenda of certain stakeholders such as the World Bank. It had a thorough impact on the development policies of donors.

Unfortunately, the results did not hold up to scrutiny. The paper did not tackle the endogeneity of aid flows, and its authors added a few data points, extending the dataset from 1993 to 1997or simply filling gaps in the original dataset. This rendered the results insignificant. The definitions of ‘aid’ and ‘good policies’ were also slightly changed when the original dataset was used. Now that the dust has settled we must conclude that, in spite of the hoopla, there is precious little in terms of content that is useful for policy makers. Unfortunately this is not an exception.

I have no problem with empirical analysis – the opposite is true. Careful empirical analysis is indispensible for informing policy makers, and cross-country regressions can provide some useful background that is somewhat based in reality. But keeping in mind Benjamin Disraeli’s warning about ‘lies, damned lies and statistics’, one cannot be too careful when weighing the ‘evidence’ that is produced.

Footnotes