In his column ‘Weak links’ (The Broker 10) professor Erwin Bulte launches a fierce attack on economists such as Collier, Sachs and Easterly who have become influential economic policy advisors in spite of their careers and reputations being ‘to some extent based on the econometric approach of cross-country regressions’. He does so by suggesting that cross-country regression analysis is often (mis)used in searching for a relationship between , for example, aid and growth. For this purpose professor Bulte presents a number of problems, deficiencies and drawbacks related with empirical analysis using cross-country regression. In response I would like to make a number of comments.
First, professor Bulte’s reproaches are surely not fresh news. In a book review from The Economist, 30 March 2002, one reads, ‘the main fault is that Mr. Easterly (The Elusive Quest for Growth) rather like the Bank (World Bank) itself, relies too much on cross-country statistical comparisons. The truth is messier; specific to each country’s time and place’.
Secondly, even if Easterly has committed such sins in the past, he is now repentant. ‘Of course’, he has said, ‘the largest question is whether aid raises economic growth. There is a vast and inconclusive literature on aid and growth. The literature suffers from such unrestricted specifications and endless iterations among these specifications that virtually any result on aid and growth is possible, and indeed virtually all possible results have already been presented in the literature…’.1
This matter has been investigated in depth by David Roodman.2 who shows that in the debate on the effectiveness of foreign aid in developing countries, the specification search as it is often practiced indeed invalidates the traditional theories of inference. That is, there is too little systematic analysis of the relationship between assumptions and conclusions. In this context Roodman tests seven important aid-growth papers for robustness. His results indicate that fragility and not robustness is the norm in the cross-country aid-growth literature.
On the other hand, Roodman emphasizes that the seven papers tested did produce useful insights in the aid-growth relationship, which should not be dismissed. To quote Roodman, ‘Are recipient policies, exogenous economic factors, and post-conflict status irrelevant to aid effectiveness? Are there no diminishing returns to aid? Is helping the neediest countries a hopeless task? No. There can be no doubt that some aid finances investment and that domestic policies, governance, external conditions, and other factor’s these authors study influence the productivity of investment’. According to Roodman the fragility of the aid effectiveness results is due to the fact that ‘aid is probably not a fundamentally decisive factor for development, not as important, say as domestic savings, inequality, or governance’. Moreover, aid is not homogenous. It consists of everything from food aid to technical assistance. In addition, it is often poorly used.
The important issue of aid effectiveness nowadays more and more attracts the attention of Dutch politicians, including the Minister for Development Cooperation. In this context a recent study by Radelet and Levine.3 on three broad views on aid effectiveness is relevant. The first strand is that aid has no effect on growth and may actually undermine it. The second strand is that aid has a positive relationship with growth on average across countries (although not in every country), but with diminishing returns as the volume of aid increases. The third strand finds that aid has a conditional relationship with growth, helping to accelerate growth only under certain circumstances. This ‘conditional’ strand depends on either the characteristics of the recipient country or the practices and procedures of the donors.
Summarizing the relevant literature, the authors conclude that research that assumes a linear relationship and that all aid is alike, the first strand, tends to find little or no relationship between aid and growth. On the other hand, there are a number of studies.4 in which these two assumptions are relaxed and which find a strong positive relationship between growth and aid aimed at growth (e.g. to build roads, ports, electricity generation, or support agriculture), leaving aside other types of aid not directly aimed at growth (humanitarian and emergency assistance). The results obtained in these studies turn out to be robust. The most prominent recent study of the second strand is by Rajan and Subramanian.5 They find no relationship between aid and growth when they restrict their model to a linear relationship. However, when they relax the linearity assumption as well as the assumption that all aid is alike, they find a strong positive and significant relationship between economic aid and growth with diminishing returns. The ‘conditional’ strand of the literature tends to results that are fragile with respect to the relevance of stronger policies and institutions. In this respect Radelet and Levine comment that although the Burnside and Dollar (2000)6 conclusion that aid stimulated growth only in countries with good policies does not hold up to modest robustness checks, it does reflect a view widely held by aid practitioners that aid is much more likely to be effective in countries with a decent policy and institutional environment than in poorly run countries.
Concluding this second point of my comments, I agree that in this respect – to the extent that there is too often a question of incorrect and incomplete specification search – professor Bulte’s criticism is – only partly – justified. However, taking into account my additional observations in this part of my comments, I concur with Dani Rodrik’s comment on this subject. ‘In particular I believe in the need for both cross-country regressions and detailed country studies. Any cross-country regression giving results that are not validated by case studies needs to be regarded with suspicion. But any policy conclusion that derives from a case study and flies in the face of cross-national evidence needs to be similarly scrutinized. Ultimately, we need both kinds of evidence to guide our views of how the world works.7
Next, I come to a number of what I consider to be mistakes in professor Bulte’s column. First, linear models. Many models using cross-country analysis, including the most important aid-growth studies, include non-linearities. In fact, the relevant relations are linearized by introducing quadratic terms, but the phenomenon or explanatory variable describes a non-linear effect. In a recent Central Planning Bureau study using cross-country data even a cubic variable is instrumental.8
Secondly, no, it is not realistic, it is even not correct to think the effect of institutions (or an abundance of national resources) on economic growth is the same in Sudan as in Canada or Peru. But it is entirely correct to assume that in each of these countries the presence of such institutions or national resources is highly relevant for economic growth. The actual effect of such presence on growth can be estimated by multiplying the relevant regression coefficients, found in the cross-country regression, with the observed values of the corresponding explanatory variables (proxies for institutions or natural resources, for example) for Sudan, Canada and Peru, respectively. It is even possible and correct to distinguish between the presence and the functioning of institutions. This can be done by introducing a set of indicators of quality of institutions in the cross-country regression analysis and applying the procedure just mentioned.9
Thirdly, cross-country studies do not in general suffer from limited data availability. On the contrary, in most of such studies the number of observations is much larger than in those using time series analysis.
Fourthly, is it indeed much more difficult than documenting correlations, to prove that, for example, good health or secure property rights do contribute to higher incomes, rather than the other way around? In what sense or to what extent is such a proof more difficult? After all such a proof will consist of a number of regressions and tests for robustness and causality, on the basis of which a null hypothesis is either rejected with a probability of 1,5 or 10 percent that it is nevertheless true (type 1 error), or accepted with the same probabilities that an alternative hypothesis is true (type 2 error). In other words, such a proof is also a statistical proof, consisting of regressions and correlations, as well as a number of probability tests and nothing more. To what extent is it more difficult than ‘documenting correlations’? Probably, because for this purpose one needs more statistical information.
More statistics; but then how about professor Bulte’s reminder of Benjamin Disraeli’s ‘bon mot’ about ‘lies, damned lies and statistics’? Disraeli was a great statesman and a very clever, witty and astute politician. He was, however, not much interested in and even afraid of figures (statistics). In 1852 he even wanted to decline Lord Derby’s invitation to become for the first time a member of the cabinet as Chancellor of the Exchequer, because it was a branch of government of which he had no knowledge. In his own words when he presented his first budget, ‘My own knowledge on the subject is of course, recent. I was not born and bred a Chancellor of the Exchequer’. Lord Derby could persuade him to accept the Chancellorship by telling him ‘They give you the numbers’. Disraeli was in fact not at all at ease with statistics. This comes to the fore in one of his numerous letters where he mentions his meeting with ‘the deputies of the Statistical Congres’, describing them as ‘a strange gathering of hideous men with bald heads and all wearing spectacles’. Disraeli did not shrink back from extravagant flattery and (little?) white lies. He is known as a man ‘… who at least half believes the fantasies he manipulates’.10 However, he was also a pragmatist. This implies that although considering statistics indeed the superlative degree of lies, Disraeli, on the other hand, used statistics as a useful instrument for exposing white lies and inaccuracies of his opponents; just as it suited him. Are there any lessons to learn from Disraeli’s ‘bon mot’ on statistics, now one is acquainted with some more background information? Surely, two at least. First, Disraeli may be an unreliable source of doubt on the true value of statistics and statistical analysis. Secondly, when properly used, statistics and statistical analysis are excellent instruments for finding the truth; when not properly used they can be worse than lies.
Finally and by the way, personally I am looking forward to future publications by Collier, Easterly, and Sachs.
Footnotes
- 1. Easterly, W. (ed) (2008) Reinventing Foreign Aid. The MIT Press.
- 2. Roodman, D. (2007) The anarchy of numbers: Aid, development and cross-country empirics. The World Bank Economic Review. 21(2): 255-277.
- 3. Radelet S. and Levin, R. (2008) ‘Can We Build a Better Mousetrap? Three New Institutions Designed to Improve Aid Effectiveness’. In Easterly, W. (ed) Reinventing Foreign Aid. MIT Press.
- 4. See for instance Clemens, M., Radelet S. and Bahvani R. (2004). Counting Chickens when they Hatch: The Short-Term Effects of Aid on Growth. Center for Global Development Working Paper 44.
- 5. Rajan A. and Subramanian, A. (2005) Aid and Growth: What Does the Cross-Country Evidence Really Show? IMF Working Paper 05/127, International Monetary Fund.
- 6. Burnside C. and Dollar, D. Aid, politics, and growth. American Economic Review. 90(4): 847-868.
- 7. Rodrik, D. (2007) One Economics, Many Recipies: Globalization, Institutions and Economic Growth. Princeton University Press.
- 8. François, J. and Rojas-Romagosa, H. (2008) Reassessing the relationship between inequality and development. CPB Discussions Paper No. 107.
- 9. See for instance Kaufmann D. and Kraay, A. 2008. Governance Indicators: Where Are We, Where Should We Be Going? The World Bank Research Observer 23(1):1-30.
- 10. Schama, S. (2003) A History of Britain 3. BBC Worldwide Ltd.