A blind-spot mirror for evaluators
Over the past few years, Dirk-Jan Koch, Professor by Special Appointment of International Trade & Development Cooperation, has been working with a team on a multi-year research programme, which looks into the possible side effects of international development work. After 4 years of research, they have published a checklist that presents the 10 most common side effects of international development interventions. To determine whether these unintended effects are a blind spot of evaluation research, the team has studied all evaluations of the Directorate of International Research and Policy Evaluation (IOB ) of the Dutch Ministry of Foreign Affairs over the past 20 years. In this article Dirk-Jan Koch shares his experiences and most important findings.
I learn so much from recently graduated researchers. I have never learned what they can do: Text mining, for example. With programmes I had never heard of before, such as Python, these researchers can figure out things in a few days that would have taken me years. In this research, with Python we were able to go through 664 evaluation documents of the IOB from the past 20 years. I can hear you thinking: why would you want to do this at all? In our attempt to gain more insight into the unintended effects of international cooperation, the IOB archive presents a potential gold mine. Either the IOB evaluations would include data and insights on many of the unintended effects or – and this would be a highly interesting find as well – they would not, showing that these effects may perhaps be a blind spot in the evaluations. According to the OECD guidelines for international development evaluations, evaluators must look for unintended effects. As the IOB claims to follow these guidelines we – myself together with a team from the Center for Global Challenges of Utrecht University, including Jolynde Vis (the main text miner), Maria vd Harst, Elric Tendron and Joost de Laat – examined whether this is really the case.
10 side effects
In our previous research we identified 10 possible side effects of international cooperation. We looked at side effects at three levels – the individual level, the community level, and the national level – and formulated a ‘checklist’ of side effects.
At the individual (micro) level we found, for example, price effects. Through external help, certain items can become very cheap, which can have an effect on the individual consumer or producer. Food aid, for example, makes food much cheaper: that is good for consumers, but disadvantageous for local farmers who can no longer sell their products. External interventions, however, can also make items very expensive. For instance, if a large number of aid expats moves to a neighbourhood, the rent is likely to increase.
Moving on, migration effects were found at the community (meso) level. These effects can be observed when people move to an area where there is a successful development project. On the flip side, people might also have to resettle (forcibly or not) if development money is used to support a plantation project that requires the area to be vacated.
Governance effects can be identified on the national (macro) level. For example, the national elite can use aid money to strengthen their position.
Incidentally, some side effects are visible on multiple levels, such as flywheel effects. Flywheel effects constitute one of the two types of side effects that appear most often in the evaluations – the other being the so-called ‘leakage’ effects. Leakage effects are usually negative. For example, an evaluation of the Dutch water policy in Egypt found that the construction of a new water supply system led to the flooding of water tanks, which was then linked to health problems. Flywheel effects, by contrasts, are positive. In the case of flywheel effects, aid successes are emulated by other families, regions or countries. An evaluation of a business programme, for example, showed that the construction of a harbour unexpectedly led to an increase in tourism, because the construction had led to the expansion of the beach as well.
The blind spots
Having identified these 10 most common unintended effects, we revisited the IOB evaluation documents. The ‘text miners’ found that only 1 in 6 IOB evaluation documents pay attention to unintended effects. So, in 5 out of 6 cases unintended effects constitute a blind spot. Could this be because policy documents hardly mention unintended effects?
Whatever the reason, the IOB is – unfortunately – in good company. Neither the American nor the Norwegian colleague-evaluators paid more attention to the unintended side-effects and arrived at the same figures. With our research we wanted to gain more insight into the story behind these low numbers. Is there a pattern to be found in the types of side effects that evaluators identify or, conversely, fail to spot? Are side effects systematically investigated or do researchers encounter them by chance?
An interesting example that shows the nature of the IOB’s ‘blind spots’ and the impact of the side effects that are overlooked is ‘national backlash’. This unintended side effect, although much discussed in academic literature, was not addressed in any of the more than 600 examined evaluation documents. In Afghanistan, however, the impact of this effect is visible in the responses of local leaders to Western organisations that are standing up for democracy and women’s rights. While intentions are good, the Western efforts allow leaders to strengthen their own legitimacy by dismissing them as foreign interference and, at times, crack down on those issues even more vigorously. Unfortunately, these nationalist backlash effects are not included in Theories of Change (ToC) of development programmes, even though ToCs should normally also include alternative impact pathways. Since rarely a mention is made of these backlash effects in the ToCs, they also remain off the radar of evaluators.
Surprisingly, our research found that, of the 177 side effects found, more than 40% was positive. It could thus be inferred that international aid often turns out to have a bigger flywheel effect than expected. And for the critics who think that development cooperation encourages a lot of corruption: ten evaluation documents show that aid also has a surprisingly positive effect on the quality of governance. For example, provinces regularly copy reforms that were successfully implemented in other regions with the help of external aid interventions.
A blind-spot mirror
While positive side effects seem to be less invisible, the fact remains that only 1 in 6 evaluation documents pay attention to unintended effects – and that is far too little. If only 1 in 6 trucks had an extra mirror to make their blind spot as small as possible, a parliamentary debate would undoubtedly be requested. The checklist we have formulated can act as the much-needed blind-spot mirror for evaluation researchers to give them a wider field of vision. Evaluators frequently assert: ‘but I don’t know where to look for side effects, because policy documents do not say anything about them either’, as an explanation for not looking into this blind spot. Well, by assembling this blind-spot mirror, that problem is now overcome.
Yet, even equipped with good rear-view mirrors, a truck can still run people over. All road users must do their part and look ahead to avoid being hit by a truck. Continuing the metaphor, our list of the 10 types of unintended effects is not only an extra mirror meant for evaluators. It is also a good starting point to structurally include unintended effects through the different stages of the project cycle, from the development phase to implementation. Everyone has a role to play in covering the blind spots and tackle possible side effects more seriously within the development sector!
Dirk-Jan Koch is professor by special appointment at Radboud University in Nijmegen and works at the Ministry of Foreign Affairs. His research focus are the unintended effects of international cooperation. He writes in a personal capacity.
For more information on the 10 most common unintended effects of international cooperation, also watch this video.