About two weeks ago, I went to a seminar in Wageningen about methodologies for impact evaluation. Impact studies have held my specific attention since I started working in the development sector. The interesting thing about impact studies is that they try to answer the main question I think all NGOs and governments should ask themselves: do my projects have any positive sustainable effect? I think that everyone working in this field wants to know if our work actually leads to something positive.
During the seminar, three recent impact studies were discussed. I will not discuss all of them here (if you were really interested, you would have been in the audience anyway), but move right to a discussion that started after a presentation by Prof. Menno Pradhan. Pradhan had conducted an impact evaluation of a prestigious development project in Indonesia. The impact evaluation used an econometric approach and made use of randomized control groups. Since this method is still pretty rare in the development sector, it immediately caught my attention. During his research, Pradhan gathered data from the control group and the target group during three rounds in a period of two years. The conclusion he was able to draw based on his research was that he could not identify any lasting impact! No change in income, no change in health standards etc. This is, of course, a shocking conclusion that no NGO wants to hear.
What surprised me even more were the reaction from the audience, in particular the reaction of some of the gurus. One of them stated that this was the problem with quantitative methods of impact assessment, and that in this case it would have been better if more qualitative methods had been used (read: just ask people if they liked the project). Furthermore, the guru stated that the impact assessment was probably conducted too soon (after two years) and therefore impact could not yet be identified. I was really surprised … do people really think that if a project has had no single effect after two years, then there will suddenly be a big improvement five years later? Or do they just not want to accept the conclusions?
Impact studies (and, in fact, all evaluations) are carried out so we can learn from them. What I noticed from the audience reactions, however, is that it seems that people do not really want to learn. You cannot learn from an impact study if you simply deny the conclusions. If you really want to learn, you accept the conclusions and try to find out why there was no single impact. You should even be willing to stop your project, or even your whole programme, if it turns out there is no social impact at all, or even worse, a negative one.
Therefore, my question to you, my fellow tree huggers, is this: what will you do if it turns out that the work you are doing has no positive social impact at all? Would you be willing to give up your job?