The starting point for development evaluations should be how the recipients of development assistance experience change, rather than the set perspectives of the evaluators. The participatory development assessment (PDA) methodology is designed to involve recipients in evaluations.
In the social sciences and humanities, the positivist assumption of objectivity in the process of knowledge creation is gradually being abandoned. Researchers no longer claim to have the monopoly on what is ‘true’ and what is ‘false’. The attention that is given to indigenous knowledge is growing, and a range of new methodologies that borrow elements from a variety of evaluation approaches, are now being developed to formulate hypotheses and test assumptions.
Philosopher Karl Popper’s theory of ‘three worlds of knowledge’ is gradually being replaced by more complex combinations of knowledge systems. What is true for systems thinking in general is certainly true for knowledge systems (see Bob Williams, ‘Bucking the system’, The Broker 11). Making sense of current and recent history is a subjective, value-driven activity, both for lay people and scientists. Back in 1928, the sociologist William Thomas formulated a theorem, which states that ‘if men define situations as real, they are real in their consequences’. His proposition has become popular among postmodern movements within the social sciences and humanities. Judging what is true or false, good or bad, useful or useless, beautiful or ugly, is a cultural process, and full of specific pathways through time and space.
All words and concepts are value-laden, and understanding them requires constant interpretation and reinterpretation. ‘Development’ has become a vague concept, and ‘evaluating development’ has become an exercise in judging norms and values. It is therefore useful to examine how subjectivity entered the debate on evaluating development, and how it has gained in popularity among researchers. In ‘Evaluation evolution’ (The Broker 8), Otto Hospes attempted to classify three evaluation approaches to development, one of which was ‘complexity evaluation’. Here, we go one step further, and look at the promise of involving recipients in the assessment of the effects and impacts of the development assistance they have received. We also examine the current shift from ‘livelihoods’ to ‘well-being’ approaches, and the growing interest in ‘deep democracy’, and what these could mean for evaluation practice. See ‘Civic-driven change’ (The Broker 10) and ‘Well-being’ (The Broker 12).
The development industry is probably one of the most evaluated professional fields. Evaluations primarily involve project and programme assessments, and are undertaken by a host of researchers and consultants. More comprehensive evaluations assess the impacts of development interventions on particular sectors such as education or water supply and sanitation within a country or region, or of development approaches such as microfinance.
Participation in development evaluations
Among development and evaluation practitioners, ‘stakeholders’ participation’ has become a key phrase. Robert Chambers of the Institute of Development Studies (IDS), UK, has led the way in increasing the attention given to the poor in development and evaluation approaches. Evaluations have thus shifted away from purely ‘technocratic’ and expert-oriented towards stakeholder-inclusive and participatory assessments. By the 1990s, participatory approaches had become accepted practice, at least on paper. But some observers began to question how they were being applied, and especially the lack of attention paid to power structures and the added learning obstacles. This successful concept became blurred by many alternative interpretations and strayed from the original intention. The same happened with the overarching concept of ‘participation’.
This article describes a participatory development assessment (PDA) methodology developed by a team of researchers from the Netherlands, Ghana and Burkina Faso in an attempt to develop a more convincing approach to participatory evaluation. It is a joint effort by the Amsterdam Institute for Metropolitan and International Development Studies (AMIDSt) at the University of Amsterdam, the University for Development Studies in Tamale, Ghana, and Expertise pour le Développement du Sahel (EDS), Ouagadougou, Burkina Faso. The initiative is funded by three Dutch development organizations – the interchurch organization for development cooperation (ICCO), Woord en Daad and Prisma.
The PDA approach
Unlike many other development evaluation methods, where the starting point is a project, programme or sector, PDA is an ‘upside-down’ or reverse approach. It seeks to draw out the collective ‘experience of change’ by the population of an area, and the total set of interventions, including those regarded as ‘development initiatives’, that people think have played a role in their experience of change. In these assessments, it is not the expert evaluators who assess the changes and the effectiveness of development initiatives, but representatives of the local population. Although outside experts organize and facilitate workshops where local people can take stock of their experiences and assist in analyzing the findings, PDA is intended to be a self-help evaluation tool that can be used by any local agency.
During the workshops, the participants share their subjective understanding and judgements of development initiatives and their impacts on the process of change in their communities. Three types of ‘research area’ are examined: those where funding agencies have been active in the past and are continuing their assistance to the local communities in the future; those where funding agencies have been active and have recently stopped their assistance to communities in the area; and those where no funding agencies have been active in the past (Dirk-Jan Koch refers to these as ‘development orphans’ and ‘blind spots’).
The research team has recently tested the PDA approach in northern Ghana and southern Burkina Faso, areas where the three PDA funding agencies have long been active. Typical research areas are neighbourhoods in and around small administrative centres, with between 20,000 and 50,000 inhabitants spread over several villages. Workshops in a particular research area bring together approximately 60 people. These include 10–15 ‘officials’ representing local government departments, NGOs, chiefs and religious and social leaders, as well as 45–50 people from the villages, with a balanced representation of young and old, male and female, literate and illiterate, and socio-cultural groups such as Muslims, Christians of various denominations and indigenous religions. The intention is to gather a diverse group of people from each research area and to work with them in various subgroups that reach in-depth judgements of the changes they have experienced. Later, the judgements of these different groups can be compared – for example, those of men versus women, officials versus villagers, old versus young – and any differences in opinion can be further explored during subsequent workshop sessions. Based on their extensive network in the region, UDS-Tamale and EDS-Ouagadougou select workshop participants using these guidelines.
A PDA workshop takes place over three days at a location where accommodation and meals are provided so that the participants do not have to return home in the evening. During the first day the participants present and discuss their own development stories, in separate groups of relatively old and relatively young women, and of relatively old and relatively young men. Local officials take part in these discussions, often in the group of relatively old men.
Following Anthony Bebbington’s useful and influential approach, the participants discuss the changes they have experienced in six ‘domains’ of their lives: in the natural environment; the physical environment; human capabilities (education levels, health status); economic facilities; social capabilities; and cultural institutions. For each of these domains, the participants compare their current situation with that of their fathers or mothers when they were the same age. Thus the groups articulate the changes that have taken place over the past 25–30 years. They assess the changes as ‘positive’ or ‘negative’, and then qualify these assessments by adding negative aspects of the changes they considered to be positive, and vice versa. Once they have concluded these assessments, the groups list the most important events that have occurred over the last three decades.
The groups continue to discuss their perceptions of wealth and the attributes typical of the research area that determine whether someone is considered to be very rich, rich, average, poor or very poor. In addition, the participants are asked to fill in a questionnaire with questions about themselves, their parents, siblings and children. Their responses can be used as additional material that can be compared with those of their next of kin. The participants complete these questionnaires over the three days of the workshop. Those who can’t are helped by those who can.
On the second day, the participants work in separate groups. The local officials form one group so that they can express their views without influencing the villagers. The other participants are split up into three to five subgroups. The day begins with a stocktaking of all development initiatives that have been launched in the research area over the past 30 years. The participants list the name of the initiative, the sector in which it took place, the initiating agency or agencies, the financial donor(s), the period during which the initiative was ‘active’ and other relevant details. In practice, the initiating agencies can be divided into six main groups: government agencies, faith-based NGOs (including the development branches of churches and mosques), non-faith-based NGOs, private sector agencies (such as private banks or telecom companies) and local private initiatives.
After this stocktaking, the groups are split into male and female subgroups that then assign values to each ‘project’ (‘intervention’ or, better, ‘initiative’). First they rank the projects on the basis of their usefulness and actual impact they have had on peoples’ lives. Subsequently, each initiative is assigned to one of the following five categories:
- relatively new and still too early to say anything certain about impact;
- very much disliked and should never have been launched;
- looked good on paper but produced very few outputs, or had a negligible impact;
- some visible or tangible outputs, but not sustainable; and
- lasting positive impact (‘successful initiatives, also in the medium and long term’).
The groups could further differentiate the initiatives with a lasting positive impact into those that reached many people, and those that affected the lives of only a few.
On the third day, the subgroups formed at the start of the second day select the five ‘best’ and five ‘worst’ initiatives from the long list they compiled the previous day. For each of the five best initiatives they decide which wealth class benefited the most and which the least. They do this for each of the projects by distributing ten stones among the five wealth classes distinguished on day one. In the most recent workshops, the participants also tried to attach values to the distribution of benefits immediately after a ‘project’ had ended. By repeating the exercise in the future, it should be possible to compare any shift in opinion about a successful initiative over time.
The PDA research: initial results
The initial results of three of the six PDA evaluations held to date are now available, and have been sent to the communities in Ghana and Burkina Faso who participated in the workshops. In the following we provide a few highlights, focusing on the workshop held in the village of Sandema in northern Ghana.
The assessment of perceived ‘changes’ in Sandema revealed a rich diversity of opinions and attitudes. This reflected a multitude of subjective views of the changes that have occurred as mixtures of ‘good’ and ‘bad’. However, underlying this diverse spectrum of opinions some clear general features can be identified. Differences in opinion are apparent between men and women, between old and young, and across village communities. Assigning values to the impacts of change involves a process of negotiation among many people who occupy different positions in society. For individuals as well as for groups, this often results in ‘yes, but ...’ responses in the case of changes that are valued as mostly positive, and ‘no, although ...’ responses in the case of changes that are judged as mostly negative. There is a strong cultural component in assigning values, and a historical path-dependency in judgements. People tend to judge certain changes in the context of their own experiences, or those of their ancestors. It is not always clear what is fact, fiction or myth.
But, as Thomas’s theorem states, what is important is that people’s opinions and behaviours are based on these mixtures of what they regard as relevant truths, even if these are obviously ‘wrong’ or ‘distorted’. Evaluations often conclude that x% of development projects have failed, or y% have been successful, judgments that often are picked up and magnified by the media, public opinion makers and political entrepreneurs. However, the truth can not be captured in such simplistic conclusions and villagers and local officials in areas such as northern Ghana and Burkina Faso will brand them as simplistic generalizations or distortions of the truth as they see it. The organization of this workshop using Bebbington’s ‘capitals and capabilities’ approach proved to be very useful indeed. Separating ‘changes in social capabilities’, which include ‘changes in political power’, from ‘opinions about cultural change’, appeared to be enriching, although with obvious misunderstandings between participants and facilitators about the difference between ‘social’ and ‘cultural’.
In Sandema the workshop participants assessed the success or failure of a total of 341 development initiatives. The graph above illustrates the combined results of the values attached to the ‘best’ initiatives (55 of them) and their impacts on the five wealth classes. It shows how a basically qualitative approach can yield rather robust quantitative results. The results probably come close to intuitive assessments of those who are familiar with the area, but the differences between the ‘types of development agent’ are subtle in degree, and should discourage us from making simplistic judgements.
It seems that many Ghanaians, whether literate or not, are experts in the subtleties of complexity thinking.
The authors wish to thank the following for their inputs on earlier drafts of this article: Martha Lahai (University of Amsterdam); Professor David Millar, Professor Saa Dittoh and Dr Richard Yeboah, University for Development Studies (UDS), Ghana; Kees van der Geest (Amsterdam Institute for Metropolitan and International Development Studies, AMIDSt); Wouter Rijneveld (Woord en Daad); and Dieneke de Groot (ICCO). We also thank the many Ghanaian, Burkinabé and Dutch facilitators, participants, advisers and reviewers for their valuable support.
International Initiative for Impact Evaluation, Designing impact evaluations: different perspectives, 3ie Working Paper 4, July 2009. With contributions from Robert Chambers, Dean Karlan, Martin Ravallion, and Patricia Rogers.
Photo credit main picture: private collection
Thomas W. and Thomas, D. (1928) The Child in America: Behavior Problems and Programs, Knopf. See also Merton, R.K. (1995) The Thomas theorem and the Matthew effect. Social Forces 72(2): 379-424.
See also Fowler, A. (2008) ‘Connecting the dots’. The Broker 7.
The Deep Democracy debate was started in The Broker 10 (‘Civic-driven change’), while the well-being debate was summarized by Romesh Vaitilingam in ‘Well-being: A new development concept’ (The Broker 12).
For an overview of a recent debate in the Netherlands about the ins and outs of Measuring Results for Development visit www.dprn.nl. In a recent Letter to Parliament’, the Netherlands Minister for Development Cooperation, Bert Koenders, wrote that ‘Development cooperation is among the most researched and evaluated policy domains in the Netherlands (Ontwikkelingssamenwerking behoort tot de meest onderzochte en geëvalueerde beleidsterreinen in Nederland, ‘Modernisering draagvlak ontwikkelingssamenwerking’; 11 May 2009). Classic texts, which were the basis of a lot of ‘evaluating development’ exercises with a rather technocratic and expert-driven approach, include Casley, D.J. and Kumar, K. (1992) The Collection, Analysis, and Use of Monitoring and Evaluation Data. World Bank.
See, for example, Hulme, D. (2000) Impact assessment for microfinance: Theory, experience and better practice. World Development, 28(1): 79–98. For interesting examples in many fields, see www.iaia.org of the International Association for Impact Assessment (based in North Dakota, US). They define ‘impact assessment’ basically as an ex-ante activity (‘Impact assessment, simply defined, is the process of identifying the future consequences of a current or proposed action’), but there are many ex-post lessons as well. With PDA we use the word assessment in an ex-post way: people assess the past development trajectory, and the initiatives of the agencies involved. An interesting example of a combination of ex-ante and ex-post impact approaches, based on the experiences of GTZ, is given by Douthwaite, B., Kuby, Th., van de Fliert, E. and Schulz, S. (2003) Impact pathway evaluation: An approach for achieving and attributing impact in complex systems. Agricultural Systems, 78(2): 243–265.
Robert Chambers’ most influential publications include: Rural Development: Putting the Last First (Longman, 1983); The origins and practices of participatory rural appraisal, World Development, 22(7), 1994; Poverty and livelihoods: Whose reality counts? Environment and Urbanization, 7(1), 1995; and Whose Reality Counts: Putting the Last First (IT, 1997). Together with others he was involved in the famous World Bank-funded exercise ‘Voices of the Poor’ (see D. Narayan, R.Chambers, M.H. Shah and P. Petesh (2000) Voices of the Poor: Crying out for Change. There is a host of related methods, with names like ‘sondeo’, ‘rapid rural appraisal’, ‘participatory appraisal’, ‘inclusive assessment’, and so on. One of the first texts was Hildebrand, P. (1981) Combining disciplines in rapid appraisal: The sondeo approach. Agricultural Administration, 9(6). This has inspired us to do ‘sondeos’, when we started our research programme to support the Arid and Arid Lands Development Programme in Kenya in 1982. Perhaps the most useful summary of these approaches is Schönhuth, M. and Kievelitz, U. (1994) Participatory Learning Approaches, Rapid Rural Appraisal, Participatory Appraisal: An Introductory Guide, GTZ Verlagsgesellschaft: Schriftenreihe der GTZ, No. 248. Recently, Chambers has further shifted the approach to what he calls PLA: Participatory Learning and Action (e.g. Participatory Workshops: A Sourcebook of 21 Sets of Ideas and Activities, London: Earthscan, 2002) and From PRA to PLA and pluralism: Practice and theory, in Reason P. and Bradbury, H. (2007) The Sage Handbook of Action Research: Participatory Inquiry (London: Sage, pp.297-318).
See Scriven, M. (1980) The Logic of Evaluation. Inverness, CA: Edgepress;
Greene, G.B. (1988) Stakeholder participation and utilization in program evaluation, Evaluation Review, 12(2): 91-116;
Guba, E.G. and Lincoln, Y.S. (1989) Fourth Generation Evaluation. London: Sage;
Garaway, G.B. (1995) Participatory evaluation, Studies in Educational Evaluation, 21-1: 85-102;
Keough, N. (1998) Participatory development principles and practice: Reflections of a Western development worker, Community Development Journal 33-3: 187-196;
Jackson, E.T. and Y. Kassam (1998) Knowledge Shared: Participatory Evaluation in Development Cooperation, West Hartford, CT: Kumarian Press.
Themessl-Huber, M.T. and Grutsch, M.A. (2003) The shifting focus of control in participatory evaluations, Evaluations 9-1: 92-111;
Holte-McKenzie, M., S. Forde, and S. Theobald (2006) Development of a participatory monitoring and evaluation strategy, Evaluation and Program Planning, 29-4: 365-376;
Forss, K., S. Kruse, S. Taut and E. Tenden (2006) Chasing a ghost? An essay on participatory evaluation and capacity development, Evaluation 12: 128-144.
Obure, J. (2008) Participatory Monitoring and Evaluation: A Meta-Analysis of Anti-Poverty Interventions in Northern Ghana, MSc thesis, University of Amsterdam (unpublished).
An interesting recent Dutch PhD study about participatory monitoring experiences is: Guijt, I. (2008) Seeking Surprise. Rethinking Monitoring for Collective Learning in Rural Resource Management, Wageningen University and Research Centre.
A good overview is presented in Estrella M. (ed) (2000) Learning from change. Issues and Experiences in Participatory Monitoring and Evaluation, Participation in Development Series, London: ITDG. See also Roche, C. (1999) Impact Assessment for Development Agencies, Oxford: Oxfam and Novib.
A classical text is Oakley, P. (1991) Projects with People: The Practice of Participation in Rural Development. Geneva: ILO/WEP. Recent critiques include:
Cleaver, F. (1999) Paradoxes of participation: Questioning participatory approaches to development, Journal of International Development, 11-4: 597-612;
Kapoor, I. (2002) The devil’s in the theory: A critical assessment of Robert Chambers’ work in participatory development, Third World Quarterly, 23-1: 101-117;
Platteau, J-P. and Abraham. A. (2002) Participatory development in the presence of endogenous community imperfections, Journal of Development Studies, 39-2: 104-136;
Cornwall, A. (2003) Whose voices? Whose choices? Reflections on gender and participatory development, World Development, 31-: 1325-1342; and
Williams, G. (2004) Evaluating participatory development: tyranny, power and (re)politicisation, Third World Quarterly, 25-3: 557-578.
A special branch is PTD, Participatory Technology development. For an evaluative overview see: Joss, S. (2002) Toward the public sphere: Reflections on the development of participatory technology assessment, Bulletin of Science, Technology & Society, 22-3: 220-231.
A recent textbook with a lot of useful insights about ‘participatory research’ is Laws, S. et al. (2003) Research for Development. A Practical Guide, London: Sage, for Save the Children.
This approach – described as ‘toppled or overturned’ – is used by Sjoerd Zaanen and Ton Dietz in an article for the Netherlands Yearbook on International Cooperation 2008: Assessing interventions and change among presumed beneficiaries of ‘development’: A toppled perspective on impact evaluation (2009, forthcoming).
As in the various contributions about ‘well-being’ in the recent issues of The Broker.
See D.J. Koch (2009) Aid from International NGOs: Blind Spots on the Aid Allocation Map. PhD dissertation, Radboud University Nijmegen, Routledge. D.J. Koch (2007) Uncharted territories, The Broker 3.
In September 2008 three workshops focused on the first type of areas: in Langbensi in the eastern part of Northern Region Ghana, in Sandema, Upper East Region Ghana, and Tô, in southern Burkina Faso. In March 2009 these were followed by three workshops about the second type of area: in Nandom in Upper West Region, Ghana, Lasei Toulu in the western part of Northern Region, Ghana, and in Silly, southern Burkina Faso.
Following an adjusted version of Anthony Bebbington’s approach: Bebbington, A. (1999) Capitals and capabilities: A framework for analyzing peasant viability, rural livelihoods and poverty. World Development, 27-12: 2021–2024.
When a participant hands in the forms, each form is checked in a personal meeting with the facilitator who coordinates this exercise.
We hope that all ‘open source reports’ of this research programme will be available in September 2009, at www.participatorydevelopmentassessment.nl. The Sandema report is available at https://home.medewerker.uva.nl/a.j.dietz (publications page, under 3.3). The Langbensi and Tô reports, and the student reports, will also be available soon. Martha Lahai recently finalized her MSc thesis at IDS on Participatory Evaluation: Perceptions of Local People on Long-Term Impact of Development Interventions in Northern Ghana (Amsterdam, 2009). She compared the findings of part of the Langbensi study with findings at a more local level, repeating the exercise there, to identify differences between ‘regional’ and ‘village-level’ approaches.
As developed by Bebbington, see note 19.
For a recent discussion of quantitative and/versus qualitative methods of impact evaluation (by Sabine Garbarino and Jeremy Holland, March 2009, for DFID and the GSDRC), View PDF.
See Fowler, A. (2008) Connecting the dots. The Broker 7.