Subjective truths – Participatory development assessment

The starting point for development evaluations should be how the recipients of development assistance experience change, rather than the set perspectives of the evaluators. The participatory development assessment (PDA) methodology is designed to involve recipients in evaluations.

In the social sciences and humanities, the positivist assumption of objectivity in the process of knowledge creation is gradually being abandoned. Researchers no longer claim to have the monopoly on what is ‘true’ and what is ‘false’. The attention that is given to indigenous knowledge is growing, and a range of new methodologies that borrow elements from a variety of evaluation approaches, are now being developed to formulate hypotheses and test assumptions.

Philosopher Karl Popper’s theory of ‘three worlds of knowledge’ is gradually being replaced by more complex combinations of knowledge systems. What is true for systems thinking in general is certainly true for knowledge systems (see Bob Williams, ‘Bucking the system’, The Broker 11). Making sense of current and recent history is a subjective, value-driven activity, both for lay people and scientists. Back in 1928, the sociologist William Thomas formulated a theorem, which states that ‘if men define situations as real, they are real in their consequences’. His proposition has become popular among postmodern movements within the social sciences and humanities. Judging what is true or false, good or bad, useful or useless, beautiful or ugly, is a cultural process, and full of specific pathways through time and space.

All words and concepts are value-laden, and understanding them requires constant interpretation and reinterpretation. ‘Development’ has become a vague concept, and ‘evaluating development’ has become an exercise in judging norms and values. It is therefore useful to examine how subjectivity entered the debate on evaluating development, and how it has gained in popularity among researchers. In ‘Evaluation evolution’ (The Broker 8), Otto Hospes attempted to classify three evaluation approaches to development, one of which was ‘complexity evaluation’. Here, we go one step further, and look at the promise of involving recipients in the assessment of the effects and impacts of the development assistance they have received. We also examine the current shift from ‘livelihoods’ to ‘well-being’ approaches, and the growing interest in ‘deep democracy’, and what these could mean for evaluation practice. See ‘Civic-driven change’ (The Broker 10) and ‘Well-being’ (The Broker 12).

Evaluating development

The development industry is probably one of the most evaluated professional fields. Evaluations primarily involve project and programme assessments, and are undertaken by a host of researchers and consultants. More comprehensive evaluations assess the impacts of development interventions on particular sectors such as education or water supply and sanitation within a country or region, or of development approaches such as microfinance.

Participation in development evaluations

Among development and evaluation practitioners, ‘stakeholders’ participation’ has become a key phrase. Robert Chambers of the Institute of Development Studies (IDS), UK, has led the way in increasing the attention given to the poor in development and evaluation approaches. Evaluations have thus shifted away from purely ‘technocratic’ and expert-oriented towards stakeholder-inclusive and participatory assessments. By the 1990s, participatory approaches had become accepted practice, at least on paper. But some observers began to question how they were being applied, and especially the lack of attention paid to power structures and the added learning obstacles. This successful concept became blurred by many alternative interpretations and strayed from the original intention. The same happened with the overarching concept of ‘participation’.

This article describes a participatory development assessment (PDA) methodology developed by a team of researchers from the Netherlands, Ghana and Burkina Faso in an attempt to develop a more convincing approach to participatory evaluation. It is a joint effort by the Amsterdam Institute for Metropolitan and International Development Studies (AMIDSt) at the University of Amsterdam, the University for Development Studies in Tamale, Ghana, and Expertise pour le Développement du Sahel (EDS), Ouagadougou, Burkina Faso. The initiative is funded by three Dutch development organizations – the interchurch organization for development cooperation (ICCO), Woord en Daad and Prisma.

The PDA approach

Unlike many other development evaluation methods, where the starting point is a project, programme or sector, PDA is an ‘upside-down’ or reverse approach. It seeks to draw out the collective ‘experience of change’ by the population of an area, and the total set of interventions, including those regarded as ‘development initiatives’, that people think have played a role in their experience of change. In these assessments, it is not the expert evaluators who assess the changes and the effectiveness of development initiatives, but representatives of the local population. Although outside experts organize and facilitate workshops where local people can take stock of their experiences and assist in analyzing the findings, PDA is intended to be a self-help evaluation tool that can be used by any local agency.

During the workshops, the participants share their subjective understanding and judgements of development initiatives and their impacts on the process of change in their communities. Three types of ‘research area’ are examined: those where funding agencies have been active in the past and are continuing their assistance to the local communities in the future; those where funding agencies have been active and have recently stopped their assistance to communities in the area; and those where no funding agencies have been active in the past (Dirk-Jan Koch refers to these as ‘development orphans’ and ‘blind spots’).

The research team has recently tested the PDA approach in northern Ghana and southern Burkina Faso, areas where the three PDA funding agencies have long been active. Typical research areas are neighbourhoods in and around small administrative centres, with between 20,000 and 50,000 inhabitants spread over several villages. Workshops in a particular research area bring together approximately 60 people. These include 10–15 ‘officials’ representing local government departments, NGOs, chiefs and religious and social leaders, as well as 45–50 people from the villages, with a balanced representation of young and old, male and female, literate and illiterate, and socio-cultural groups such as Muslims, Christians of various denominations and indigenous religions. The intention is to gather a diverse group of people from each research area and to work with them in various subgroups that reach in-depth judgements of the changes they have experienced. Later, the judgements of these different groups can be compared – for example, those of men versus women, officials versus villagers, old versus young – and any differences in opinion can be further explored during subsequent workshop sessions. Based on their extensive network in the region, UDS-Tamale and EDS-Ouagadougou select workshop participants using these guidelines.

Workshop agenda

A PDA workshop takes place over three days at a location where accommodation and meals are provided so that the participants do not have to return home in the evening. During the first day the participants present and discuss their own development stories, in separate groups of relatively old and relatively young women, and of relatively old and relatively young men. Local officials take part in these discussions, often in the group of relatively old men.

Following Anthony Bebbington’s useful and influential approach, the participants discuss the changes they have experienced in six ‘domains’ of their lives: in the natural environment; the physical environment; human capabilities (education levels, health status); economic facilities; social capabilities; and cultural institutions. For each of these domains, the participants compare their current situation with that of their fathers or mothers when they were the same age. Thus the groups articulate the changes that have taken place over the past 25–30 years. They assess the changes as ‘positive’ or ‘negative’, and then qualify these assessments by adding negative aspects of the changes they considered to be positive, and vice versa. Once they have concluded these assessments, the groups list the most important events that have occurred over the last three decades.

The groups continue to discuss their perceptions of wealth and the attributes typical of the research area that determine whether someone is considered to be very rich, rich, average, poor or very poor. In addition, the participants are asked to fill in a questionnaire with questions about themselves, their parents, siblings and children. Their responses can be used as additional material that can be compared with those of their next of kin. The participants complete these questionnaires over the three days of the workshop. Those who can’t are helped by those who can.

On the second day, the participants work in separate groups. The local officials form one group so that they can express their views without influencing the villagers. The other participants are split up into three to five subgroups. The day begins with a stocktaking of all development initiatives that have been launched in the research area over the past 30 years. The participants list the name of the initiative, the sector in which it took place, the initiating agency or agencies, the financial donor(s), the period during which the initiative was ‘active’ and other relevant details. In practice, the initiating agencies can be divided into six main groups: government agencies, faith-based NGOs (including the development branches of churches and mosques), non-faith-based NGOs, private sector agencies (such as private banks or telecom companies) and local private initiatives.

After this stocktaking, the groups are split into male and female subgroups that then assign values to each ‘project’ (‘intervention’ or, better, ‘initiative’). First they rank the projects on the basis of their usefulness and actual impact they have had on peoples’ lives. Subsequently, each initiative is assigned to one of the following five categories:

• relatively new and still too early to say anything certain about impact;
• very much disliked and should never have been launched;
• looked good on paper but produced very few outputs, or had a negligible impact;
• some visible or tangible outputs, but not sustainable; and
• lasting positive impact (‘successful initiatives, also in the medium and long term’).

The groups could further differentiate the initiatives with a lasting positive impact into those that reached many people, and those that affected the lives of only a few.

On the third day, the subgroups formed at the start of the second day select the five ‘best’ and five ‘worst’ initiatives from the long list they compiled the previous day. For each of the five best initiatives they decide which wealth class benefited the most and which the least. They do this for each of the projects by distributing ten stones among the five wealth classes distinguished on day one. In the most recent workshops, the participants also tried to attach values to the distribution of benefits immediately after a ‘project’ had ended. By repeating the exercise in the future, it should be possible to compare any shift in opinion about a successful initiative over time.

The PDA research: initial results

The initial results of three of the six PDA evaluations held to date are now available, and have been sent to the communities in Ghana and Burkina Faso who participated in the workshops. In the following we provide a few highlights, focusing on the workshop held in the village of Sandema in northern Ghana.

The assessment of perceived ‘changes’ in Sandema revealed a rich diversity of opinions and attitudes. This reflected a multitude of subjective views of the changes that have occurred as mixtures of ‘good’ and ‘bad’. However, underlying this diverse spectrum of opinions some clear general features can be identified. Differences in opinion are apparent between men and women, between old and young, and across village communities. Assigning values to the impacts of change involves a process of negotiation among many people who occupy different positions in society. For individuals as well as for groups, this often results in ‘yes, but …’ responses in the case of changes that are valued as mostly positive, and ‘no, although …’ responses in the case of changes that are judged as mostly negative. There is a strong cultural component in assigning values, and a historical path-dependency in judgements. People tend to judge certain changes in the context of their own experiences, or those of their ancestors. It is not always clear what is fact, fiction or myth.

But, as Thomas’s theorem states, what is important is that people’s opinions and behaviours are based on these mixtures of what they regard as relevant truths, even if these are obviously ‘wrong’ or ‘distorted’. Evaluations often conclude that x% of development projects have failed, or y% have been successful, judgments that often are picked up and magnified by the media, public opinion makers and political entrepreneurs. However, the truth can not be captured in such simplistic conclusions and villagers and local officials in areas such as northern Ghana and Burkina Faso will brand them as simplistic generalizations or distortions of the truth as they see it. The organization of this workshop using Bebbington’s ‘capitals and capabilities’ approach proved to be very useful indeed. Separating ‘changes in social capabilities’, which include ‘changes in political power’, from ‘opinions about cultural change’, appeared to be enriching, although with obvious misunderstandings between participants and facilitators about the difference between ‘social’ and ‘cultural’.

In Sandema the workshop participants assessed the success or failure of a total of 341 development initiatives. The graph above illustrates the combined results of the values attached to the ‘best’ initiatives (55 of them) and their impacts on the five wealth classes. It shows how a basically qualitative approach can yield rather robust quantitative results. The results probably come close to intuitive assessments of those who are familiar with the area, but the differences between the ‘types of development agent’ are subtle in degree, and should discourage us from making simplistic judgements.

It seems that many Ghanaians, whether literate or not, are experts in the subtleties of complexity thinking.

The authors wish to thank the following for their inputs on earlier drafts of this article: Martha Lahai (University of Amsterdam); Professor David Millar, Professor Saa Dittoh and Dr Richard Yeboah, University for Development Studies (UDS), Ghana; Kees van der Geest (Amsterdam Institute for Metropolitan and International Development Studies, AMIDSt); Wouter Rijneveld (Woord en Daad); and Dieneke de Groot (ICCO). We also thank the many Ghanaian, Burkinabé and Dutch facilitators, participants, advisers and reviewers for their valuable support.

Related Resource

International Initiative for Impact Evaluation, Designing impact evaluations: different perspectives, 3ie Working Paper 4, July 2009. With contributions from Robert Chambers, Dean Karlan, Martin Ravallion, and Patricia Rogers. View PDF

Footnotes