News

Dancing to the tune: assessing aid effectiveness

Development Policy01 Jul 2010Nancy Okail

This story chronicles my involvement in the Organisation for Economic Co-operation and Development (OECD) monitoring survey on aid effectiveness in a North African country in 2006. The OECD monitoring survey was a tool designed to assess how aid was spent in this recipient country and measure progress in relation to the five dimensions of the Paris Declaration: ownership, alignment, harmonization, results and mutual accountability. I was a participant observer at one of the offices in the country’s Ministry of International Cooperation entrusted to conduct the survey.

They viewed me as a good vehicle for lifting the process and helping my new colleagues meet the OECD’s deadline. I entered the picture as a former member of the ministry and was there from the UK to do fieldwork as a doctoral candidate at a time when many people were struggling with the survey process.

But tensions soon started to rise. There was conflict, censorship and a seemingly impenetrable hierarchy, which initially made it difficult to offer constructive criticism or introduce changes into the process.

My aim as an action researcher was to introduce changes to the survey process. The survey was mainly based on quantitative indicators. It had a rigid structure and left little room for a qualitative analysis of the indicators. I was not convinced by the way the survey was set up, nor by the procedures we had to follow. My first reaction was to resist and question the entire process, but I faced other priorities, such as meeting the deadline and proving our proficiency to the OECD office in Paris.

Tears for fears

There was an immediate sense of urgency during the entire survey process. Everybody involved had to rush and struggle to move forward, and work faster. Ironically, no one had time to stop and wonder what this progress was that they were rushing towards. We were not expected to examine the value of the Paris Declaration’s five dimensions, and yet they represented what we were apparently progressing towards. Those who dared to question the process – and I was one of them – were threatened with exclusion.

When I first joined the team working on the survey, my boss called me into her office and asked me to close the door. I was surprised to find her distressed and even tearful. She explained that the process had put her under tremendous pressure, and she feared she was losing her grip on it. ‘This is so complex. I am struggling with the indicators,’ she said, ‘and need your help. I believe God has sent you to help us!’

To question or not to question, that is the question

I started working with a team of researchers on the survey, organizing workshops and conducting interviews with government officials and donor organizations according to the OECD guidelines. However, the more I was drawn into the process, the less convinced I was by its principles and foundations. I found the survey design, with its quantitative indicators, far removed from reality and could seldom measure the five dimensions of aid effectiveness accurately.

For example, alignment was measured by comparing Official Development Assistance according to donor budgets with recipient government data. If there were any discrepancies between the two sets of data, they were reconciled in the records to indicate progress. The rest of the dimensions were measured along the same lines, with little opportunity to explain the resulting data.

So I started questioning the process and demanded that we ask the OECD office to make changes, or at least use the little space allowed for qualitative analysis to express the critical views that had emerged in the workshops and interviews. My next meeting with my boss was nothing like the first. Gone was the flattery. Gone the kind words.

This time she did not ask me to close the door. She banged her hands on the desk. ‘What you are doing is unacceptable. You’re questioning too much. We have a deadline to meet and guidelines to follow.’

I was no longer God’s gift to the ministry, that much was clear. Suddenly all I was good for was making noise and hampering progress. This situation reminded me of something the French sociologist Pierre Bourdieu once wrote in his 1990 collection of essays In Other Words: ‘Belonging to a professional group brings into play an effect of censorship which goes far beyond institutional or personal constraints: there are questions that you don’t ask, and that you can’t ask.’

I was suddenly acutely aware of this censorship, and yet the conflict persisted as we carried on with our work on the survey. The more pressing the OECD deadlines, the more tension we felt at the office, until finally we submitted our first draft.

D is for docility

A few days later, the department head urgently called our team together. He explained that our draft had been shared with the minister and the OECD office in Paris. ‘We got a D!’ Everyone looked alarmed, none more than my boss. ‘We got a D? Why? What for?’ she asked. ‘We worked hard. We showed professionalism. How can they give us a D?’

The department head kindly clarified that the D was given for the country’s performance in relation to the indicators of the Paris Declaration survey, not on how we conducted the survey. My boss was visibly relieved. Luckily it was only the country that got a D, not her.

The irony could not have been more profound. The entire Paris process, which was supposed to enhance ownership, came down to this. All people were interested in was finishing assignments and getting the work done. The main concern was whether we followed the guidelines and procedures correctly or not. The D, it struck me, was for docility.

As I reflected on the situation, I was again reminded me of something I had once read. Martin Ruef and W. Richard Scott wrote in their 1999 essay A Multidimensional Model of Organizational Legitimacy that when outcomes are not guaranteed or predictable, legitimacy prefers to rely on the soundness and appropriateness of the procedures being followed. Operations cannot derive legitimacy in the field of aid based on highly unpredictable outcomes either. Thus, the focus on professional adherence to methodology, procedure and measurement is necessary to create legitimacy. But this reinforces the supremacy of procedure over outcome in gaining legitimacy within aid operations.

Mark C. Suchman exposes this paradox in his 1995 publication Managing Legitimacy: Strategic and Institutional Approaches. He explains that although we tend to celebrate effectiveness, we also go to great lengths to identify causes. ‘A hospital is unlikely to lose legitimacy simply because some patients die. However, it is quite likely to lose legitimacy if it performs involuntary exorcisms, even if all patients get well.’

But the paradox here is that if this is how aid policy acquires legitimacy, why is a results-based management (RBM) approach currently being used to measure aid effectiveness? If the above analysis were true, then the emphasis should be on process-based management. However, although the focus of aid seems to be geared towards outcome at present, in practice it is not.

RBM is inherently procedural and does not rely onresults. For example, a discussion in one of the survey workshops on RBM focused on how to train development professionals to follow the RBM approach, not on the desired results. RBM has become another sophisticated method that needs to be mastered, and those who master it become legitimate actors with authority.

It’s just an illusion

The process continued. The OECD results report from Paris invited us to provide our comments on the draft (in consultation with donor and government representatives). Accordingly, we organized a meeting with several government officials to whom we had sent the report in order to get their feedback. It was a high-level meeting, and as the chairman finished his opening speech about the importance of the survey, he invited our comments on the report.

He was met by a long silence. It lasted for minutes. People were bending their heads down and guiltily flipping through the report, which they had received a month before. It was like a room full of undergraduates who had not done their reading and were doing everything in their power to avoid drawing the professor’s attention. These were not undergraduates, though. These were adults gathered to discuss a government ownership process.

It was not really their fault, though. They were products of a bureaucratic government system. All their professional lives they had been told what to do and what not to do. Suddenly they were given a complex report and asked to comment on it. The OECD had succeeded in creating an illusion. They assumed that by engaging government officials to fill in a survey according to guidelines dictated from Paris – which respondents did not really understand – they would have actively participated in the process of enhancing aid effectiveness and increasing country ownership.

That was not the end of the journey. It continued for another five months. Fortunately, I was able to introduce some changes to the process, but only after a great deal of manoeuvring. Two things happened, however. In order to be heard I had to dance to the tune played by the powerful actors in this process. And I had to prove my intimate knowledge of all the concepts and techniques used in this process, such as RBM, mutual accountability and harmonization . Along the way, I learned to tone down my criticism. The final outcome of the process, I believe, legitimized the knowledge it produced.