Towards more meaningful evaluation of Aid for Trade investments

8 December 2011
ITC News

Measuring the impact of Aid for Trade (A4T) development programmes, and analysing how they might have been implemented differently to achieve better results, is fundamental. High-quality monitoring and evaluation (M&E) provides insights that can lead to programme enhancements or to more successful outcomes of similar projects elsewhere.

The Third Global Review of Aid for Trade organized by the World Trade Organization (WTO) in July was the most comprehensive and evidence-based review so far. Sessions attracted contributions from many of the most experienced A4T practitioners, policymakers and recipients who discussed a wide range of issues, including M&E.

Likewise, the recent OECD Monitoring Survey and the Independent Evaluation Report on aid effectiveness provided a rich database of experiences in nearly 80 countries and several bilateral and multilateral development assistance institutions. Interestingly, the key findings of these reviews are nearly identical, stressing the relevance of the principles of the 2005 Paris Declaration on Aid Effectiveness and the positive impact of stronger partner country ownership and leadership in managing development cooperation. They point to the importance of outcomes rather than inputs, call on assistance providers to speed up reform of their systems and practices, and underline the need to strengthen M&E capacity in partner countries.

It is worth noting that the global economic environment is currently such that some decline in official development assistance (ODA) is to be expected in the foreseeable future. There are growing concerns about how long it will take the major sources of ODA – traditional and emerging – to overcome recent slowdowns, mounting deficits and higher unemployment levels. The fact that non-Western economies have shown signs of slowing, particularly China and India, is not reassuring. These rather gloomy prospects make it all the more important for the international community to extract more value from every A4T dollar, euro or yen.
With these points in mind, how can M&E mechanisms and practices be made more effective?

Improved monitoring and evaluation

Evidence of the impact of A4T assistance gathered through case stories and surveys by the WTO and OECD is a step forward in understanding what works and what does not. Econometric models and other forms of quantitative analysis have also contributed some interesting findings. Some have gone as far as estimating the added value of every A4T dollar spent in terms of additional exports, etc.

Nevertheless, there is room to improve how A4T effectiveness is measured. As was well recognized during the Global Review sessions, case stories are not created with the same rigour as in-depth case studies. Stories tend to highlight the positive results and, understandably, pass lightly over any negative aspects. They also are more anecdotal and less analytical, usually involving fewer target stakeholders. A case study has an analytical frame of reference and requires the use of validation techniques to verify evaluative statements made by respondents.

Meanwhile, more quantitative methods often make assumptions of questionable validity and make assertions about impact that require careful scrutiny before using them as guides for trade policymaking. Unfortunately, estimates of improved export performance can be dangerously attractive.

How, then, can monitoring and assessment of future A4T performance be improved? First, while case stories should continue to be gathered, these need to be supplemented by case studies representing a range of different situations and regions, involving interviews with a cross section of stakeholders active in A4T projects. Such studies will require more time and resources to create, but the value they contribute will be well worth it.

Second, greater reliability of quantitative methods is needed to minimize assumptions considered by critics to be too simple to reflect real situations, and to verify the findings of these methods by gathering qualitative evidence in support of such findings.

Third, it is important to invest in strengthening the M&E capacities (institutional and personal skills) of developing countries so they can carry out M&E functions increasingly on their own. Evidence shows that most countries resort to what might be called ‘routine reporting’, focusing on inputs and outputs instead of outcomes and impact. To illustrate, typical M&E practices in infrastructure tend to report how much money was spent building roads, installing quality-assurance systems or setting up new facilities to handle perishable export products. While these are important, they do not explain how such inputs and outputs have resulted in improved export performance, though a logical conclusion would suggest that these have a positive impact. The issue to underline here is that M&E should be based on reliable evidence. A few developing countries are taking measures to move away from routine reporting to more credible M&E practices. Such a shift requires the leadership of relevant institutions to change the way they look at M&E and to take steps to equip M&E units with required skills.

Even with these improvements – more in-depth case studies, improved quantitative methods and enhanced M&E capacities in developing countries – the effective practice of M&E depends on the extent to which leaders of these institutions take the findings seriously and act on them.