Difficulties of impact assessment in ‘semi-emergencies’
By Tayech Yimer, Anne Marie Mayer and Arabella Duffield
Tayech Yimer is a nutritionist with SC UK in Ethiopia.
Anne-Marie Mayer has worked with SC UK for nine months in Ethiopia as Nutrition and Health Research Advisor. She has a PhD in Nutrition from Cornell University (USA).
Arabella Duffield is a nutrition advisor with SC UK, based at the headquarters in London.
Save the Children UK (SC UK) is currently involved in trying to assess the level of impact and cost-effectiveness of different interventions designed to reduce rates of chronic malnutrition in a number of countries, including Somalia,Vietnam and Ethiopia. Debates about what is the most appropriate type of nutrition programming for the poorest sections of these communities are ongoing but, for now, there is virtually no good quality data on the impact or cost of different intervention models in these countries.
Lack of evidence base
The lack of published impact and cost effectiveness information is of concern. There are key areas of uncertainty regarding the utility of certain types of intervention, e.g. nutrition/health education versus food security support, and over issues of design within programme types, e.g. general food distributions versus cash transfer. This lack of impact and cost-effectiveness information militates against cross-sectoral comparison of interventions in relation to nutrition and mortality impact1. This information is also important in understanding the outcomes of the project in order to address the lessons learnt, improve on any weaknesses and build up best practices.
SC UK has produced guidelines on evaluation, monitoring systems and impact assessments for planning projects2. Many different theoretical frameworks are available for monitoring and evaluation of programmes for nutritional benefit. SC UK has adopted a model of evaluation based on the following five basic indicators: (i) provision, (ii) utilisation, (iii) coverage, (iv) outcome and (v) impact3.
Habicht et al distinguish between three types of evaluations that provide varying levels of confidence in the results; adequacy, plausibility and probability evaluations4. Probability evaluations are the gold standard and involve randomised trials. These are simply not manageable in the contexts in which SC UK works. Instead, SC UK is focusing their efforts on obtaining plausible measures of impact. For an impact assessment of food security or nutrition/ health education programme to be plausible, it is necessary to be able to attribute benefit to the programme rather than to external circumstances. For this, not only should data be collected before and after intervention, but also a control group is necessary to allow for external events, such as seasonal shortages. Baseline data is needed to have an understanding of the capacity and the gaps that the communities have prior to intervention.
There are a number of reasons for the dearth of good quality impact and cost-effectiveness information. Here is presented, by way of an example, one programme funded by USAID in Amhara Region of Ethiopia and the issues that have been identified by SC UK that make it difficult to evaluate this programme for nutritional impact.
The Reducing Dependency Increasing Resilience (RDIR) programme The RDIR programme's overall goal is 'to effectively reduce the dependency on external assistance and increase the resiliency of the chronically food insecure populations of Gubalafto, Sekota, Dehana and Habru Woredas'5. The programme is based on SC UKs 30 year experience of working in Amhara, combining relief and development resources to protect and restore assets while strengthening the institutional capacity of community and local governments to prevent further destitution. The design has also benefited from SC UKs detailed research into the causes of acute and chronic malnutrition in the area6.
The programme includes many components: microfinance, agricultural support, kitchen gardens, grain banks, restocking, poultry production, fodder development, bee-keeping, reforestation, soil and water conservation, small scale irrigation, community spring development, rural infrastructure development, community- based health care, nutrition and health education, HIV/AIDS work, harmful traditional practice awareness, hygiene and sanitation promotion, promotion of energy-efficient stoves, support to local health administrations and gender and women focused activities. Many of the components have the potential to impact on nutritional status of children, but in different timeframes and in different ways. The multi-faceted causes, and hence possible solutions to addressing malnutrition, present challenges for evaluation.
Donor requirements for monitoring the RDIR programme
The system for monitoring and evaluation (M & E) for the RDIR programme is laid out in the Technical Proposal7 and applies to a range of USAID-funded programmes. The M & E framework indicates the results to be achieved at all levels of project, including indicators and verification procedures. It gives details of requirements for non-government organisations (NGOs) to follow with regard to data collection. This includes baseline surveys and a plan for on-going monitoring and a participatory planning process. However, baseline surveys are currently only being undertaken in the project woredas (not in control areas).
Evaluation issues for SC UK
We have considered several issues identified from earlier programmes and the current RDIR programme that will need to be considered for SC UK to produce a plausible, effective impact evaluation. These are described below.
Tight deadlines for new projects
The timescale for proposal writing often does not allow for detailed M & E planning. Staff with M & E expertise are usually based at HQ but deadlines are often very short and proposals are written in country. Due to requirements to start up a project immediately, baseline data may not be collected on time. The RDIR programme started before a baseline survey was carried out.
Problems associated with the selection of control areas
There are ethical issues concerning the use of control groups. We cannot withhold programme components from people who could benefit in areas where we could intervene. In some situations, a stepped-wedge design may be used, i.e. control areas may be identified in areas where an intervention is planned but delayed for logistical reasons, but in this case we need to consider what bias may be created by the reasons for a delayed implementation.
The presence of other NGOs in control areas makes the process of selecting control areas problematic. The activities of the other NGOs could mean that similar programmes are undertaken in both control and intervention areas, resulting in an underestimation of the effect of the programme being evaluated. If other programmes start up in the intervention areas, the effectiveness of the programme might be over-estimated.
There is also the difficulty in getting the control group from a similar agro-ecological zone, i.e. matching the intervention group by food security. If neighbouring areas are used for the control group, there is a danger of the control group being 'contaminated' by the programme by the spread of health education messages, for example. However, using a distant area for a control group creates another problem of a different agro-eco zone and different programmes in place covered by other groups. In Amhara, there are several agro-ecological zones in each administrative area (e.g. lowland, midland and highland) but programmes are usually implemented according to administrative area. This makes the evaluation more complex.
Multiple causes of malnutrition
Multiple causes of malnutrition require different indicators and a complex evaluation model. SC UK believes it is always important to assess any change in household income/expenditure during the nutrition programme's life. This is because a change in income/expenditure will almost certainly have an impact on children's nutritional status. SC UK usually uses the Household Economy Approach (HEA) to assess changes in income/expenditure. The causes of malnutrition, however, are complex and require a proper analysis in order to have understanding of the causes with reference to the UNICEF causal framework. It is important to consider and setup a number of appropriate indicators. However, the usual exercise is often only collecting economic/household food security indicators. This requires some fine-tuning of survey tools and M & E frameworks. More complex surveys cost more in terms of time and resources.
Complexities of design of the programme
Although the benefit of the programme to individual households is likely, the number of households targeted for any one intervention may be too small in any given area to measure impact. The interventions are not given to households as a package but rather one intervention to a few households at a time. Not all households are targeted for each intervention within a given zone. This could be due to logistical constraints, or due to particular requirements for different interventions. Beekeeping, for example, is only possible in low-lying areas. This creates inclusion bias: high land areas will never be able to benefit from this particular component. Also, in the project area there are some overlapping activities, which make the evaluation of impact and calculations of costeffectiveness very complex.
Emergencies can occur whilst the programme is on-going
Unfortunately, in a relief to development setting, the population may experience extreme bouts of food insecurity or very poor public health conditions due to drought, epidemics or insecurity. This can lead to an 'emergency' situation when there is a shift from normal patterns of income and expenditure patterns. Relief interventions may be needed to supplement other programming at this point. This looks likely to occur in Sekota Woreda of the RDIR programme this year and will clearly complicate SC UK's evaluation of impact.
Government policy changes
Changes in targeting of food aid may occur as the programme is progressing. This means that different beneficiaries receive food or cash aid at different times in the programme.
Funding
Funding for evaluations is expensive and does not attract grants easily. Many donors are reluctant to fund research in what they see as a 'semi-emergency' situation as they believe it is unethical. Alternatively, as in RDIR, donors may have a broader plan to evaluate the impact of all their programmes in one country - this is not helpful for individual NGOs who want to evaluate the impact of their own programme.
Programme length
NGO programmes are frequently funded for only 2-3 years, especially in unstable areas. In some situations, households will not have had the time to make significant changes to their behaviour that result in a change in their children's nutritional status. Thus, no anthropometric impact may actually be seen in the short term.
Suggestions for improvement
M & E systems need to be incorporated at the design stage of the programme. It is important to formulate a clear M & E framework with the staff who will be involved in the actual implementation of the project and data collection. SC currently has a small group of highly skilled mobile consultants (who are kept on retainers) in order to assist this process.
AM & E capacity can be established if sufficient budget is allocated and a high level of management support is available for the implementation. It is necessary to train staff in methods of M & E to ensure quality systems are in place.
A Market in Ethiopia
It is unlikely that SC UK will ever be able to undertake a classic control versus programme evaluation because of the ethics of withholding assistance. It is more likely that a steppedwedge design evaluation or a comparison of impact and costs between different types of interventions in similar agro-ecological zones will be made. Communities can be consulted as to which areas should receive interventions first. It is also important to keep good records of all events and confounding factors that are expected to interfere, e.g. other programmes, environmental shocks, food shortages, harvests failures and health hazards, in order to understand changes overtime. Baseline data should be collected from all groups in a population to anticipate changes in policy.
There is a need for donors (individually or as a group) to take more responsibility and fund/help establish research/implementing agency partnerships which aim to address specific questions in particular programme areas where impact and cost-effectiveness information are urgently needed. SC UK is currently trying to access funding specifically for this. SC UK is also looking for funding to measure the impact of programmes 2-3 years after implementation has finished.
Without an efficient monitoring and evaluation capacity at all levels, it is difficult to assess the progress made towards achieving set objectives and targets. With sufficient allocation of resources, careful design and on going monitoring, we can increase the likelihood of producing evaluations that will enable institutional learning and wider dissemination of these lessons.
For further information, contact: Arabella Duffield, Save the Children (UK), 1 St Johns Lane, London EC1M 4AR, email: A.Duffield@savethechildren.org.uk and Tayech Yimsr, Save the Children (UK) Ethiopia, PO Box 7165, Addis Ababa, Ethiopia. Telephone +251(0)1293469 Fax +251(0)1293470, email: tayech.y@savethechildren.org.et
1Duffield A, Reid G, Walker D, Shoham J (2004). Review of the published literature for the impact and cost-effectiveness of six nutrition related emergency interventions. Report for the Emergency Nutrition Network. December 2004. Available from the ENN.
2Save the Children, Toolkits: A practical guide to planning, monitoring, evaluation and impact assessment. Second edition, 2003. London
3Save the Children (UK) (2004). Reference document for SC UK Nutrition Monitoring and Evaluation Group. Report from a meeting. Unpublished
4Habicht, JP, Victora, CG and Vaughan, JP (1999). Evaluation designs for adequacy, plausibility and probability of public health programme performance and impact. International Journal of Epidemiology 28: 10-18.
5Save the Children. Organisation for Rehabilitation and Development in Amhara. Reducing Dependency and Increasing Resiliency: Improving Capacity to Implement Safety Net and Farmer Led Livelihood Development Programs. USAID RFA 663-04-014 and Title II Productive Safety Nets Progamme.
6Save the Children (UK). Health Wealth and Knowledge: Determinants of malnutrition in North Wollo, Ethiopia. 2002 London
7See footnote 5
Taken from Field Exchange Issue 27, March 2006
http://fex.ennonline.net/27/difficulties.aspx