Oxfam GB’s Global Performance Framework is part of the organization’s effort to better understand and communicate its effectiveness, as well as to enhance learning for staff and partners. Under this Framework, a small number of completed or mature projects are selected at random each year for an evaluation of their impact, in an exercise known as an ‘Effectiveness Review’. One key focus is the extent to which the projects have promoted change in relation to relevant Oxfam GB global outcome indicators.
During the 2016/17 financial year, one of the projects that was randomly selected for an Effectiveness Review was the Citizen Participation in Adaptation to Climate Change (CPACC) project, carried out in Zambia from April 2013 to March 2015. The CPACC project was the third in a series of projects that were launched in response to flooding across southern Africa in 2008 and 2009. Oxfam implemented this sequence of projects in Zambia in partnership with three local non-governmental organizations (NGOs), the People’s Participation Service (PPS), the Program for Sustainable Rural Development (PSRD) and the Southern Partners Organization (SPO).
The project was designed to increase the resilience of participants to a number of different shocks and stresses. In the project districts, households’ livelihoods are based principally on agriculture, which is threatened by erratic and uncertain weather events, including floods and droughts. The project worked at a number of different levels to try to reduce households’ vulnerability to these risks. At a local community level, the project aimed to help households diversify their livelihood approaches and adapt to increasingly uncertain weather patterns by providing training and inputs for producing new crops and using more sustainable agricultural practices. Training was also provided on the use of both indigenous and scientific early warning information, as well as on broader topics related to climate change. In addition, the project worked with community members, particularly the Satellite Disaster Management Committees (SDMCs), to produce and update action plans detailing how the community as a whole could develop in the long term and respond to crises such as floods and droughts. These plans were then integrated into district government structures, and community members participated in training sessions and workshops to help them access extension services and advocate for improvements to local infrastructure. To further foster information-sharing between communities, the project also broadcast training messages and interviews on local radio stations, and conducted exchange visits so that farmers could observe agricultural practices in other project communities.
The Effectiveness Review used a quasi-experimental evaluation design to assess the impact of the project activities among the households whose members directly participated in them. This involved comparing households that participated in the project with a group of comparison households who were thought to have had similar characteristics to the project participants before the project was carried out. Using this methodology, the Effectiveness Review was able to identify the household- and community-level effects of the project only. The impact of those activities that were intended to benefit people across the entire local area – including the radio broadcasts and capacity-building at the district level – could not be assessed in this evaluation.
The Effectiveness Review was carried out in Mongu District, one of the three districts in which the CPACC project was implemented. In Mongu, direct project activities (such as the provision of training and the distribution of seeds and other agricultural inputs) had been carried out in eight communities. Random samples of the direct project participants in each of these eight communities were interviewed. For comparison purposes, 12 communities were identified that were thought to have similar characteristics to the project communities, but with which neither the CPACC project nor any similar projects had been implemented. In the comparison communities, households were selected at random for interview from among the whole population of the community. A total of 282 project participants and 545 comparison respondents were interviewed.
At the analysis stage, to increase confidence when making estimates of the project’s impact, the statistical tools of propensity score matching and multivariate regression were used to control for apparent baseline differences between the households in the project and comparison communities.
The results of the Effectiveness Review are discussed below and are summarized in Table 1. The primary aim of the Review was to investigate the project’s impact on building resilience to shocks, stresses and uncertainty. This was assessed by identifying 28 characteristics that are thought to be associated with resilience, and which can be measured at the household level. These characteristics were identified through discussions with project staff and focus groups conducted in local communities. The characteristics were categorized according to Oxfam’s framework in which resilience is understood to be composed of absorptive capacity, adaptive capacity and transformative capacity. A full list of these resilience indicators and a breakdown of the results for each are shown in Table 2.