Long-term Impact Evaluation of the Malawi Wellness and Agriculture for Life Advancement Program
The Wellness and Agriculture for Life Advancement (WALA) project, which operated from 2009-2014, sought to improve the food security and vulnerability of 214,974 chronically food insecure households in eight districts in Southern Malawi (Nsanje, Chikwawa, Thyolo, Mulanje, Zomba, Machinga, Chiradzulu, and Balaka). This project was funded by the Office of Food for Peace (FFP) of the United States Agency for International Development (USAID). WALA was implemented by a consortium of organizations that included Catholic Relief Services (CRS)/Malawi, ACDI/VOCA, Africare, Chikwawa Catholic Diocese (CCD), Emmanuel International (EI), Project Concern International (PCI), Save the Children (SAVE), Total Land Care (TLC), and World Vision International (WVI). Project activities included maternal and child health projects; nutrition, agriculture, natural resource management, irrigation, and economic activity treatments; and disaster risk reduction activities. Sustainability of the project relied on household members’ continued motivation to use the practices taught and supported under WALA, as well as continued access to public and private services.
As initial trends looked promising, USAID commissioned an evaluation to assess the long-term impacts of WALA. At endline, the project reported a substantial reduction in child undernutrition prevalence rates among project beneficiaries, as well as in communities’ need for food aid during crises. These changes took place within WALA-treated villages, but with no comparison group identified, it was not clear whether these changes could be attributed to WALA activities. Moreover, after WALA activities ended in 2014, it was possible that some gains had faded while others had been sustained.
In order to assess WALA’s long-term impacts, the Expanding the Reach of Impact Evaluation (ERIE) team created a rigorously identified group of comparison villages to compare to the treated WALA villages. These comparison villages were identified by using data from the 2010 Demographic and Health Survey (DHS) and a variety of spatial covariate datasets that reflect pre-WALA conditions. The team combined these data to generate an interpolated gridded geographic surface of the percentage of stunted under-5 children at baseline in 2009. We used this surface to match each WALA village with a comparison village that had similar stunting rates at baseline. The final evaluation sample then included 100 village pairs randomly selected from the full set (with 100 WALA villages and 100 comparison villages).
Both quantitative and qualitative methods were utilized to collect follow-up data from WALA villages and comparison villages. The quantitative survey was administered in 2018 to 21 households in each of the 200 selected villages across the eight WALA districts. A subset of the sample involved revisiting households interviewed in 2013 as part of the project endline survey, while the remainder were randomly selected households. The household survey questionnaire covered health, nutrition, agriculture, income, and development project participation. It also included an under-5 child nutrition questionnaire, which included questions on maternal health and child nutrition. The enumeration team also collected the anthropometric measurements of children aged 6-59 months.
Since the comparison group was initially selected based on comparability in predicted baseline stunting rates, it was possible that residual prediction error could potentially have resulted in a comparison group that was different from the treatment group in ways that were correlated with long-term outcomes. If this was the case, causal attribution based on comparisons between these groups could be biased. However, the research team confirmed that the treatment and comparison groups were indeed comparable by looking at a variety of demographic indicators in the household survey that were unlikely to change quickly.
An additional threat to the evaluation included potential contamination in the comparison group from other development activities. Should other development activities specifically target comparison group villages, it could attenuate any differences in true treatment status across the groups and thus hinder the research team’s ability to causally attribute results to these treatments. In order to address this potential threat, the household survey collected information about household participation in development activities in the past ten years and today. The data shows that households in WALA villages were 7.5% more likely to report having participated in at least one development activity than those in the comparison group. Given that WALA engaged a subset of households within villages and that a number of technical reasons shaded this estimate downwards, this suggests that the comparison group indeed reflects the counterfactual conditions for credible causal impact estimation.
Qualitative data collection was completed in 14 villages across four of the eight districts. Two districts were served by WALA follow-on projects (UBALE and Njira) while the other two districts did not have either follow-on project. The researchers selected villages that were surveyed by the quantitative team and that were nominated by implementers as locations where the WALA project was well implemented, the community was receptive, and interventions were perceived to be most successful in terms of adoption and outcomes. On top of this, researchers also looked at villages with the largest drop in stunting rates during the project. From this selection criteria, the qualitative team selected 14 villages where they conducted a total of 28 focus group discussions and 28 key informant interviews.
The team also interviewed a total of eight implementing partners and eight government officials. During the interviews, the discussion covered a wide range of topics including perspectives on WALA activities in general and their sustainability, household resiliency, coping mechanisms, livelihoods in the face of shocks, diversification of income sources, and reliance on direct food aid.