World + 10 more

Global Facility for Disaster Reduction and Recovery: Synthesis evaluation report final version - September 2016

Format
Evaluation and Lessons Learned
Source
Posted
Originally published
Origin
View original

Attachments

Executive Summary

At the request of the Global Facility for Disaster Reduction and Recovery’s (GFDRR) Consultative Group, this report consolidates the findings and recommendations of two recent independent evaluations of GFDRR, with a focus on identifying a potential path forward for improved results measurement. The consolidation focused on synthesizing the common elements across both evaluations, namely GFDRR’s contribution to results achievement at the output, intermediate outcome, and outcome and impact levels, including GFDRR’s informing of larger investments by the World Bank and other partners, and observations on GFDRR’s monitoring and reporting systems.

The consolidated evaluation findings and recommendations are presented below.

Findings

Progress toward Results Evidence across from the 2014 and 2015 evaluation suggests that GFDRR has successfully delivered outputs, and that those outputs were reasonable in scope and scale given the size of the grants. Table ES-1 below illustrates outputs achieved in the case study countries: Bangladesh, the Eastern Caribbean, Ethiopia, Guatemala, Indonesia, Malawi, Nepal, Sri Lanka, and Vietnam.

Both evaluations found evidence that most of the observable results of GFDRR interventions are in the intermediate outcome step of the results chain.

Most GFDRR activities in the ten countries visited are making valuable contributions to achieving processoriented (i.e., intermediate) outcomes. Intermediate outcomes observed include: raising disaster risk awareness at local and national levels and increasing the availability of disaster risk information; building capacity of national and local governments, as well as civil society, for disaster risk preparedness, reduction, and response; developing and demonstrating innovative tools and approaches for DRM; strengthening policy dialogue and supporting policy development and implementation, including around disaster risk financing and insurance; and influencing and leveraging significant resources for DRM.
GFDRR has leveraged DRM resources primarily through support for the preparation of post-disaster needs assessments (PDNAs) and technical assistance that informed a World Bank lending operations, as well as, to a lesser extent, by informing recipient country government expenditures. Table ES-2 summarizes these results for the 10 case study countries covered by the 2014 and 2015 evaluations.

Overall, GFDRR has been relatively successful in identifying entry points for small grant contributions to demonstrate or advance DRM activities that can inform larger-scale investment operations. Combined, the 2014 and 2015 evaluations identified about $2 billion of project operations informed by GFDRR in the ten countries studied, including $1.7 billion of World Bank commitments with the remainder in co-financing from recipient governments, bilateral donors, and the Global Environment Facility, among others. Approximately $800 million is associated with disaster risk and climate resilience projects, with the remainder associated with mainstreaming disaster risk considerations into infrastructure investments (e.g., transport, water management, urban development) and poverty reduction programs. These values should be interpreted cautiously, however, because the scale or significance of GFDRR’s contribution to individual operations varies significantly.

Limited evidence was found of outcomes and impacts achieved at-scale, although some activities show strong potential. In all countries studied, the 2014 and 2015 evaluations found that sustained engagement is needed to improve the likelihood that some activities’ intermediate outcomes will proceed toward outcomes and impacts.

Linking GFDRR small grants with larger World Bank investment operations or broader government initiatives may reinforce potential for downstream results, but GFDRR’s contribution to those operations’ outcomes and impacts is difficult to discern. Many World Bank investment operations for which GFDRR has contributed to the incorporation or improvement of DRM components will achieve sizeable outcomes, if successfully implemented. However, these impacts cannot be directly attributed to GFDRR, and even GFDRR’s relative contribution to these impacts is difficult to establish. Current monitoring and reporting systems are not designed to differentiate the impacts of such contributions to larger investments—not only for GFDRR, but for other comparator funds as well.

Both the 2014 and 2015 evaluations identified the important role of an in-country focal point as a driver of deeper engagement and conditions for results.

GFDRR’s partnership with the World Bank has also been important to enable high-level engagement and provide opportunities for GFDRR’s relatively small grant activities to have a broader impact via World Bank operations. Other factors for success have been GFDRR’s strong partnerships, which have enhanced the scope of GFDRR’s potential results, as well as strong choices for executing agencies, which has built capacity among local actors. In some countries, a programmatic approach or cohesive strategy has also supported results achievement.

Challenges to success have included lack of readiness or capacity to use some of the technologies piloted by GFDRR, long development periods for some technical assistance activities, and the use of less-effective activities, such as one-time training events or conference attendance support. The observation of these particular challenges suggests that a long-term approach is especially needed to solidify results for certain activity types, such as the introduction of new technologies and support for disaster risk financing and insurance. In addition, in Bangladesh, the evaluation observed that GFDRR utilized a co-financing modality ineffectively, lacking strategic dialogue during the creation of that arrangement and engagement during implementation.

Monitoring and Evaluating Results

GFDRR lacks a systematic process to monitor and report results beyond the output level. Measuring and evaluating the results of technical assistance programs—especially those focused on resilience—is difficult. Moreover, monitoring and evaluation is in the early stages of implementation for climate and disaster resilience programs. Many global programs hosted by the World Bank also lack evidence of results beyond the output level. GFDRR’s challenge is further compounded by the broadening scope of its work plan, with focus on a variety of issues from resilient cities, to infrastructure, to gender, to climate change, and the way that its technical assistance often informs broader investment operations.

A key shortcoming identified by the 2014 and 2015 evaluations is that there is too much “distance” between GFDRR’s output and outcome indicators, meaning that GFDRR’s results beyond the output level are not being adequately captured. Intermediate (i.e., shorter-term) and longer-term outcomes have not yet been conclusively defined for GFDRR, nor have they been clearly integrated into the program’s theory of change. A secondary issue is that some of the GFDRR “output” indicators—as currently defined in Annex III of the 2016-18 Business Plan—blur the lines between outputs and intermediate outcomes.
A more robust assessment of GFDRR’s M&E system, as well as the development and implementation of a refined M&E plan, could help better articulate GFDRR’s expected process of change, identify a set of suitable and relevant intermediate outcome indicators, as well as longer-term outcome indicators, and move the program toward a stronger results orientation.

Recommendations

To improve future GFDRR results measurement and achievement, the evaluations made the following recommendations. These recommendations are based on evidence and findings from 2014 and 2015; since then, GFDRR has taken decisions that affect its strategies and approaches, and that may or may not align with these recommendations. The recommendations below should be understood in this historical context.

Recommendation #1: Deepen and sustain engagement on the ground. Deeper and more sustained engagement could improve potential for achieving downstream results by addressing several of the challenges identified by both the 2014 and 2015 evaluations, including limited readiness or capacity to use some of the technologies piloted by GFDRR, long development periods for certain types of interventions, high government turnover, and occasional lack of follow-up by GFDRR. Improved engagement could be fostered as follows:

  • Prioritize interventions that link to broader initiatives and make use of GFDRR’s wellrecognized technical expertise. Country studies suggested that activities that are linked to World Bank, government, and other donor initiatives and programs are more likely to have strong stakeholder support, show better potential for contributing to results at-scale, and achieve leverage or influence. Similarly, interventions that make use of GFDRR’s comparative advantages in the DRR community, including technical expertise and regional thematic initiatives, also show strong promise for achieving results.

  • Support and coordinate through DRM focal points. The 2014 and 2015 evaluations found that DRM focal points have helped ensure that activities maintain momentum and advance toward outcomes at scale. These staff have also been instrumental in mainstreaming DRM into World Bank operations, particularly where such mainstreaming is a stated objective of the GFDRR program in that country. For example, follow-up to ensure that communities of practice, technologies, and other GFDRRsupported activities continue to be implemented after individual grants have closed may lead to better outcomes.

  • Consider more focused or cohesive approaches within countries. Both evaluations found that GFDRR is producing valuable results, but the size of GFDRR’s program is small compared to overall country needs. Within each country, focusing in on activities that provide added value and build on GFDRR’s strengths could support better achievement of sustainable and higher-order outcomes. For example, limited evidence was found by both evaluations of sustained results of one-time training events or conference attendance not connected to other, ongoing GFDRR initiatives. In contrast, both the 2014 and 2015 evaluations noted that a cohesive strategy has supported results achievement in countries where it has been used. At the country level, grants could be more purposefully designed to build on and reinforce each other; results are stronger in countries where there is a clearer linkage and trajectory among grants. Focusing efforts on building institutional capacity—rather than individual staff capacity— may also be an effective strategy.

Recommendation #2. Strengthen GFDRR monitoring and evaluation of results beyond the output level. GFDRR needs a sound methodology that clarifies the theory of change with a straightforward results framework, identifies a limited number of meaningful and measurable indicators at the outcome and impact level, and explains the role of evaluation in answering questions that are of interest to GFDRR stakeholders, and particularly the Consultative Group. Tying these elements together in a coherent M&E system would enable GFDRR to better communicate how the program is delivering results through its engagement in countries.

Some of these improvements could include:

  • Refine the existing program results framework to “close the gap” between current GFDRR outputs and outcomes. Long time horizons for achieving disaster resilience outcomes and non-linearity makes monitoring long-term outcomes very challenging. Instead, monitoring at the intermediate outcome level (e.g., two-tofive years) is being increasingly recognized as a viable approach for resilience programs. The 2014 and 2015 evaluations also showed that this is the level of results where GFDRR’s contributions are most evident, given GFDRR’s valuable role as a facilitator/catalyzer of progress in DRM performance at the country level. Including and monitoring intermediate outcomes in the M&E framework would more accurately hold GFDRR accountable for its own performance.

  • Make further refinements to the M&E framework to enable more robust M&E. These refinements could include defining a limited number of SMART (Specific, Measureable, Achievable, Relevant, and Time-bound) intermediate outcome indicators that are logically tied to the results framework; developing indicator definitions and measurement protocols, and identifying data sources to improve the reliability and validity of reporting; better aligning the M&E framework with the Sendai Framework indicators (under development) to improve relevance; developing approaches and tools to measure gender outcomes within GFDRR activities; and defining where and how evaluation can help address the questions and needs of GFDRR constituents.

  • Address operational aspects of monitoring and reporting. Making improvements to GFDRR’s M&E system as described above may also require some operational changes to improve efficiency and ease of implementation. Systems or tools might be developed or improved, roles and responsibilities might be clarified, and different staff or resources might be required. In addition, clearly stating how monitoring information will be used is important— for example, how the information could be integrated into the Annual Report, and what other reports and learning products might be produced.