Ukraine

Evaluation of the European Union’s humanitarian assistance in Ukraine, 2014-2018 Final Report

Format
Evaluation and Lessons Learned
Source
Posted
Originally published
Origin
View original

Attachments

Executive summary

Objectives and scope

This is an independent retrospective evaluation of the Directorate-General for European Civil Protection and Humanitarian Aid Operations’ (DG ECHO's) interventions in Ukraine (2014-2018). The evaluation has three objectives:

• Accountability: it assesses the performance of European Union (EU) interventions compared to initial expectations, engages stakeholders and encourages feedback, and offers an independent and objective judgement based on available evidence

• Learning: it supports organisational learning by identifying areas for improvement and encouraging the sharing of (good and bad) practices and achievements

• Strategy: it aims to make information available in time to support planning for 2021

The geographic scope is the entire region covered by DG ECHO’s Ukraine Humanitarian Implementation Plans (HIPs), which include Government-Controlled Areas (GCAs) and Non-Government-Controlled Areas (NGCAs) as well as contributions for displaced Ukrainians in the Russian Federation and Belarus.

Context and EU interventions

Millions of people have suffered the direct humanitarian consequences of the armed conflict in eastern Ukraine: damaged property, loss of life, displacement, lost livelihoods, reduced services, and isolation. However, beyond the fighting itself, the population now suffers the more complex humanitarian consequences of the de facto separation of eastern Ukraine into two regions, one controlled by the government and the other by separatist entities, divided by a narrow zone known as the contact line. After almost six years, the humanitarian needs in eastern Ukraine are still significant, with 3.5 million men, women and children dependent to some extent on humanitarian assistance and protection services. A defining characteristic of the crisis is that approximately 30% of the population in need (and 38% of the population along the GCA contact line) is aged over 60, with high rates of disability, immobility, neglect, malnutrition, and economic insecurity. The needs in the NGCAs are even greater than in the GCAs, but accessing the NGCA population remains a central challenge.

DG ECHO has provided a total of EUR 118.4 million over the five-year period under evaluation, through 65 actions with 21 implementing partners (IPs). DG ECHO’s funding represents 16.5% of the total humanitarian funding received over the period, second after the USA, and equivalent to Germany. DG ECHO’s humanitarian funding is complemented by substantial funding from the EU’s Instrument contributing to Stability and Peace (IcSP) (EUR 81 million) and the European Neighbourhood Instrument (ENI) (EUR 75 million), that have since 2017 coordinated with DG ECHO through a Joint Humanitarian-Development Framework (JHDF).

DG ECHO’s programme has evolved over the five-year period, achieving greater sectoral focus, and moving from a broad approach to one that targeted the most vulnerable population close to the contact line and in the NGCAs. Since the start, DG ECHO has been a key coordination actor in Ukraine (initially facilitating the coordination of all the humanitarian response, and subsequently chairing the Humanitarian Donor Group (HDG)), as well as an advocate for humanitarian objectives. In this unusual humanitarian context of a middle-income country on Europe’s eastern border, strong coordination and humanitarian advocacy have been almost as important as humanitarian programming.

Key findings and conclusions

Overall, DG ECHO has provided a rapid and effective response to the humanitarian crisis in Ukraine, and the DG ECHO programme has performed very well. Furthermore, DG ECHO has demonstrated system-wide leadership in strategic thinking and coordination, and has supported several innovations: notably advancing an explicit JHDF with the other EU services, and supporting the creation of the Ukraine Humanitarian Fund (UHF).

Relevance

DG ECHO’s Ukraine HIPs were clear and context-adapted. The geographic focus, first on Donetsk and Luhansk oblasts and then on the contact line and the NGCAs, was appropriate and evolved according to the changing needs. By 2019, the humanitarian needs in the GCAs were either stable or gradually reducing in some sectors (especially in urban areas). DG ECHO’s sectoral emphasis was also well-matched to the needs, and its small financial allocations to education were appropriate in the circumstances.

DG ECHO’s partners consulted beneficiaries in the design of their initiatives, although DG ECHO’s partners overall fell short of the full expectations of Accountability to Affected Populations (AAP) in respect to beneficiary participation in initiative design and performance assessment. By 2018, DG ECHO’s programme accounted well for the needs of vulnerable groups. Although DG ECHO and its partners were aware from the start of the high numbers of vulnerable elderly within the affected population, they were slow to adapt their approaches to this underlying structural factor. DG ECHO and its partners could have made greater efforts early in the response to reach out to include beneficiaries who were bedridden or socially isolated. As a result, the elderly received “normal” support, but not the particular kinds of support that they might need because of their special needs as elderly beneficiaries.

By 2019, needs assessment was comprehensive, vulnerability targeting was much improved, most of the pressing humanitarian needs in the GCAs were being met, and the remaining area of significant under-addressed needs was in the NGCAs, where access remains difficult.

Coherence

DG ECHO contributed substantially to creating a system of joint and impartial needs assessments through their partnership with REACH. DG ECHO made conscious and visible efforts to maintain humanitarian principles, and its actions in Ukraine were aligned with DG ECHO’s relevant thematic/sector policies, although there was room for improvement regarding the elderly.

DG ECHO has been a strong coordination actor in its own right, as well as a strong supporter of other humanitarian coordination actors in Ukraine. DG ECHO’s coordination work was vitally important in the initial stages of the response, when few humanitarian agencies were present. EU member states (MS) and other donors valued DG ECHO’s leadership and information sharing, enabled by the fact that DG ECHO has more specialised humanitarian staff in-country, and unique access to the NGCAs.

In Ukraine, system-level coordination between the humanitarian and development communities is not as advanced as internal nexus coordination within each donor government. Within the EU, DG ECHO has made substantial efforts to encourage EU humanitarian-development coordination, and the evaluation team assessed the drafting and implementation of the JHDF as a qualified success. The perceived weaknesses of the JHDF are not so much weaknesses with the strategy itself, as with the different mandates of the EU’s Directorate-General for Neighbourhood and Enlargement Negotiations - Support Group for Ukraine (DG NEAR-SGUA) and DG ECHO, underpinned by different planning and programming systems.

EU value-added

The broad humanitarian community felt that DG ECHO added value to programme delivery, as well as at the policy and system levels, particularly in speed, scale, agility and coordination. EU MS further agreed that DG ECHO added considerable value above and beyond the efforts of individual donor governments, in particular through its convening of information sessions for MS, access to NGCAs, and advocacy. The foundation for this added value was the strong and stable country team, backed up by regional technical expertise, with funding at scale, and access to the NGCAs.

Effectiveness

Although there is less data on the NGCA side, the target number of 400,000 vulnerable people each year have reduced protection risks and are able to meet most of their basic needs. If we consider counterfactuals, there were certainly no protection crises or acute gaps in basic needs reported during the evaluation period, and as time went on it was clear that DG ECHO and its partners were reaching more into the “forgotten corners” of the response: the hardest-to-reach people, and the most isolated settlements. DG ECHO provided an appropriate mix of cash and in-kind assistance. Furthermore, DG ECHO’s support for coordination and for quality needs assessment, through the humanitarian organisation REACH, benefited the whole humanitarian community. Through its leadership and advocacy DG ECHO had an important impact on the overall quality and direction of the humanitarian response, but there is still need for continuous advocacy on access to the NGCAs, and on the humanitarian-development nexus.

DG ECHO’s programme increased in effectiveness as humanitarian organisations became better established and better coordinated. By the end of 2018, it was becoming increasingly clear that the critical “life-saving” humanitarian needs were largely being met. By 2019, some DG ECHO-funded activities were working in the space that should be covered by government services, especially in health and education, although some humanitarian aid was still warranted. The remaining frontier, where humanitarian needs remained high and where effectiveness was less certain, was the NGCAs.

Efficiency

The overall response of all agencies was less efficient in the first 18 months as the humanitarian system was being built up. DG ECHO’s proposal application and review process (the eSingleform and annual HIP cycle) was a source of frustration for partners. Regarding some of the key components of the Grand Bargain, DG ECHO partners did not gain the benefits of multi-year funding, but efficiency was advanced through localisation. DG ECHO partners valued the field monitoring undertaken by DG ECHO staff, but felt that similar monitoring by several donors could have been better coordinated.

Regarding the overall level of funding relative to needs, stakeholders felt strongly that DG ECHO’s response was not commensurate with the needs. However, DG ECHO funding for Ukraine was proportionally higher than it is globally. The evaluation team found that it could not assess whether DG ECHO funding to Ukraine was sufficient: it is clear how much DG ECHO provided, but it is not clear if this was enough because there is no objective measure of the monetary value of the humanitarian needs. What is more certain is that DG ECHO’s funding to the NGCAs is not yet sufficient.

In 2020, the humanitarian needs in the GCAs and humanitarian funding are reducing, and the challenge of humanitarian efficiency is to maintain a sufficient humanitarian response in a context of reducing funding. To do this requires that donors and implementing agencies make significant changes to the way they work but, so far, the observed changes have been piecemeal and incremental. Some measures taken by the humanitarian community including DG ECHO have helped improve efficiency, notably shaving back on the costs of cluster coordination, localising staff positions, encouraging the creation of the ACCESS consortium, some pooling of resources for shared services such as REACH’s needs assessments, and the UHF. However, these measures are not yet going far enough to gain step changes in efficiency. In the view of the evaluation team, a relatively “quick win” to increase efficiency is to move from annual to multi-year planning and programming.

Sustainability and connectedness

In the GCAs, the crisis is taking place in a middle-income context, where government services function and the Government accepts its responsibility to assist its own population. Given this conducive environment for sustainability, DG ECHO did well strategically, for example rapidly phasing out of areas and sectors where the Government was able and willing to step in, facilitating the access by affected persons to their social benefits, moving its centre of effort to the most affected zone along the contact line and later to the NGCAs, and encouraging development donors to step in. In this “big picture” perspective, DG ECHO has taken major steps to increase sustainability and connectedness by linking affected populations to government systems.

However, the evaluation team also concludes that DG ECHO could have made more progress on sustainability in the GCAs. First of all, DG ECHO overestimated the willingness and agility of development donors to bring longer-term development-oriented support to the most affected regions of the GCAs: despite DG ECHO advocacy, this has been slow to materialise. Secondly, DG ECHO paid less attention to recovery and sustainability at the sector and action levels. In some areas such as health, mental health, elder care and winterisation, DG ECHO supported partners that were effective but insufficiently linked to government systems: they either developed parallel service delivery structures, or developed models of support to government that are beyond what government systems can sustain in the long run. In these areas, there is a risk that government will not step in, either because they might not want to create a precedent for a higher level of service than they can sustain, and/or because there is little incentive to step in - for as long as humanitarian actors are willing to continue. Concretely, although DG ECHO has had exit in mind since early in the response, there was little sign that DG ECHO’s partners considered exit strategies until they were pressed to do so by declining funding.

Importantly, the depth of humanitarian needs and gaps in services in the NGCAs are such that there is no prospect of sustainability there for the foreseeable future, and neither are development donors able to work there.