Project End-line Evaluation Project Name: Strengthening Public Health Capacity to respond to Myanmar’s Disease Transition

Job
from HelpAge International
Closing date: 15 Nov 2019

Project Background

A project "strengthening public health capacity to respond to Myanmar's disease transition" funded by European Union is being implemented by Help-Age International with the collaboration of University of Public Health(UPH), University of Medicine (2) and Thailand's Thammasat University. Overall objective of the project was to contribute to the process of health reform in Myanmar so that the goals of equitable and universal healthcare can be achieved. Specific objective of the project was to strengthen the University of Public Health and wider health stakeholders to support evidence-based policy and improved health services, particularly in preparation for the transition towards non-communicable diseases in Myanmar. The project is being implemented since February 2015 and has made progress on various activities with several achievements. As part of this European Union supported project, we are supporting the Ministry of Health and sports, Government of the Union of Myanmar for rolling-out of package of essential non-communicable disease interventions across the country. The project will end in January 2020. As one of the major activities, project end evaluation will be conducted to review the implementation progress, achievements and lesson learned for documentations of best practices in order to guide the planning for further NCDs interventions. and will review the project implementation, progress achievements, lessons learned to guide planning of the further NCDs prevention and control programes.

. These expected results were:

Result 1: Enhanced research and evidence-based policy analysis to inform the government’s response to the health transition in Myanmar;

Result 2 : Greater knowledge among 1,500 policy makers and public health implementers across Myanmar on how to address the rising burden of NCDs;

Result 3: Increased in-country policy dialogue and consultation with a wide range of Myanmar stakeholders on the rising challenges of NCDs;

Result 4: Stronger UPH academic capacity (90 academic staff) to deliver high quality teaching and research, guided by a clear institutional strategy;

Result 5: Effective UPH operational capacity including project and financial management

Result 6: Active professional linkages with health institutes in other countries and South-South collaboration.

Purpose of the end line evaluation

The purpose for this evaluation is to assess project performance to capture its achievements, challenges and best practices to inform future similar programming. the performance of the project and capture project achievements, challenges and best practices to inform future similar programming. It will also review the recommendations of the project Mid Term Review (MTR), periodic monitoring visit by the EU monitoring facility and Result Oriented Monitoring (ROM) reports and assess the extent to which these were implemented. Furthermore, the evaluation will ensure accountability towards project donor, European Union while it also offers a learning platform for all stakeholders. The evaluation will also ensure accountability towards European Union as a donor and the beneficiaries of the programOn the other hand, it offers a learning aspect for all stakeholders. Finally, the evaluation will also identify key lessons learned and challenges/obstacles.

The end of programme evaluation will have three objectives:

Objective 1: To what extent "strengthening public health capacity to respond to Myanmar's disease transition" (SPHC) project has delivered effective, efficient, relevant and timely activities to beneficiaries as set in the project log frame to achieve the expected results

Objective 2: To Identify and assess key lessons learned, challenges/obstacles to achieving the project objective and expected accomplishments and draw recommendation for future programming of similar projects

The end line evaluation should assess the following evaluation/review criteria and include all of the following questions: The results of this evaluation will be used to improve future public health capacity strengthening projects.

Scope

The evaluation will cover the entire project period from February, 2015 to November 2019, and all activities planned in the project as defined by the project document.

The evaluation process will engage national experts, policy makers from relevant national ministries and agencies as well as any other entities involved in project implementation. Below are the list of stakeholder involved in the project.

Stakeholders in Myanmar: University of Public Health, University of Medicine-2, University of Community Health, Ministry of Health and Sports, HelpAge International, National NCD alliance, WHO, UNOPS

Beneficiaries in Myanmar: Health Work force at Township and station hospital, rural health centre and sub rural health centre levels. Indirect beneficiaries are the community at large.

Stakeholders in Thailand-Thammasat University, Bangkok, Thailand

Issues

The evaluation will seek to report on the efficiency, effectiveness and relevance of the project. Key questions that the evaluation seeks to answer include: Effectiveness, efficiency, relevant/impact, timeliness, reach and quality.

Objective 1: To what extent "strengthening public health capacity to respond to Myanmar's disease transition" (SPHC) project has delivered effective, efficient, relevant and timely activities to beneficiaries as set in the project log frame to achieve the expected results

The end line evaluation should at least include one lesson learned and recommendation per evaluation category, i.e. effectiveness, efficiency, relevance etc.

Effectiveness

The end line evaluation should assess the following:

  1. To what extent the indicators in the log frame of the project have been measured per indicator and to what extent the action achieved per expected result?

  2. To what extent have the project activities contributed to the overall objective? Was the project effective in strengthening the University of Public Health and wider health stakeholders to support evidence-based policy and improved health services, particularly in preparation for the transition towards non-communicable diseases?

  3. What were the major factors influencing the achievement of the objectives of the project?

  4. What opportunities for collaboration have been, utilized and how have these contributed to increased effectiveness? or otherwise?

  5. Have proper accountability and risk management framework(s) been in place to minimize risks on program implementation?

  6. Give advices, lessons learned and recommendations (if any) on how the expected accomplishments of the project could have been more effectively achieved?

Efficiency:

  1. Were the resources allocated in the SPHC project used efficiently and commensurate to the project results?

  2. Was the project activity implementation (modality), considered to have been cost-efficient, while not compromising quality?

  3. Were project activities implemented in an efficient manner?

  4. What would have been opportunities within project to reach more beneficiaries with the available budget or to reduce costs while reaching at least the same number of beneficiaries without compromising quality?

  5. What choices were, made in terms of collaboration and non-collaboration during program design? Why were, these choices made?

  6. Were, alterations made to the program design in terms of collaboration during the implementation phase based on the reality on the ground?

  7. What were the outcomes of these choices for effective and efficient program implementation?

  8. What are the lessons learned and recommendations?

Relevance/Impact

  1. How relevant were the objectives, activities and major achievements, implemented by the project, in addressing NCD prevention and control and strengthening public health capacity to respond to Myanmar’s disease transition?

  2. To what extent did the project serve the needs of its main beneficiaries, health workers and targeted population?

  3. How do beneficiaries perceive the relevance of the project and how has the activities implemented improved their NCD prevention control activities (health workforce) and improved their lives (community)? Are there any stories of change?

  4. How has the collaboration between HelpAge International, University of Public Health, University of Medicine 2, Thammasat University, local partners and line ministries contributed to appropriate response of project activities?

  5. To what extent was project able to adapt and provide appropriate response to context changes and emerging local needs, and priorities of the beneficiaries?

  6. To what extent were the topics of capacity building activities aligned with national policy on NCD?

  7. What are the lessons learned and recommendations?

Timeliness

  1. Were the project design and interventions timely in responding to the country’s needs?

  2. Were the activities timely implemented in comparison to project planning?

  3. To what extent did the collaboration between HelpAge International and line ministries and other stakeholders contributed to efficient and timely coordination of activities and processes?

  4. What are the lessons learned and recommendations?

Can we say anything about cross-cutting issues?

Reach

  1. To what, extent have the project beneficiaries been reached and what mechanisms were in place to improve coverage?

  2. What are the main reasons that the project provided or failed to provide to its target beneficiaries proportionate to their need?

  3. What are the lessons learned and recommendations?

Quality:

The end line evaluation should assess the overall quality of the implementation. It is important to include beneficiaries’ opinion on the quality of the services received.

  1. What mechanisms are in place to track project implementation of HelpAge International? (i.e. internal monitoring, evaluation, accountability, quality assurance mechanisms)?

  2. How have they been, utilized to increase quality within the project?

  3. Did the quality of activities, delivered by project meet the needs and expectations of the beneficiaries? What do beneficiaries feel could be improved in Contribution to improving local capacity?

  4. To what extent, did the project interventions contributed to strengthening public health capacity to respond to rising NCD trend?

  5. To what extent, did the project interventions contributed to build long-term strengthening public health capacity to respond to rising NCD trend?

  6. To what extent, was the project participatory in all the project cycle?

  7. What are the lessons learned and recommendations?

Objective 2: To Identify and assess key lessons learned, challenges and draw recommendation for future programming of similar projects

Learning:

  1. Which of the approaches and modalities/strategies have been most effective according to HelpAge International and project co-applicants?

  2. Is there any substantial evidence on how project learning, were generated and applied to improve the delivery or effectiveness or efficiency of activities?

  3. Who benefited from shared learning experiences mainly the BHS, community members and beneficiaries?

  4. How did the different actors learn from these experiences?

  5. To what extent were the activities of the project complement to the work of other stakeholders, i.e. prevented duplication and contributed to the larger response activities in Myanmar?

  6. What are barriers and/or enablers to this project?

Sustainability

  1. Will the changes caused by this programme continue beyond the life of the project?

  2. What, mechanisms have, HelpAge International, and partners put in place to sustain the key programme outputs and outcomes?

  3. How has the programme worked with local partners to increase their capacity in a sustainable way?

  4. What motivations /mechanisms exist for partners to continue playing their roles?

  5. What are the risks facing sustainability of programme Outputs and Outcomes?

1. Methodology

The evaluation will be carried out by an independent consultant, who will be responsible for the design of the evaluation methodology and carryout the evaluation. The six results should, be assessed, including all research questions under each result. The data collection should include the use of diverse approaches to gain a deeper understanding of the outcomes of the project, including:

§ Desk review of background documents (project document, project monitoring data, progress report, mid-term review report, activities reports etc). HelpAge will ensure that the evaluator receives all relevant documentation to enable a thorough desk review.

§ Key informant interviews (e.g. with Key HelpAge International project staff, co-applicants, beneficiaries, local partners, Self Help Groups, community members and representatives from the line ministries) to gather substantial anecdotal evidence on the effectiveness, efficiency, relevance and timeliness of the project activities implementation and delivery

§ Focus group discussions (e.g. with stakeholders, community members, representatives, The FGD will serve as input for the narrative anecdotal evidence.)

§ Interviews with selected relevant staff and stakeholders of the project will take place if necessary and the project manager will provide the list with contact details. Project manager will provide support and further explanation to the evaluation consultant when needed.

§ Disseminate lessons learned through a presentation and a workshop facilitated by the consultant

Next to the data collection method, an appropriate and strategic sampling method should be selected e.g. snowball sampling, purposeful random sampling or mixed purposeful sampling methods.

The evaluation consultant will prepare a report on the results of the evaluation based on these terms of reference. The draft report will be submitted to the Technical Lead-Public Health, for comment and quality assurance.

2. Deliverables

The consultant is, expected to accomplish and submit the following deliverables within the agreed timeframe.:

An inception report, which will serve as an agreement between parties on how the evaluation will be, conducted. Items to address:

§ Understanding of the issues and questions raised in the ToR

§ Data sources; how to assess the questions in the ToR

§ Research methodology, including suggested sample and size

§ Schedule of activities and traveling (timeline)

§ Proposal for a learning event/validation of evaluation findings

§

§ Appropriate validated draft data collection tools (e.g. methodological guidelines, group interview questions)

A max 40-page draft and final evaluation report (in MS Office and PDF for final), excluding annexes and in English.

A max of 20-page draft and final tracer report (in MS Office and PDF for final), in English.

Both reports should be in the format indicated below, to be submitted to, HelpAge International Myanmar. It is preferable to illustrate the results by appropriate graphs, visuals, tables and/or a dashboard with an accompanied explanatory text. The report should consist of:

a) Executive Summary in bullets

b) Introduction

c) Methodology, including sampling and limitations

d) Analysis and findings of the evaluation. The analysis should be done according to the objectives:

o evaluation objective 1

o evaluation objective 2 including an implementation strategy for the recommendations

e) Address concerns, lessons learned and comments from HelpAge International

f) Stories of change and quotes from respondents

g) Conclusions for each of the end line evaluation objectives

h) Recommendations for future projects

i) Annexes:

§ Relevant maps and photographs of the evaluation areas where necessary

§ Bibliography of consulted secondary sources

§ Finalized data collection tools

§ List of interviewees with accompanying informed consent forms

o PowerPoint presentation of preliminary findings to HelpAge International Myanmar

o Learning event for project stakeholders, partners, CSO in Myanmar.

3. Indicative timescales

The data collection phase in the field is to be, confirmed between the consultant and HelpAge International, but ideally would start in 1, December 2019, with the final report deadline to HelpAge International by January 15,2020. However, the evaluation contract agreement will be signed in the week of November , 2019.

Phase

Deliverables

Payment

Working days indication

Inception phase

Draft inception report

5

Tools development phase

Final inception report including, methodology and research tools, approved by HelpAge.

10 %

2

Data collection phase

Desk review, interviews in and field visits Myanmar and Thailand

20%

15

Data analyses phase

Presentation of initial findings

20 %

6

Evaluation report phase

Draft Evaluation Report, for comment by project team

30 %

12

Total

100%

days

4. Scope of Work

Consultant will

  1. Develop an inception report, detailing the methodology, stakeholders to be interviewed, tools to be developed, time frame for the evaluation.

  2. Hold the overall management responsibility of the review, including designing and carrying out the evaluation, drafting the final report and debriefing the project team and key stakeholders

  3. Liaise with HelpAge staff throughout the process, providing weekly updates and seeking their input and advice where necessary.

  4. Sign the HelpAge’s Code of Conduct and protection policy abide by the terms and conditions thereof.

  5. Carry out the responsibilities relevant to all deliverables within the set indicative time frame.

  6. Be under the supervision of Project Manager and Technical Lead, Public Health.

5. Qualifications and experience

Required

§ At least Master’s degree in public health, International Development Studies, , or a related field

§ At least 7 years’ experience in working with development programs relating to public health interventions

§ Good knowledge and experience of evaluation, project management, social and demographic statistics

§ Demonstrated methodological knowledge of evaluations, including quantitative and qualitative data collection and analysis for end-of-cycle project evaluations

§ Experience of working in Myanmar/South East Asia

§ Preferable experience of evaluating strengthening public health capacity strengthening or similar programs

§ Relevant subject matter knowledge and experience regarding the major components of this SPHC project

§ Proven record of communicating with beneficiaries.

§ Understanding of Myanmar national health system policy, standards and protocols

§ Ability to assess and further develop a conceptual evaluation tool

§ Strong understanding of development sector and evaluation ethics and a commitment to ethical working practices

§ Action-oriented and evidence-based approach and strong drive for results;

§ Highly developed self-management, and communication skills;

How to apply:

6. Selection process

HelpAge will use its internal guidance, checklists and an interview process to select the successful consultant. HelpAge will also provide guidance on design and quality assurance of the evaluation. The guidelines require the consultant to submit, a proposal explaining, their comprehension of the ToR, and how they would approach this assignment, with a summary of their methodology, especially in terms of how they plan, to meet the objectives, including a period and consultancy fee. The candidates are welcome to propose an evaluation team to complete the evaluation however, HelpAge International will sign the consultancy contract only with the lead consultant. Selection of the consultants are based on experience, quality of the technical proposal and the proposed consultancy fee.

7. Submission of application

Along with their CV interested candidates should submit

§ A technical proposal explaining, their comprehension of the, ToR, and how they would approach this assignment, summarizing, the methodologies, and approaches they, plan to use, including a timeline. (Maximum five pages)

§ Three samples of similar previous assignments

§ A financial proposal outlining their expected fees for a maximum of 45 working days. International travel, visa, boarding, lodging and local travel will be taken care by HelpAge International.,

§ Interested candidates/institutions should submit a technical and financial proposal and three samples of similar previous assignments. Applications should be, submitted to hr.recruitment@helpagemyanmar.org no later than November 15, 2019.