Mental Health Leadership and Advocacy Programme(mhLAP)- Final Evaluation
BACKGROUND of Project
This project (P3574 mhLAP) is the continuation phase of a previous one that ended in December 2014. It was designed to improve mental health services in five Anglophone West African countries of Ghana, Liberia, Nigeria, Sierra Leone and The Gambia through improved leadership skills in stakeholder groups, advocacy and stigma reduction. The main strategies for achieving the goals of mhLAP include capacity building and the formation of stakeholder groups committed to advocating for improved mental health service in their respective countries. The overall objective is to reduce the disability associated with mental disorders, based on the principles of promoting the rights to health, to be treated with dignity and to be free from discrimination. The activities focus on building capacity at high level, strengthening systems that can then reduce the treatment gap for mental illness, promoting the delivery of improved mental health service that respects patients’ rights and autonomy, and reducing the stigmatization of mental illness.
Building on the achievements in the first phase of mhLAP, the following results were achieved in the second phase of the project.
a. In trying to address the problem of poor policy attention and implementation of mental health in the countries, the project conducted five annual 2-week leadership and advocacy training workshops and various support for country Stakeholder Councils to be able to influence policies. As a result, the project was able to achieve the following:
· Buy-in and acceptance of the mhLAP as a major partner on mental health related issues in all the the project agenda and by all the participating countries
· Active involvement of the mhLAP in issues relating to service development and strenghtening in all the countries
· mhLAP Ghana actively involved in the process that led to the enactment of the 2012 Mental Health Act as well as in the development of the Legslative Instrument (LI)
· All the other country stakeholder councils actively involved in the process of drawing up new mental health policies and legislation. mhLAP was able to get official backing and support for the successful implementation of the WHO mhGAP Implementation Guide and the Quality Rights tool in all the other participating countries
b. 105 doctors and mental health nurses were trained as local trainers (Ghana 40, Gambia 21, Liberia 17, Sierra Leone 27) to address the problem of treatment gap within the participating countries. These mental health nurses have been able to train 217 doctors and primary care workers in the use of the mhGAP-IG to expand the provision of effective mental health service in the countries.
c. Members of the Stakeholder Councils in the five countries have been empowered to address the problem of poor quality of service and abuse of the human rights of patients in treatment through developing and implementing quality improvement activities for mental health service in their countries using the WHO QualityRights Toolkit. Two members of the Stakeholder Council in each country were trained in the use of the QualityRights Tool to help facilitate this process.
d. The Stakeholder Councils implemented various anti-stigma activities targeted at addressing the problem of the widespread level of stigmatization of persons with mental illness, guided by the principles of promoting the right to health, the right not to be subjected to inhumane or degrading treatment and the right not to be discriminated against. The anti – stigma activities that were conducted include the following:
· MH capacity building workshops for media personnels in all the countries
· Sensitization of health workers on human rights approach to mental health care
· Regular and sustained public awareness raising activities
· Promoting the establishment of service user groups in countries where they do not exist
Overall, these have brought about improved reporting on mental health issues in all five countries where the project was implemented; allocation of media slots on print and electronic media for discussions on MH issues; service users are now quite willing to speak publicly about their illness; a measure of improvement on the public perceptions about mental illness and people suffering from mental illness.
3. Evaluation Purpose, Scope and Intended use
The purpose of the evaluation is to assess the overall project achievements or otherwise during the implementation period, including documentation of lessons learnt that will be used to improve and inform future programmatic orientation/direction, as well as for external promotion giving the impact from implementation of the project. The evaluation will focus on all the activities conducted during the project implementation. Findings and recommendations will be disseminated and used to improve the overall quality and impact of CBM’s work.
The scope of the evaluation will cover activities from January 2015 to June 2019. This evaluation is being carried out as part of the programme development requirements. Its outputs will be used by a variety of stakeholders for different purposes:
implementing partners will use the information from this evaluation
to establish lessons that will be used to plan future programming
to plan for the sustainability of the programme
to understand the impact of the programme on the service users CBM will use the information from this evaluation
to identify key factors and influences in developing community mental health services that may be applicable for scale-up services in other settings in Africa and globally
to inform the way in which CBM can partner and work with government to establish accessible, quality mental health services
to identify areas that may require further strengthening to ensure sustainability and quality of mental health services
to guide provision of technical support to future projects
to feed into practice of CBM Country Office (CO)/Regional Office (RO), CBM Australia (CBM AUS) and partners.
CBM Australia will use the information from this evaluation:
- to inform programming decisions;
- to draw case studies and lessons learnt to support advocacy
- to communicate results to Australian stakeholders as part of accountability practice.
The evaluation process will take a participatory approach by involvement of programme stakeholders, especially the beneficiaries (primary, secondary and tertiary). Programme staff and partners will play a strong role in the evaluation.
4. Evaluation objectives
· To review the achievements or otherwise of the project objectives thus far and the contributing factors
· To address the sustainability of the approach and the degree to which the programme has set a foundation for more promotion and systemic policy consideration of mental health issues in each of the five countries.
· To document lessons learnt from project planning, implementation, monitoring and learning that would guide future project management and share these with other mental health programmes
· To determine the extent to which program design was relevant and appropriate in meeting the needs of the beneficiaries/ achieving the set objectives.
· To build capacity of CBM and implementing partners in the evaluation process.
- Evaluation Questions
Area of Enquiry
Suggested guiding questions
Human Rights violations in the national health system are systematically assessed and a process of reform is started
Have the project’s interventions led to a decrease in the number of people being abused as a result of having psychosocial disability in the participating countries?
What strategies have been put in place to address human rights violations in the national health systems?
What is the level of engagement and ownership of respective national governments to prevention of human rights violations in the health system?
What successes have been achieved with government and what processes are now embedded within government systems? What has been difficult to achieve with government, what have been the barriers and how has the project responded to this?
How effective is the anti-stigma strategies? What are recommendations to ensure sustainability of the strategies?
mhGAP training of trainers, National mhGAP training and mhLAP training are effective
How adequate and effective was the training model used for this project? If doing the trainings provided in another context, what is the important things to be considered to achieve desired results?
Country Stakeholder councils – reduce the abuse of the human rights of persons with psychosocial disabilities
How strong is the Stakeholder Council in each country? What are their focus, their influence, and what areas need further support? Are there particular advocacy successes that can be attributed to the work of the stakeholder councils (achieved or in progress?) What are key elements of the stakeholder councils that have been successful? Looking forward, what do the councils need to do/assure in order to keep functioning now that the specific work of the project is finished?
Project monitoring, reflection, learning and sharing of lessons learned
What are the significant learnings from the project, how can these learnings be applied to help improve the program?
Has there been any published work from the project or presentations in conferences?
Have there been any unplanned or unintended consequences from the project work – positive or negative?
What significant changes have happened as a result of the investments in implementing the project. To what extent do these outcomes potentially lead to sustainability of the project? Document five case studies for each country.
Evaluation Questions Cont’d
To what extent did the objectives and implemented activities meet the needs and priorities of the beneficiaries?
Were the strategies used the most suitable for achieving intended outcomes?
What other strategies or ways could have been used to achieve intended outcomes.
What changes took place as a result of the project implementation whether positive or negative?
Were the project’s intended goals achieved?
Were any unintended change (s) achieved (positive or negative)?
What factors affected the achievement of intended outcomes or otherwise?
To what extent were the expected objectives achieved?
Were activities implemented as planned?
What factors contributed whether or not activities led to intended outputs and outcomes?
Efficiency / cost-effectiveness:
To what extent were the expected objectives efficiently achieved?
Could the project outcomes have been achieved at a lower cost?
To what extent were persons with psychosocial disability included in the project at a ll levels?
How did their involvement or non-involvement affect the achieved outcomes?
To what extent were the services accessible to the beneficiaries?
To what extent are the achieved outcomes in the course of the project likely to continue after project close-out?
To what extent are the achieved outcomes sustainable?
What factors are likely to affect the sustainability of project outcomes?
The evaluation must consider gender aspects of the programme and report on this. Data collected must be disaggregated according to gender.
The evaluator will develop the evaluation methodology in the framework of the available evaluation budget in collaboration with CBM and the programme partner’s representative. He/She is expected to submit a brief inception report where an evaluation methodology should be proposed. The evaluation must meet the principles of being inclusive, participative and interactive, involving both male and female beneficiaries. As a minimum, the evaluation process should include the following key steps:
Ø review of relevant project documents
Programme proposal, budget and log frame
Financial and narrative reports January 2015 – June 2019
MoUs and Agreements
Ø initial briefing session with CBM Country Office and the partner
Ø Application of appropriate data collection tools.
Qualitative methods used could include;
· open-ended questions on questionnaires
· personal interviews
· Logs, journals, records etc.
Quantitative methods used could include;
· close-ended questions on questionnaires
· observation checklists
Regarding confidentiality/ data protection, the evaluator must take all reasonable steps to ensure that the respondent is not adversely affected by taking part in the evaluation. He/She must keep their responses confidential, unless their permission is granted, and must not do anything with their responses that they are not informed about at the time. Also, particular care must be taken with children and teenagers. Permission must be granted from a parent or responsible adult, such as a teacher, for interviews with children aged under 18 and CBM’s child safeguarding policy should be applied in all circumstances.
During the evaluation, the stakeholders consulted by the evaluator should include, project staff (personal interviews) and management, psychiatrists and other clinical and field staff; beneficiaries (Trainers, Mental Health leaders in the countries, patients - children and adults); key community leaders where applicable (traditional rulers, pastors and other sectional leaders in the community); representatives of the Stakeholder Council, National MoHs; service providers; CBM Country Office; and CBM Australia.
Following data collection and analysis, the consultant will share preliminary findings with CBM CO, project partners and CBM AUS. This shall be achieved through debrief sessions at CBM CO and the evaluation sites. The sharing of preliminary findings is an opportunity for the stakeholders to hear what the evaluation has found and to be involved in thinking through recommendations. It should include constructive discussions around the key issues identified by the evaluation.
8. Evaluation team and Management Responsibilities
CBM is responsible for commissioning the evaluation.
CBM Nigeria CO will be responsible for planning and managing the evaluation and checking that quality standards are met, ensuring the evaluation conclusions and recommendations are communicated effectively.
The evaluation team will be identified and approved in consultation with CBM AUS. The consultant will develop an inception report for review by CBM AUS and approval by CBM CO.
The draft evaluation report will be shared with project partners, CBM CO and CBM AUS for review and feedback. The final evaluation draft will be sent to CBM CO and CBM AUS for approval and sign off. The final instalment of consultant’s fees will be disbursed following sign off by CBM CO and CBM AUS.
CBM CO and mhLAP will ensure that some feedback or learning events will be carried out within the mental health stakeholder groups at the different participating countries on the evaluation report and findings.
CBM CO and CBM AUS will agree on dissemination plan of evaluations findings and define action points and ways forward.
The evaluator will be a consultant. S/he will be an experienced evaluator, who will be responsible for the overall evaluation process and the production of the evaluation report. The evaluator will have to sign CBM’s child safeguarding policy prior to any field work. S/he should be familiar or familiarize herself/himself with disability inclusive practices in evaluations.
The evaluator will be supported by a staff of mhLAP during the field visits.
The evaluator will be selected based on the following criteria:
ü Seven (7) to 10 years’ proven experience in programme design, implementation and evaluation with five (5) of those years in evaluating mental health programmes
ü Good knowledge of mental health programming in Nigeria and Sub-Saharan Africa related to advocacy and public awareness, and knowledge and understanding of stakeholders
ü Master’s in Public Health or related background; bias in mental health is an added advantage
ü Knowledge of disability inclusive practices in evaluations is an added advantage
ü Analytical skills proven through submission of a past evaluation report
ü References to include clients and other evaluators
ü Proposed costs (details regarding costs are shown in Section 12)
9. COORDINATION AND LOGISTICS
Country office has responsibility for:
· Overall coordination of evaluation process
· Recruitment of evaluator
· Gathering documents and data for evaluator in collaboration with mhLAP
· Liaising with mhLAP
· Organising post-evaluation debriefing with the evaluator, including mhLAP in the process.
· Covering the evaluator’s transportation to the evaluation site & transportation cost around the evaluation sites
· Covering the evaluator’s hotel accommodation and feeding costs
mhLAP has responsibility for:
· Working with the Country Office to organise meeting schedule for evaluation team.
· Identifying “neutral” and disability accessible locations for interviews/ meetings to take place (where people will feel free to speak as openly as possible).
· Organising for interviews with beneficiaries and community leaders according to the evaluator’s requests/methodology.
· Organising for interviews with the stakeholders in mental health in the participating countries, according to the evaluator’s requests/methodology.
· Organising for an interpreter for the evaluation exercise, if need be
· Provision of guidance on security and safety at the evaluation sites
(i) An Inception report, produced before 6 October 2019
(ii) A draft evaluation report produced no later than 2 November 2019
(iii) A final evaluation report produced by end of 10 November 2019. This needs to be written up highlighting country-level contexts, and should showcase the achievements of the project and indicate what each of the five governments or councils can do once the project ends to ensure that this foundational work to have an ongoing impact.
(iv) Data sets for all collected data (quantitative and qualitative). Qualitative data should be transcribed for future use by CBM. The data sets should be in an appropriate format (SPSS, Excel or Word) and will be submitted together with the final evaluation report.
(v) PowerPoint presentation, summarizing the key finding from the evaluation submitted together with the final evaluation report by 10 November 2019.
The Consultant is expected to submit an inception report before 6 October 2019. The purpose of this report is to ensure that the evaluator covers the most crucial elements of the exercise including the appropriateness and robust methodology to be employed. The inception report provides the organization and the evaluation team with an opportunity to verify that they share the same understanding about the evaluation and clarify any misunderstanding at the outset. The report should reflect the evaluation team’s review of literature and the gaps that the field work will fill.
The draft evaluation report must be submitted to CBM Country Office no later than 2 November 2019. The draft report will be circulated by CBM Country office to key stakeholders for review and feedback. These stakeholders will include project partners and CBM CO. Feedback on the draft report will be shared with the evaluation team no later than 8 November 2019. The report should be prepared using the template in Appendix 2.
The final report of the evaluation must be submitted to CBM after review and incorporation of the various comments made by 10 November 2019. The main aim of the report is not only to feedback on this program, but to improve on the quality of work by the partner and CBM.
The evaluation report is an exclusive property of CBM and should not be released without prior authorization to any other party. The final report will be available through CBM as well as being specifically circulated (by CBM) to the project stakeholders, including the project partners who will be able to use the report freely.
The Consultant will be expected to submit complete data sets (in Excel) of all the quantitative data as well as the original transcribed qualitative data gathered during the exercise. These data sets should be provided at the time of submission of the final report by 10 November 2019.
On submission of the final report, the consultant is expected to submit a PowerPoint presentation (maximum 12 slides), summarizing the methodology, challenges faced, key findings under each of the evaluation criteria and main recommendations. This should be submitted together with the final report by 10 November 2019.
11. Duration and Phasing
11.1 Proposed Time Frame
The proposed time-frame for this evaluation is 30 October to 30 November 2019.
11.2 Duration of Activities
The duration of the evaluation exercise shall be 31 working days from a mutually agreed date no later than 23 September 2019. The evaluation will follow the key phases:
Phase I - Desk study: Review of documentation, elaboration of inception report, and development of evaluation tools [2 days]
The consultant will review relevant documentation from section 7 above. Based on this review, he/she will produce an inception report which will include an evaluation plan, methodology and sampling strategy of the data collection for evaluation study.
Phase II: Field Data Collection (25 days)
This phase of the evaluation will seek to collect primary data on the key evaluation questions explained under evaluation criteria. The evaluation team will use the agreed plan, methodology and sampling strategy from phase 1 to conduct the field work.
Phase III – Data analysis and production of evaluation report [11 days]
The evaluation team will draw out key issues in relation to the evaluation questions and produce a comprehensive report. This analysis should draw on the wider issues in the development sector and how the mhLAP has shaped/affected national, regional and global level mental health services.
The table below summarizes the key activities outlined above
No of Days
Phase I –
Desk research /literature review
Submission of Inception Report
CBM Country Office
Development of evaluation tools
CBM Country Office
Field Data Collection
Field Visits & data-collection (with debriefing at the end of each visit)
Phase III –
Analysis and production of evaluation report
Presentation of initial findings to CBM CO
CBM Country Office (via email)
Data analysis and preparation of draft report
Submission of draft report
CBM Country Office (via e-mail)
Review of draft report after feedback from CBM and partners
Submission of final report
CBM Country Office
12. COSTS AND PAYMENTS
The consultant will submit to CBM an expression of interest indicating the evaluation team’s daily rates for the assignment, detailing professional fees and per-diems. CBM will negotiate with them the final fees in line with the budget available for this evaluation and based on the experience of the chosen candidates.
12.2 SCHEDULE OF PAYMENT
· Signing Contract: First advance of 20%
· Submission of draft report: Second advance of 30%
· Approval of Final Report by CBM Country office and CBM AUS, datasets and PowerPoint Summary: Final payment following receipt of invoice from consultant (50%).
Qualified evaluators should submit via email:
· A cover letter indicating interest
· Curriculum Vitae
· A short summary of your understanding of the brief
· Three previous evaluation reports
· A technical offer, which must include the technical requirements and the evaluation approach/methods, the plan and the timeframe to address them. It would also be an opportunity for the Evaluator to challenge the ToR and offer options.
· A financial offer, including a budget for the evaluation.
The information should be sent to email@example.com titled “Statement of Interest: mhLAP Final Evaluation Consultant” in the subject line. The deadline for applications is 25 October 2019.
The future job holder adheres to CBM values and commits to CBM’s Child Safeguarding Policy.
CBM is an equal opportunities employer, and particularly encourages qualified people living with disability to apply.