Comprehensive Community Mental Health Programme-Mid year review
Background of Project
The Comprehensive Community Mental Health Programme, Phase 2 (CCMHP-2) is an extension of the pilot project funded by Australian Government’s Department of Foreign Affairs and Trade through CBM and implemented by the Methodist Church Nigeria (MCN), Otukpo Diocese; the initial funded phase having elapsed in 2016. CCMHP-2 commenced in January 2017 and is expected to end in December 2021.
The goal of CCMHP-2 is improved quality of life of people with psychosocial disability in Benue State through access to integrated healthcare, empowerment, and reduced barriers to social inclusion.
A layout of results-based activities was developed, and stakeholder engagement was strengthened towards achieving the project objectives. These activities focused on organizational capacity strengthening, community mental health service integration – which includes scaling up services in primary and secondary healthcare levels, in humanitarian emergencies and prisons settings – and promotion of mental health. A strong element of knowledge management is built into the project to generate learnings from implementation of the project for CBM and its stakeholders, as well as the global mental health community.
The project is building on the existing community mental health structures that were established in the first phase of the project to improve access to integrated mental health services and treatment through the primary and secondary health care levels for both general health (maternal and child health, and HIV), prisons services, and humanitarian emergency settings. In this phase, the project is working to achieve these by engagement with government ministries, departments and agencies (State Ministry of Health and Human Services, State Primary Healthcare Development Board, Hospitals Management Board, State AIDS Control Agency, State Ministry of Education, and the Nigeria Prisons Service) to deliver primary care and community-based treatment and psychosocial services to vulnerable populations. To achieve this, various capacity building workshops were undertaken to equip frontline healthcare providers to provide the needed care to persons living with mental and psychosocial disabilities, address stigma due to mental illness to reduce the burden of mental illness and disabilities on persons with lived experience (service users) and their families, and change community attitudes and perceptions about mental illness.
The main stakeholders in this project include persons with lived experience and their families, persons with other forms of disabilities and their associations, persons living with HIV, government officials, health care workers, civil society organizations, and traditional and religious leaders. Government officials, religious leaders and service users were also trained in mental health leadership and advocacy to add voice to the advocacy for mental health policy and services in the state. All these are geared towards consolidating on the development of the stakeholder alliance that was formed in the first phase of the project to make it more robust and active in the state.
The project is working with no less than 100 families to develop livelihoods by providing support for vocational training and business start-ups, and developing community-based savings groups.
As a result of the persistent farmers-herders and community clashes in the state, the project is working with over 50 mainstream actors (including government) to mainstream disability-inclusive response into the state emergency management plans through training of first responders in disability inclusion, psychological first aid and mental health psychosocial support. These responders are expected to deliver mental health services and mainstream disability more broadly into their mainstream activities within emergency settings.
An essential component of the project in this phase is to document and share learnings from project implementation. A portion of the budget is dedicated for knowledge management and is being used to conduct researches and communicate findings and the project’s achievements to the wider community, as well as test specific behavior change communication model and an electronic database for reporting community mental health services and generate data for service coverage in the state as part of efforts to integrate mental health indicators and data elements into the Federal Government’s health management information system.
3. Evaluation Purpose, Scope and Intended use
The purpose of the evaluation is to assess the project design, scope and overall project achievements midway into the project life span to ascertain the level of success attained measured against the project benchmarks. The evaluation will also explore implementation status and whether the expected outcomes are achievable. It will also seek to collate challenges, lessons learnt and best practices obtained during the implementation period (January 2017 – June 2019) and suggest possible reviews to getting the project on track, if required.
The scope of the evaluation will cover overall achievements from January 2017 to June 2019.
This evaluation is being carried out as part of the project requirements. Its outputs will be used by a variety of stakeholders for different purposes:
v Implementing partners will use the information from this evaluation
ü To gauge their level of implementation and refocus the project for greater effectiveness for the remaining period, if need be.
ü to plan for the sustainability of the project
v CBM will use the information from this evaluation
ü to identify areas that may require further strengthening to ensure sustainability and quality of mental health services
ü to learn and guide its monitoring functions
ü to guide provision of technical support to future projects
ü to feed into the community of practice of CBM Country Office (CO)/Regional Office (RO), CBM Australia (CBM AUS) and partners.
The evaluation process will take a participatory approach by engaging programme stakeholders, especially the beneficiaries (primary, secondary and tertiary). Project staff and partners will play a strong role in the evaluation.
4. Evaluation objectives
The objectives of the evaluation include:
· To review the achievement of the project objectives thus far and the factors contributing to its successes or otherwise.
· To identify areas that need further actions that will contribute towards delivery of project goal
· To assess the sustainability of the project
· To build capacity of CBM and implementing partners in the evaluation process.
5. Evaluation Questions
Area of Enquiry
Suggested guiding questions
a. Access to integrated mental health services and treatment through the primary and secondary health care levels for both general health (maternal and child health, and HIV) and prison services, and humanitarian emergency settings
How accessible in Benue State are community mental health services as a result of the project? How many people have accessed services since the beginning of this phase?
What measures are in place to ensure sustainability of mental health service in primary and secondary health care delivery?
How sustainable is the drug revolving fund in ensuring adequate supply of psychotropic medication?
How has the project ensured quality of care in delivering services?
Stakeholder empowerment and engagement – collective advocacy for sustained quality mental health services
How effective is the stakeholder alliance or movement in Benue State? What areas of the stakeholder alliance need further support?
How has the project contributed to improving the mental health in secondary schools as a result of the school-based intervention?
C. Access to sustainable livelihoods for persons with psychosocial disability
What are the impacts of inclusive savings groups and vocational skills training for persons with psychosocial disabilities??
To what extent have both men and women with (psychosocial) disabilities been engaged in the livelihoods interventions?
What successes have been recorded in self-help group development
Disability inclusive practice
To what extent does the project mainstream disability into its activities - particulary in state MoH policies and practice?
Project monitoring, reflection, research, learning
To what extent has the project team engaged in ongoing project monitoring/reflection? To what extent has the project been able to build on lessons learned from the first phase of the project and other in-country mhGAP projects?
What are the significant learnings from the project implementation? How can these learnings be applied to this project and other programs?
EVALUATION QUESTIONS CONT’D
To what extent are the expected outcomes of the programme on track?
What factors have led to the achievement or hindered the achievement of expected outcomes?
To what extent has CBM / University of Ibadan supported contributions to achievements.
Is there an approved budget, log frames to support implementation?
Are the strategies used for programme implementation effective?
Are the strategies meeting the needs of the beneficiaries?
Could the use of different strategies have produced better results?
What are the strengths, weaknesses, opportunities and treats to the programme implementation?
To what extent are the advantages of the programme likely to continue after this programme?
What is the probability of sustainability of the expected program outcomes?
What factors will be required in order to improve the possibility of sustaining the programme.
What are the major lessons learnt?
What are the recommendations for better programme implementation in the future?
The evaluation must consider gender aspects of the programme and report on this. Data collected must be disaggregated according to gender.
The evaluator will develop the evaluation methodology in the framework of the available evaluation budget in collaboration with CBM and the partner project representative. He/she is expected to submit a brief inception report where an evaluation methodology should be proposed. The evaluation must meet the principles of being inclusive, participative and interactive, involving both male and female beneficiaries. As a minimum, the evaluation process should include the following key steps:
Ø Review and analysis of relevant project documents
Project proposal, budget and log frame
Financial and narrative reports from January 2017 to June 2019
MoUs and Agreements with partners
Ø Initial briefing sessions with CBM Country Office and the partner
Ø Application of appropriate data collection tools.
Data are being collected and stored by the project in various formats; there are large volumes of data stored in Excel and on District Health Information Software (DHIS), and others collected as descriptive and so, are stored as narrative reports. The data collected are disaggregated according to gender, age and disability or psychosocial impairments. Qualitative methods used could include;
· open-ended questions on questionnaires
· personal interviews
· Logs, journals, records etc.
Quantitative methods used could include;
· close-ended questions on questionnaires
· observation checklists
Regarding confidentiality or data protection, the evaluator must take all reasonable steps to ensure that the respondent is not adversely affected by taking part in the evaluation. He/she must keep their responses confidential, unless their permission is granted, and must not do anything with their responses that they are not informed about at the time. Also, particular care must be taken with children and teenagers. Permission must be granted from a parent or responsible adult, such as a teacher, for interviews with children aged under 18 and CBM’s child safeguarding policy should be applied in all circumstances.
During the evaluation, the stakeholders consulted by the evaluator should include, project staff (personal interviews) and management, psychiatrists and other clinical and field staff; beneficiaries (Trainers, Mental Health leaders in Benue State, patients - children and adults); key community leaders where applicable (traditional rulers, pastors and other sectional leaders in the community); representatives of the Stakeholder movement, State Ministry of Health and related departments and agencies of government; service providers; CBM Country Office; and CBM Australia.
Following data collection and analysis, the consultant will share preliminary findings with CBM CO, project partners and CBM AUS. This shall be achieved through debrief sessions at CBM CO and the evaluation sites. The sharing of preliminary findings is an opportunity for the stakeholders to hear what the evaluation has found and to be involved in thinking through recommendations. It should include constructive discussions around the key issues identified by the evaluation.
8. Evaluation team and Management Responsibilities
CBM is responsible for commissioning the evaluation.
CBM Nigeria CO will be responsible for planning and managing the evaluation and checking that quality standards are met, ensuring the evaluation conclusions and recommendations are communicated effectively.
The evaluation team will be identified and approved in consultation with CBM AUS. The consultant will develop an inception report for review by CBM AUS and approval by CBM CO.
The draft evaluation report will be shared with project partners, CBM CO and CBM AUS for review and feedback. The final evaluation draft will be sent to CBM CO and CBM AUS for approval and sign off. The final instalment of consultant’s fees will be disbursed following sign off by CBM CO and CBM AUS.
CBM CO and CCMHP will ensure that some feedback or learning events will be carried out within the mental health stakeholder groups at the different participating countries on the evaluation report and findings.
CBM CO and CBM AUS will agree on dissemination plan of evaluations findings and define action points and ways forward.
The evaluator will be a consultant. S/he will be an experienced evaluator, who will be responsible for the overall evaluation process and the production of the evaluation report. The evaluator will have to sign CBM’s child and adult-at-risk safeguarding policy prior to any field work. S/he should be familiar or familiarize herself/himself with disability inclusive practices in evaluations.
The evaluator will be supported by a staff of CCMHP during the field visits.
The evaluator will be selected based on the following criteria:
ü Seven (7) to 10 years’ proven experience in programme design, implementation and evaluation with five (5) of those years in evaluating mental health programmes
ü Good knowledge of mental health programming in Nigeria and Sub-Saharan Africa related to advocacy and public awareness, and knowledge and understanding of stakeholders
ü Master’s in Public Health or related background; bias in mental health is an added advantage
ü Knowledge of disability inclusive practices in evaluations is an added advantage
ü Analytical skills proven through submission of a past evaluation report
ü References to include clients and other evaluators
ü Proposed costs (details regarding costs are shown in Section 12)
9. COORDINATION AND LOGISTICS
Country office has responsibility for:
· Overall coordination of evaluation process
· Recruitment of evaluator
· Gathering documents and data for evaluator in collaboration with CCMHP
· Liaising with CCMHP
· Organising post-evaluation debriefing with the evaluator, including CCMHP in the process.
· Covering the evaluator’s transportation to the evaluation site & transportation cost around the evaluation sites
· Covering the evaluator’s hotel accommodation and feeding costs
CCMHP has responsibility for:
· Working with the Country Office to organise meeting schedule for evaluation team.
· Identifying “neutral” and disability accessible locations for interviews/ meetings to take place (where people will feel free to speak as openly as possible).
· Organising for interviews with beneficiaries and community leaders according to the evaluator’s requests/methodology.
· Organising for interviews with the stakeholders in mental health in the participating countries, according to the evaluator’s requests/methodology.
· Organising for an interpreter for the evaluation exercise, if need be
· Provision of guidance on security and safety at the evaluation sites
(i) An Inception report, produced before 30 September 2019.
(ii) A draft evaluation report produced no later than 12 October 2019.
(iii) A final evaluation report produced by end of 14 October 2019.
(iv) Data sets for all collected data (quantitative and qualitative). Qualitative data should be transcribed for future use by CBM. The data sets should be in an appropriate format (SPSS, Excel or Word) and will be submitted together with the final evaluation report.
(v) PowerPoint presentation, summarizing the key findings from the evaluation submitted together with the final evaluation report by 14 October 2019.
The Consultant is expected to submit an inception report before 24 September 2019. The purpose of this report is to ensure that the evaluator covers the most crucial elements of the exercise including the appropriateness and robust methodology to be employed. The inception report provides the organization and the evaluation team with an opportunity to verify that they share the same understanding about the evaluation and clarify any misunderstanding at the outset. The report should reflect the evaluation team’s review of literature and the gaps that the field work will fill.
The draft evaluation report must be submitted to CBM Country Office no later than 12 October 2019. The draft report will be circulated by CBM Country office to key stakeholders for review and feedback. These stakeholders will include project partners and CBM CO. Feedback on the draft report will be shared with the evaluation team no later than 14 October 2019. The report should be prepared using the template in Appendix 2.
The final report of the evaluation must be submitted to CBM after review and incorporation of the various comments made by 14 October 2019. The main aim of the report is not only to feedback on this program, but to improve on the quality of work by the partner and CBM.
The evaluation report is an exclusive property of CBM and should not be released without prior authorization to any other party. The final report will be available through CBM as well as being specifically circulated (by CBM) to the project stakeholders, including the project partners who will be able to use the report freely.
The Consultant will be expected to submit complete data sets (in Excel) of all the quantitative data as well as the original transcribed qualitative data gathered during the exercise. These data sets should be provided at the time of submission of the final report by 14 October 2019.
On submission of the final report, the consultant is expected to submit a PowerPoint presentation (maximum 12 slides), summarizing the methodology, challenges faced, key findings under each of the evaluation criteria and main recommendations. This should be submitted together with the final report by 14 October 2019.
11. Duration and Phasing
11.1 Proposed Time Frame
30 September to 14 October 2019
11.2 Duration of Activities
The duration of the evaluation exercise shall be 20 working days from a mutually agreed date no later than 23 September 2019. The evaluation will follow the key phases:
Phase I - Desk study: Review of documentation, elaboration of inception report, and development of evaluation tools [2 days]
The consultant will review relevant documentation from section 7 above. Based on this review, he/she will produce an inception report which will include an evaluation plan, methodology and sampling strategy of the data collection for evaluation study.
Phase II: Field Data Collection (13 days)
This phase of the evaluation will seek to collect primary data on the key evaluation questions explained under evaluation criteria. The evaluation team will use the agreed plan, methodology and sampling strategy from phase 1 to conduct the field work.
Phase III – Data analysis and production of evaluation report [5 days]
The evaluation team will draw out key issues in relation to the evaluation questions and produce a comprehensive report for presentation to stakeholders in the state to stimulate discussions on sustainability of the project beyond the current funded phase. This analysis should draw on the wider issues in the development sector and how the CCMHP has shaped/affected national, regional and global level mental health services – how the project is contributing to changes in policy and services, and the extent to which the project activities and services have been embedded in government services and community structures. A PowerPoint presentation of the analysis and findings from the evaluation is also envisaged.
The table below summarizes the key activities outlined above
No of Days
Phase I –
Desk research /literature review
Submission of Inception Report
CBM Country Office
Development of evaluation tools
CBM Country Office
Field Data Collection
Field Visits & data-collection (with debriefing at the end of each visit)
Phase III –
Analysis and production of evaluation report
Data analysis and Presentation of initial findings to CBM CO
CBM Country Office (via email)
Presentation of initial findings of the evaluation with project stakeholders
Preparation of draft full report
Submission of draft full report
CBM Country Office (via e-mail)
Review of draft report after feedback from CBM and partners
Submission of final report
CBM Country Office
12. COSTS AND PAYMENTS
The consultant will submit to CBM an expression of interest indicating the evaluation team’s daily rates for the assignment, detailing professional fees and per-diems. CBM will negotiate with them the final fees in line with the budget available for this evaluation and based on the experience of the chosen candidates.
12.2 SCHEDULE OF PAYMENT
· Signing Contract: First advance of 20%
· Submission of draft report: Second advance of 30%
· Approval of Final Report by CBM Country office and CBM AUS, datasets and PowerPoint Summary: Final payment following receipt of invoice from consultant (50%).
12.3 MODE OF PAYMENT
Qualified evaluators should submit via email:
· A cover letter indicating interest
· Curriculum Vitae
· A short summary of your understanding of the brief
· Three previous evaluation reports
· A technical offer, which must include the technical requirements and the evaluation approach/methods, the plan and the timeframe to address them. It would also be an opportunity for the Evaluator to challenge the ToR and offer options.
· A financial offer, including a budget for the evaluation.
The information should be sent to firstname.lastname@example.org titled “Statement of Interest: CCMHP Mid-year Evaluation Consultancy” in the subject line. The deadline for submission of applications is 24 October 2019.
CBM is and equal opportunities employer, and particularly encourages qualified people living with disability to apply.