ATTACHMENT J.5 - CHASS SMT Mid Term Review.pdf

EVALUATION

PERFORMANCE EVALUATION OF THE CLINICAL

HIV/AIDS SYSTEM STRENGTHENING PROJECT IN

SOFALA-MANICA-TETE PROVINCES

December 30, 2015

This report was produced by AGEMA Consultoria Lda, for the United States Agency for International

Development/Mozambique.

Authors: Peter S. Wandiembe, PhD, Rosemary Barber-Madden, PhD, Esther Kazilimani-Pale, MPH, and

Verona Parkinson, PhD

Performance Evaluation of the CHASS-SMT Project iii

PERFORMANCE EVALUATION OF THE

CLINICAL HIV/AIDS SYSTEM

STRENGTHENING PROJECT IN SOFALA-

MANICA-TETE PROVINCES

December 30, 2015

Agema Consultoria Lda. submits this report to USAID/Mozambique as a deliverable under

Contract No. 656-O-00-15-00048-00 for the Performance Evaluation of the Clinical HIV/AIDS

System Strengthening Projects–Sofala, Manica, and Tete, implemented by Abt Associates Inc, with a sub-award to FHI 360.

DISCLAIMER

The authors’ views expressed in this publication do not necessarily reflect the views of the

United States Agency for International Development or the United States Government.

Performance Evaluation of the CHASS-SMT Project i

CONTENTS

ACRONYMS AND ABBREVIATIONS .......................................................................... iii

EXECUTIVE SUMMARY .................................................................................................. v

1 EVALUATION PURPOSE AND QUESTIONS

1.1 EVALUATION PURPOSE

1.2 EVALUATION QUESTIONS

2 PROJECT BACKGROUND

2.1 GENERAL CONTEXT

2.2 OVERVIEW OF CHASS-SMT IMPLEMENTATION

3 EVALUATION METHODS AND LIMITATIONS

3.1 EVALUATION METHODS

3.2 SITE SELECTION AND DATA COLLECTION

3.3 DATA ANALYSIS

3.4 STUDY LIMITATIONS

4 FINDINGS, CONCLUSIONS AND RECOMMENDATIONS

4.1 FINDINGS

4.2 OVERALL CONCLUSIONS

4.3 FUTURE DIRECTIONS

4.4 SUMMARY OF RECOMMENDATIONS

5 REFERENCES

ANNEXES

ANNEX I: SCOPE OF WORK

ANNEX II: DATA COLLECTION INSTRUMENTS: KII, FGD

ANNEX III: LIST OF INTERVIEWEES

ANNEX IV: SOURCES OF INFORMATION

ANNEX V: SUMMARY OF QUANTITATIVE DATA

ANNEX VI: EVALUATION TEAM MEMBERS

ii Performance Evaluation of the CHASS-SMT Project

LIST OF TABLES, FIGURES AND CHARTS

Table 1: Selected districts and health facilities

Table 2: Categories of key informant interview respondents

Table 3: Summary of selected indicators to illustrate achievements in clinical services

Table 4: Recommendations for future programs

Figure 1: Patients already on ART and newly enrolled

Figure 2: Key PMTCT indicators

Figure 3: Performance of systems and capacities at the districts within the provinces at different rounds of assessments (Source: Project WPY 2015)

Figure 4: Retention and community linkage indicators

Chart 1: Clinical services weaknesses and recommendations

Chart 2: HSS weaknesses and recommendations

Chart 3: HSS weaknesses and recommendations (2)

Chart 4: Challenges to community linkages

Chart 5: Constraints to activities aimed at improving retention and recommendations

Chart 6: Challenges associated with improvement of health facility management

Chart 7: Challenges to data management capacities at health facilities

Chart 8: Challenges and weaknesses in scaling-up GBV services

Performance Evaluation of the CHASS-SMT Project iii

ACRONYMS AND ABBREVIATIONS

Acronym/

Abbreviation Definition (English) Definition (Portuguese)

AIDS Acquired Immune Deficiency Syndrome Síndrome de Imunodeficiência Adquirida

ANC Antenatal care Cuidado pré natal

APEs Agentes Polivantes Elementer

ART Antiretroviral treatment Tratamento Anti-retroviral

ARV Antiretroviral Anti-retroviral

CBO Community-based organization Organização Baseada na Comunidade

CD4 Cluster of Differentiation 4 Cluster de Diferenciação 4

CHASS-SMT

Clinical HIV/AIDS System Strengthening

Project in Sofala, Manica and Tete

DPAF

Provincial Administration and Finance

Department

Departamento Provincial de Administracao e

Financas

DPS Provincial Health Directorate Direcção Provincial de Saúde

EP Multidisciplinary team Equipa Polivalente

EPTS Electronic patient tracking system Sistema Electrónico de Seguimento de

Pacientes

FILA ARV drug pick-up form Folha de Informação de Levantamentos de

ARV

GAACs Community Support and Adherence

Groups Grupo de Apoio e Adesão da Comunidade

GBV Gender-based violence Violência Baseada no Género

HAI Health Alliance International

HCT HIV counseling and testing Aconselhamento e Testagem em Saúde

HF Health facility Unidade de saúde

HIV Human Immunodeficiency Virus Vírus da Imunodeficiência Humana

HR Human resources Recursos Humanos

M2M Mother-to-Mother Groups Mãe para Mãe

MCH Maternal and child health Saúde Materno-Infantil

MOH Ministry of Health Ministério da Saúde

MSF

Médecins Sans Frontières (Doctors without Borders) Médicos sem Fronteiras

PCC Community Care Project Projecto de Cuidados Comunitários

PCR Polymerase Chain Reaction Reacção em Cadeia da Polimerase

PEP Post-exposure prophylaxis Profilaxia Pós-Exposição

PEPFAR

U.S. President’s Emergency Plan for

AIDS Relief

PIMA CD4-analyzing machine Máquina para Analisar as Amostras de CD4

PLHIV People living with HIV/AIDS Pessoas que Vivem com o HIV/SIDA iv Performance Evaluation of the CHASS-SMT Project

Acronym/

Abbreviation Definition (English) Definition (Portuguese)

PMTCT

Prevention of mother-to-child transmission Prevenção da Transmissão Vertical

QA Quality assurance Controlo da qualidade

QI Quality improvement Avaliação da Qualidade de Dados

SA Sub-agreement

SDSMAS District Health Directorate Serviços Distritais de Saúde, Mulher e Acção

Social

TB Tuberculosis Tuberculose

Performance Evaluation of the CHASS-SMT Project v

EXECUTIVE SUMMARY

EVALUATION PURPOSE

The Clinical HIV/AIDS System Strengthening Program in Sofala, Manica and Tete provinces (CHASS-

SMT) was a five-year project funded by the United States Agency for International Development. It was implemented by Abt Associates Inc. with an overall goal of improving public sector HIV clinical services in the three provinces within a strengthened comprehensive primary health care system.

The purpose of this performance evaluation of the CHASS-SMT was to determine how the project’s activities were performed relative to their objectives. The evaluation was to provide an objective view of progress towards the expected results. The main objectives were to (a) assess CHASS-SMT achievements, emphasizing project-level results; (b) identify implementation successes, as well as any internal and external constraints that hindered the implementation of planned activities; and (c) propose recommendations for future directions of the CHASS project.

KEY EVALUATION QUESTIONS

The evaluation was guided by 10 questions,1 with the following four key questions:

1) What are the strengths and weaknesses of the activities as seen by the stakeholders, and how can these weaknesses be improved?

2) What constraints have the activities faced in improving retention of patients in antiretroviral treatment (ART) and pre-ART programs?

3) To what extent has knowledge (community and health worker) and utilization of gender-based violence (GBV) services increased over the life of the project?

4) What are the benefits and challenges of the activities’ model of working with government

(at provincial and district levels) through sub-agreements?

PROJECT BACKGROUND

The CHASS Program was designed to address the HIV situation in Mozambique and focused on three components: (a) Improving service quality in HIV prevention, care and treatment; (b) Enhancing program linkages and integration to provide a continuum of accessible services; and (c) Creating stronger and more sustainable Mozambican systems and institutions. In Sofala, Manica and Tete, CHASS was implemented by Abt Associates Inc. from November 2010 to July 2015.

EVALUATION DESIGN AND METHODS

The evaluation data were collected September 4–28, 2015. The evaluation was based on a non-experimental design using a mixed-methods approach. The evaluation team reviewed the existing quantitative project data and documentation and also conducted field key informant interviews with staff from the project (18), 15 health facilities (HFs), health and social welfare departments at the three provinces and eight districts (89), the Ministry of Health (MOH) (3), USAID/Mozambique (5) and community-based organizations (CBOs) supporting CHASS-SMT (9). The evaluation used a multi-channel data collection approach, using mobile technology when possible and paper-based methods when handheld computers/tablets were not appropriate. All of the interviews were recorded on tablets.

Some of the evaluation team members handwrote their field notes directly onto the tablets using a stylus beta enhanced Open Data Kit (ODK) program. The qualitative data were analyzed using a

1The ten questions are listed in Section 1 of this report: Evaluation Purpose and Evaluation Questions.

vi Performance Evaluation of the CHASS-SMT Project qualitative data analysis matrix. Information from this analysis was triangulated with the available quantitative data from CHASS project documents.

EVALUATION LIMITATIONS

The limitations of the evaluation are as follows: Firstly, in many cases, the evaluation team was unable to ascertain consistent factors responsible for the project’s success and weaknesses as the project evolved over time. Therefore, some of the findings reported are from a summative perspective. Secondly, the findings reported here are responses provided by the interviewees to the evaluation team. The likelihood of interviewee bias and recall bias cannot be ruled out.

FINDINGS, CONCLUSIONS AND RECOMMENDATIONS

The findings for each evaluation question are summarized below:

1) Project achievements and challenges:

HSS activities: The introduction of the graduation path to assess for weaknesses and develop action plans for Provincial Health Directorate (DPS) systems and capacities was successful. This led to improvement in the management capacities from below 50 percent to 70 percent, according to

MOH management standards. Manica province was the most improved. At the HF level, the project revised a tool used for supportive supervision and converted it to an electronic clinical monitoring and tutoring tool aimed at quality improvement. This helped in systematic identification of training needs, on-the-spot targeted tutoring, and development of detailed action plans. The project helped reduce the human resource shortage through a pre-service training of more than 400 health workers, who were deployed in the districts. Technical skills of existing staff were improved through on-the-job and in-service training, and maternal and child health (MCH) nurses were trained to prescribe antiretrovirals (ARVs).

The project lacked a well-planned, appropriate exit strategy. Tools and methodologies such as the graduation path and the clinical monitoring tool that were key in HSS have not been institutionalized in any DPS/District Health Directorate (SDSMAS), and hence their experiential learning was limited.

Further, most of the project’s operational costs were managed by the project team.

The sustainability of HSS is subject to staff turnover and transfers; high transfer rates reduce the capacity at the SDSMAS and HFs. The project also did not put a mechanism in place for

SDSMAS/health staff to train new staff. Limited operations research was completed, and quality improvement approaches were not sufficiently promoted.

Clinical service delivery activities: HIV/AIDS clinical services were improved and expanded to new HFs through staff training and mentoring, minor facility renovations, improvement of auxiliary services, and creation of demand and active referrals from the communities through activists, in coordination with CBOs. The project contributed to structuring and organization of HIV services, data review, expanded number of ART sites, and scale-up of Option B+, leading to more than a threefold increase in HIV counseling and testing (HCT), ART and Option B+ coverage. The table below summarizes key indicators.

These impressive improvements are threatened by the insufficient human resources and high rates of staff transfers. Provincial, district and HF-level stakeholders reported an insufficient number of staff to manage improved activities, meet increased demands and maintain high-quality services. For example, MCH nurses in some HFs reported attending to more than 60 patients per day. One of the notable consequences of this is the little time taken to counsel clients. Inadequate counseling was noted as a barrier to improvements in retention rates in care and treatment, and to recruitment of infants into care. Clients who do not understand the importance of staying in care or

Performance Evaluation of the CHASS-SMT Project vii early infant diagnosis are less likely to return for scheduled clinic visits. In addition to insufficient staff, the infrastructural design and small physical spaces at some HFs limit the privacy of clinical consultations and pose a barrier to adequate counseling.

Table 1: Key indicators for HIV care, ART and prevention of mother-to-child transmission (PMTCT) programs

Indicator FY

FY

FY

2013 FY 2014

Percent of

2014 Target

Achieved

Number of patients currently enrolled on ART 57,894

(48,106)

61,016

(70,012)

102,198

(77,039)

155,588

(107,879) 144%

Number of patients newly enrolled on ART 13,955

(19,558)

23,922

(22,908)

32,481

(29,966)

47,675

(44,305) 107%

Percent of patients currently enrolled on ART that are men

33%

31%

30%

29%

No target

Percent of patients currently enrolled on ART that are children

8%

(N/A)

8%

(N/A)

8%

(11%)

8%

(13%)

Retention rate in 12-month ART cohort (Adult) 68% 65% 74% 72% 85%

Percent of pregnant women with known HIV status (newly tested and known positive at antenatal care (ANC) entry)*

90%

(90%)

92%

(90%)

91%

(90%) 101%

Percent of HIV-positive pregnant women in ANC who have initiated Cotrimoxazole*

21%

33%

64%

Percent of HIV-positive pregnant women who received ARVs to reduce the risk of MTCT

88%

86%

86%

Percent of infants born to HIV-positive women who received HIV test within 12 months of birth*

45%

46%

52%

Percent of HIV test results for infants born to

HIV-positive women who received an HIV test within 12 months of birth that are positive*

11%

9%

7%

*No data were available for most of the PMTCT-related indicators in FY 2011 and are left blank in the table.

**Targets were not set for these indicators.

2) Most improved HSS component: The project strengthened systems and capacities of various components of the health system. The human resources (HR) component was identified by both

DPS and SDSMAS key informants as the most significantly improved. The DPS and SDSMAS staff were trained on HR management and use of electronic HR information systems, and health workers were trained through mentoring and tutoring, on-the-job training and pre-service training.

3) Ready-to-transition project activity or component: Most routine project activities were part of MOH activities that are supposed to be executed by provincial and district technical teams.

Whereas the project team led the implementation of these activities, they involved DPS and

SDSMAS. Key project-specific activities that should be transitioned to the DPS and SDSMAS team leadership are the graduation path and clinical monitoring methodology and assessments.

4) Strengthening community linkages: The project moderately improved linkages between the

HFs and communities in all 36 districts. This led to increased uptake of HCT, PMTCT care, ART and

TB care services and slight improvements in retention of patients in pre-ART and ART services. By working with eight CBOs (three each in Sofala and Tete, and two in Manica), the project created new linkages between the HFs and the communities. Through CBOs, the project provided logistics to community activists for case-finding of pre-ART care and ART defaulters, community sensitization, mobilization and community-based HIV testing. The facility-based case managers hired by CBOs were instrumental in generating lists of defaulters, receiving referrals from the viii Performance Evaluation of the CHASS-SMT Project communities and communicating with health workers. Through provision of training and logistics to the adherence support groups (GAACs), people living with HIV (PLHIV) and Mother-to-Mother

(M2M) activities, the project improved knowledge of services and improved retention in ART programs. Overall, across all three provinces, adult and pediatric retention rates increased from 65 percent and 64 percent in FY 2011 to 74 percent and 72 percent in FY 2014, respectively, though these are still below the 85 percent target. However, the community reach to men was still low.

Less than 10 percent of male partners of pregnant women were tested in the PMTCT setting.

Stakeholders cited various barriers to retention, including long distances to the facilities, poor quality of counseling due to insufficient number of HF staff and lack of privacy due to limited physical space, and time constraints, especially for working men.

5) Constraints to improving retention rates: Retention rates persisted below the 85 percent target across the provinces. A major retention constraint was the insufficient number of staff to provide counseling in some HFs and communities. Retention is a cycle that begins with counseling, and if counselors do not take time to counsel and answer patients’ questions so that they understand the need for regular clinical reviews or adherence to ARVs, retention becomes a problem. Other constraints (see point 6 below) included the long distances activists had to travel to reach defaulters; an insufficient number of activists to cover wide catchment areas; wrong addresses given by patients; patients moving out of the community during farming season; small physical space at HFs, limiting privacy of counseling sessions; and time constraints, especially for men.

6) Most effective method for improving retention: Key retention strategies included (1) active case-finding, where activists use a defaulters’ list generated by the HF case managers to search for defaulters in communities; (2) GAAC members picking up drugs for each other and providing psychosocial support; (3) community mobilization by activists, encouraging retention and adherence to ART; and (4) psychosocial support from lay counselors, PLHIV and M2M groups. In many HFs, these activities were executed in a complementary manner. Although it was reported to be costly and did not cover all communities, active case-finding was said to have been the most effective strategy. The GAAC activities were moderately effective, followed by the activities of the PLHIV and M2M groups. Community mobilization by activists and counselors also played a supportive role.

7) Improvement in HF management: The use of one-stop PMTCT and TB models, training of HF staff in process organization, reinstallation of the fluxogram (flowchart) card system and reengineering of patient flow led to impressive improvements in patient flow and waiting times.

8) Improvement in HF data management capacity: Error rates of filling in the HIV/AIDS registry decreased substantially, and correct filing of the registers and summarization of registry records for data entry improved moderately. HF data from Sofala, Manica and Tete passed the data quality audit conducted by the External Data Quality Assessments evaluation team in late 2014.

9) Knowledge of GBV (among health workers and community) and service utilization:

CHASS trained provincial and district focal points and trained activists to refer cases to HFs.

Activists also conducted community sensitization. Cases reported at the HF from the community, police and judiciary increased. However, community linkages for GBV services are still weak, with no active follow-up on victims, and GBV services are not well integrated with other services, except in MCH. Questions remain regarding accessibility of these services to men and boys.

10) Benefits and challenges of working with government through SAs: SAs enabled the development of systems and capacities of the SDSMAS and DPS, “forcing” them to learn. To some extent, SAs fostered mutual understanding of the activities and wider ownership. However, they were based on project activities and had no allowance for executing any activities outside that

Performance Evaluation of the CHASS-SMT Project ix scope. Further, there were delays in reimbursement. Overall, the capacity of the DPS and SDSMAS to manage direct funding needs further strengthening.

CONCLUSION

The project achieved some of its key objectives, including a significant contribution to HSS in the three provinces. However, limited operations research was conducted to understand reasons for low retention, poor patient flow (especially in rural HFs) and low recruitment of children on treatment. The effectiveness of new strategies and quality improvement (QI) initiatives introduced by the project–e.g., reengineering of patient flow, ARV stock management through mobile phones, and improvement of PCR sample quality and turnaround of results–was also not assessed. Overall, Sofala province performed best on most targets.

RECOMMENDATIONS

Health Systems Strengthening

1) Involve the DPS and SDSMAS in the project’s design; they should play an active role in all stages. A transition plan should be agreed on with the SDMAS, DPS and MOH and be explicitly incorporated in the project documents starting the first year. Face-to-face interactions with the DPS and MOH should be a preferred method of communication. The MOH should actively play its agreed role to ensure its deliverables, success and sustainability.

2) Introduce a component of operations research in project activities and in local capacity building.

3) Provide logistics (including tools, manuals, computer tablets, and financial logistics for DPS travel to districts and DPS/SDSMAS travel to HFs) for institutionalization of the graduation path and clinical tutoring tool by DPS.

4) Establish a training mechanism: This could include a peer-to-peer model in which trained health workers tutor at nearby HFs, work with training institutions to establish courses in HSS for all health workers, and training a group of national and district-level trainers.

5) Expand mHealth (mobile health) in project areas. Most aspects of data management and patient follow-up can be performed with mobile technology. Tablets or PDAs could be used by community case managers or activists to input client information and for text message reminders for patient clinic visits, including mothers returning babies for HIV testing and adherence to ARVs. Working with a mobile phone company, the tablets can also be used for online tutorials for health workers.

This has worked well in agricultural health systems in other African countries.

6) Advocate for and support the MOH in rolling out the electronic patient tracking system (EPTS) to moderate- to large-volume HFs. Meanwhile, encourage and train health workers to correctly fill in

ARV drug pick-up forms (FILAs) and other retention forms and file them appropriately, and initiate conversations with DPS about the roll-out plan for EPTS once the MOH approves it, including but not limited to data validation processes and tools, plan and terms of reference for data managers, and use of EPTS data for reporting and the QI program.

Clinical Services (enrollment and retention)

7) The GRM/MOH should develop a better staff development and retention package or plan to guarantee staff retention in post for at least three years to allow for consolidation of experience.

Also, increase staffing levels to meet increasing client load, especially for HIV/AIDS services.

8) The GRM/MOH, with partners’ support, should invest in improving HF physical structure and space.

x Performance Evaluation of the CHASS-SMT Project

9) Implement an augmented MOH strategy of a lay counselors’ workforce by recruiting and training lay counselors or retraining case managers and Agentes Polivantes Elementer (APEs) as lay counselors.

Some counselors should be stationed at HFs and one in each community to provide services such as adequate counseling for ART initiation, community sensitization against HIV stigma, psychosocial support to HIV-positive pregnant women to disclose HIV status, psychosocial support to GAACs and individual PLHIV, and adherence counseling.

10) Reinvigorate linkages between HF and community-based services, and promote regular information exchange for inter-referrals and health information messaging to communities:

Conduct operations research to map and improve the functioning of community linkages.

Explore opportunities to partner with private telephone companies to introduce mHealth for direct communication between HFs and the community–CBOs, faith-based organizations, activists, traditional healers, religious leaders, etc.–using mobile phones or phone credit, patient monitoring devices, PDAs and tablets, to assure continuous information exchange, data collection and reporting, and health messaging.

Pilot and expand HIV service coverage in underserved rural areas, using mobile clinics with mHealth to provide HCT, HIV care, ARVs, ANC/PMTCT, CD4 testing, etc.

11) Integrate HIV prevention, care and support in adolescent health services at HFs, with emphasis on adolescent girls. Develop a school-based integrated health program with adolescent health and HCT services.

12) Improve HIV testing and ART initiation among infants through training all HF health workers in: PCR sample-taking to provide HIV testing in all pediatric entry points, including the children of patients on ART; linkage of records in labor/delivery units and at-risk child consultations; and adequate counseling of pregnant women. In addition, distribute SMS printers to rural HFs.

Community Linkages

13) Reinforce community-HF linkages, and introduce mHealth in mobile clinics and HFs.

14) Partner with employer-based health programs, to provide training and technical support to institute worksite HCT testing, referral, and dispensing ARVs, particularly for male workers.

GBV Services

15) Strengthen the GBV community linkages and work with a NGO/CBO that specializes in gender issues to support community activities.

Sub-agreements

16) Establish a budget item for non-HIV-related services: This can fund emergencies or disease outbreaks that are not necessarily within the scope of CHASS.

17) Improve on the reimbursement process: Review internal financial controls to make them realistic for GRM collaborations.

Performance Evaluation of the CHASS-SMT Project 1

1 EVALUATION PURPOSE AND

QUESTIONS

1.1 EVALUATION PURPOSE

The five-year Clinical HIV/AIDS System Strengthening Project in Sofala, Manica and Tete provinces

(CHASS-SMT) was funded by the United States Agency for International Development (USAID). It was implemented by Abt Associates Inc. from November 2010 to July 2015. The project’s goal was to improve HIV/AIDS and related primary health services in the three provinces by strengthening the primary health care system.

This performance evaluation of CHASS-SMT was commissioned by USAID, and its purpose was to determine how the project activities were performing relative to their objectives. The evaluation was to provide an objective view of progress toward the expected results. Its outcomes will inform the transition to the future activity that will support USAID system strengthening and clinical service delivery activities. The main audience for the evaluation is USAID. In addition, results will be shared with

Abt Associates, the MOH, U.S. Government agencies, and other stakeholders. The results will also be made available on the Development Experience Clearinghouse.

The objectives of the CHASS Evaluation are to:

Assess CHASS-SMT achievements, emphasizing objectives and activity- and project-level results

Identify implementation successes, as well as any internal and external constraints that hindered the implementation of planned activities

Propose recommendations for future directions of CHASS-SMT and for future activities in system strengthening and service delivery to support improved performance in addressing the HIV epidemic, in line with the GRM HIV/AIDS acceleration plan.

An additional goal was to use mobile technology, when possible and appropriate, to increase the efficiency, transparency and accuracy of performance data, and to take advantage of multiple data sources (pictures, videos, GPS data).

1.2 EVALUATION QUESTIONS

The three main evaluation question areas include 10 evaluation questions. All question areas were answered fully and completely, underscoring both positive and negative outcomes. Where sufficient quantitative data were available, gender analysis was conducted. In order to accomplish the above-identified evaluation objectives, the evaluation sought to answer the following questions:

Question Area 1: Project Achievements and Challenges

1) What are the strengths and weaknesses of the activities, as seen by the implementing partner staff, Provincial Directorate of Health (DPS), District Directorate of Health and Social Welfare

(SDSMAS and chief medical officer), HFs, USAID, and the USAID-funded Community Care

Project (PCC), and how can weaknesses be improved, according to these stakeholders?

2) Where has the most progress been seen in strengthening systems (e.g., planning, financial management, supply and logistics, information systems)?

3) Which activities or project components will be most feasible to transition from the project to the GRM?

2 Performance Evaluation of the CHASS-SMT Project

Question Area 2: Linkages

4) To what extent has the project been able to create and strengthen linkages between HFs and communities to allow for increased service uptake, specifically in the areas of:

(a) Community-based counseling and testing to treatment (for both men and women)

(b) Retention of pre-ART and ART patients (through the use of adherence groups, active case-finding, and other community groups)

(c) Knowledge, demand, and access of services by men

5) What constraints have the activities faced in improving retention of patients in pre-ART and

ART?

6) What has been the most effective method found by the project to improve retention?

Question Area 3: Health System Strengthening

7) To what extent has HF management (improved patient flow, etc.) improved over the life of the project?

8) To what extent is data management capacity built at the HFs with regard to HIV/AIDS registry data?

9) To what extent has knowledge (community and health worker) and utilization of GBV services increased over the life of the project?

10) What are the benefits and challenges of the activities’ model of working with government (at provincial and district levels) through sub-agreements?

Specific questions that guided the performance evaluation are stated in the evaluation matrix included in

Annex I.

Performance Evaluation of the CHASS-SMT Project 3

2 PROJECT BACKGROUND

2.1 GENERAL CONTEXT

In July 2009, USAID/Mozambique launched a request for applications for a results-oriented five-year project to improve HIV clinical services in Manica, Niassa, Sofala and Tete provinces within a strengthened, comprehensive primary health care system. The CHASS project was designed to address the HIV situation in Mozambique and focused on three components:

a) Improving service quality in six important areas: HCT, laboratory services, prevention of mother-to-child transmission (PMTCT), adult care and treatment, pediatric care and treatment, and the prevention, diagnosis and treatment of HIV-TB co-infection

b) Enhancing program linkages and integration to provide a continuum of accessible services, including MCH and reproductive health (RH) services, within facilities and between facility and community-based services

c) Creating stronger and more sustainable Mozambican systems and institutions

At the time of project design in 2010, the average HIV prevalence in Mozambique was estimated at 16 percent nationwide (ANC survey in pregnant women aged 15-49 years). Nearly 1.6 million people were living with HIV, and nearly half of all HIV-infected (48.4 percent) were identified at the time as having active tuberculosis. Within the Central Region, Sofala and Manica provinces have the highest prevalence, 23 percent and 16 percent respectively. Tete was identified as having a very mature epidemic, and existing infrastructure that was unable to accommodate the numbers of patients requiring care and treatment. Niassa was considered a particularly underserved province, with vastly inadequate infrastructure.

To address these issues, two separate agreements were awarded to two implementing partners: FHI

360 for CHASS Niassa, and Abt Associates Inc. for CHASS-SMT (Sofala, Manica and Tete). Activity funding for CHASS-SMT was $111,545,465 for the period of November 1, 2010 to October 31, 2015 and was to support USAID/Mozambique’s Country Assistance Strategy’s (CDCS DO4) priority goal number three, “Improved health of Mozambicans.” It was to contribute specifically to the following focal areas in USAID’s Health Results Framework: (a) Improved access to and delivery of quality integrated services; (b) Increased adoption of healthy behaviors and informed use of services; and (c) Strengthened health systems. Implementation of CHASS-SMT started in early 2011.

After the two CHASS activities were awarded, the MOH developed the HIV Acceleration Plan in 2011.

In addition, both activities included programming to address gender-based violence (GBV) within the

HIV platform, with a total life-of-project funding of approximately $1.5 million. GBV fosters the spread of HIV by limiting a person’s ability to negotiate safe sexual practices, disclose HIV status, access services

(due to fear of reprisal), adhere to treatment and access care.

Activities were implemented in all four provinces to:

Expand and improve coordination and effectiveness of GBV prevention efforts

Improve policy implementation in response to GBV

Improve the availability and quality of GBV services

2.2 OVERVIEW OF CHASS-SMT IMPLEMENTATION

CHASS-SMT aimed at achieving the following four objectives:

4 Performance Evaluation of the CHASS-SMT Project

1. Strengthen Mozambican health systems and institutional capacity to provide high-quality services and ultimately receive and manage direct support from the U.S. Government

2. Improve integration of HIV and related primary health care services and linkages between the community and the health system

3. Increase demand, use and provision of high-quality HIV services

4. Increase coverage of care and treatment as well as PMTCT services

CHASS-SMT followed the Health Alliance International (HAI) project that was mainly focused on delivery of HIV/AIDS and related services at the HF level. Building of local leaders’ capacity and HSS were done at both the provincial and district levels. Integral to this was the gradual transition of resources and responsibilities to the GRM staff at these levels. The project operated within the framework of GRM decentralization plan for health services delivery as well as MOH protocols and guidelines, and it aimed to align its work-plans with the Planos Económicos e Social (economic and social plan) and Plano Económico e Social Operacional Distrital (District Social Economic Operational Plan).

After 18 months of implementation, the project restructured its operational model to focus, support and integrate technical support to improve capacity of the districts (SDSMAS) to manage and supervise health programs and HIV/AIDS service delivery. The revised model organized project staff into multidisciplinary teams, or Equipas Polivalentes (EPs), to collaborate with DPS to integrate technical assistance (TA) to SDSMAS and health facilities. TA by EPs involved short training, mentorship, supportive supervision and clinical tutoring of the DPS, SDSMAS and HF staff. They also developed tools, standard operating procedures and guidelines.

Most project activities were achieved through SAs between the project and the three provinces, 36 districts and eight CBOs. In addition to TA and financial support, the project procured office materials, equipment (for laboratory and information systems) and cars or motorbikes, and it did some basic infrastructural rehabilitation. A basic package of clinical and lab consumables and equipment (kit básico) was distributed.

CHASS-SMT trained new health workers through SAs with training institutions in the three provinces.

Since the government takes more than a year to clear new staff into their system, the project provided 12 months of gap funding for these new staff.

The project also used s graduation path strategy with which EPs assisted SDSMAS to systematically identify specific service delivery and management needs and gaps using a 23-item Standards Compliance

Tool. This was followed by technical support from EPs to address identified needs. These efforts contributed towards the districts’ ‘graduation’ from direct TA and subsequent ability to receive and manage support through funding from SAs only. Further, the project developed a clinical monitoring and tutoring tool, used jointly with DPS and SDSMAS staff to monitor HF service delivery.

Some of the USAID-funded projects CHASS-SMT worked with included PCC and ROADS for community mobilization activities; TB Care for TB/HIV co-infection activities; and SCMS/SPS/Deliver for supply chain logistics in the three provinces.

Performance Evaluation of the CHASS-SMT Project 5

3 EVALUATION METHODS AND

LIMITATIONS

3.1 EVALUATION METHODS

This performance evaluation was based on a non-experimental design and was executed by a team independent and external to the CHASS project. The evaluation used a mixed-methods approach, utilizing mostly qualitative data collection and evaluation methods. The evaluation involved extensive desk review and analysis of existing quantitative project data and documentation, and primary collection and analysis of qualitative data. Quantitative data were extracted from the quarterly progress reports

(QPR) and project datasets held by the USAID team. The evaluation was conducted September 4-28, 2015.

Primary data collection methods included key informant interviews and observational analysis at HFs.

The key informant interviews served to: (a) validate and, where possible, verify project approaches or activities, interventions and achievements, extent of gains and changes over time; and (b) identify gaps and weaknesses in project activities or performance. Key informants included staff from the MOH, DPS, SDSMAS, HFs, CHASS implementing partner and CBOs that participated in the project. Some members of the USAID/Mozambique team were also interviewed. The structured interview guides are included in

Appendix III. The survey instruments were pre-tested in Beira city before fieldwork began. Interviews with patients were not within the remit of the scope of work.

The evaluation used multi-channel data collection, using mobile technology when possible and paper-based methods when handheld computers/tablets were not appropriate. The team noted that the use of mobile technology is feasible, and some of the members handwrote their field notes using the ODK program, enhanced with a stylus beta program. However, for the mobile technology to be most useful for collection of qualitative data, it was noted that the key informant and focus group guides should be semi-structured. Also, for the locally recruited research assistants, more time was required for extensive practice of handwriting on the tablets. This was not possible in this evaluation due to time constraints.

3.2 SITE SELECTION AND DATA COLLECTION

Districts were stratified based on their performance on SAs, relative to the average performance for their province in 2014. Within each stratum, a random sample was taken to represent dominantly rural, semi-urban and urban districts. Data collection took place in the following locations: Beira, Nhamatanda and Dondo in

Sofala province; Chimoio, Gondola and Manica in Manica province; and Tete and Changara in Tete province.

Within each district, the team selected HFs purposively to represent low and high client volumes, extent of

CHASS activities at the site and rural or urban location. Fifteen HFs were visited, and various categories of health workers were interviewed (Table 1). Categories of respondents are summarized in Table 2. The

DPS/SDSMAS senior staff included the directors and medical chiefs, and the HF management team included the HF in-charges. Respondents are listed in Appendix III.

6 Performance Evaluation of the CHASS-SMT Project

Table 1: Selected districts and health facilities

Province District Location

District’s performance on SA activities relative to the province’s performance in 2014 Name of HFs selected

Sofala

Beira Urban Below Munhava and Ponta Gea

Nhamatanda Rural Above Nhamatanda Sede and Tica

Dondo Semi-urban Below Dondo Sede & Mafambisse

Manica

Chimoio Urban Below 1 de Maio

Manica Semi-urban Above Manica DH and Messica

Gondola Rural Below Gondola Sede and Amatongas

Tete

Nucleo

Provincial de

Combat ao

Sida

Changara Rural Average Changara Sede, Chipembere

Tete Urban Above Bairro Francisco Manianga (No.

3) and Bairro Muthemba (No. 4)

A total of 18 CHASS-SMT staff, eight PCC staff and nine staff from CBOs that worked closely with

CHASS-SMT were interviewed. In addition, two staff from the MOH and five from USAID were interviewed.

Table 2: Categories of key informant interview respondents

Group Sofala Manica Tete Total

DPS/SDSMAS senior staff 4 4 2 10

HF management team 6 5 4 15

HIV focal points/technicians (at DPS, SDSMAS and

HF level) 7 7 4 18

SMI focal points/technicians (at DPS, SDSMAS and

HF level) 7 8 4 19

HSS/monitoring and evaluation focal points (at DPS

& SDSMAS)

3 3 3 9

Logistics/finance (at DPS and SDSMAS) 2 0 2 4

GBV focal points (at DPS, SDSMAS and HF level) 3 3 2 8

Other (lab, pharmacy) 1 2 3 6

Total 33 32 24 89

Percent of target 100% 97% 86% 95%

3.3 DATA ANALYSIS

The analysis included data in the performance monitoring system and program reports, and included trend analysis of results and progress made on planned results. The qualitative data were analyzed using a qualitative data analysis matrix. Information from this analysis was integrated or triangulated with the available quantitative data from CHASS documents and reports.

The analysis was guided by the 10 evaluation questions listed in the scope of work (Appendix I) and in

Section 1 of this report. For questions where gender-related data or information are relevant, gender-related differences are presented. The end summary of the analysis was focused on the priority issues

Performance Evaluation of the CHASS-SMT Project 7 for CHASS to address and main lessons learned, based on the answers provided in examining the evaluation questions.

3.4 STUDY LIMITATIONS

Limitations of the evaluation include the following: First, the evaluation was initially planned to take place in the project’s fourth year, but instead was conducted at the end of the project. Secondly, the evaluation team was not able to ascertain consistent factors responsible for the project’s success and weaknesses, as the project evolved substantially over time. Therefore, some of the factors reported are from a summative perspective. The findings reported here are according to what the interviewees told the evaluation team. The likelihood of interviewee bias cannot be ruled out. Lastly, the time allocated for the evaluation was limited compared to the complexity of the project, and this, in turn, constrained the detailed evaluation of each of the components within each province. To some extent, this also constrained the sample size for the key informant interviews. Further, because the evaluation was conducted at the project’s end, when staff were transitioning, some of the interview scheduling was delayed.

8 Performance Evaluation of the CHASS-SMT Project

4 FINDINGS, CONCLUSIONS AND

RECOMMENDATIONS

4.1 FINDINGS

The CHASS-SMT project suffered a slow start in the first year, partly because the team was trying to reorient the DPS to the concepts of HSS and working jointly through SAs. In the third year, the project had to change its approach from HSS at DPS only to HSS at both DPS and the districts. Despite this, the project achieved most of its objectives, including improved HSS (project objective 1) and coverage of

HIV/AIDS services. The evaluation findings on project achievements and challenges are organized in the subsequent sections according to the evaluation questions.

Question 1: What are the strengths and weaknesses of the activities, and how can weaknesses be improved, according to the interviewed stakeholders?

This section summarizes strengths and weaknesses, organized according to project objectives 1-4. The team notes that the achievements listed should not solely be attributed to CHASS-SMT strategies, considering the interventions of other actors in the provinces.

Strengths and weaknesses of activities to improve clinical service delivery

HIV care and support: In line with MOH guidelines and protocols, CHASS-SMT provided training and technical support to DPS’ and SDSMAS’ efforts toward the decentralization and integration of ART services at the primary health care level, with an emphasis on introducing ART at HFs in peripheral areas. As a result, 77 new ART sites were opened. In addition, the project provided SDSMAS and HFs with technical support to improve logistics and transport of CD4, PCR samples and some medications, and it provided a Kit Basico with basic equipment for SDSMAS and HFs in selected districts. Health workers were also trained and provided with logistics for quality improvement/assurance (QI/QA) activities at HFs and labs.

Health facility clinical staff emphasized that technical support by EPs contributed to structuring and organization of HIV services, data review and increased demand for HIV care and treatment. The strength of the technical support was that it integrated many aspects, and not clinical issues only. These included tutoring in data quality, improved patient flow, record keeping and sharing, stock management and staff motivation, among others.

Through training and technical support, enrollment of patients on ART increased threefold between

2010 and 2014 (Figure 1). Similarly, the number of individuals currently enrolled on ART increased nearly threefold from 57,894 in 2011 to 155,588. These were more than 100 percent of the project’s targets. Children represented 1 percent of ART enrollees in 2011, increasing to 8 percent by 2014.

More women than men enrolled for ART in all three provinces, which is consistent with stakeholder reports. Further, the percentage of clients currently on ART who are men declined from 33 percent in

2011 to 29 percent in 2014 (Figure 1). This is partly due to scale-up of Option B+ among pregnant

“We saw improved performance in four HFs and ART increase to 11 sites; their indicators improved over time.” (Director, DDS, Messica, Manica)

“With structuring and organization of Clinical HIV,TB, PMTCT, mapping HFs using indicators, team data review and database graphs, we detected mistakes and could improve.” (Monitoring and Evaluation

Technician, Nhamatanda)

Performance Evaluation of the CHASS-SMT Project 9 women. This is expected, as HIV-positive men are mainly reached through HCT programs, but HIV-positive women are reached through both HCT and PMTCT programs.

Figure 1: Patients already on ART and newly enrolled

N u m b e rs

Integration of HIV services in primary care: CHASS-SMT provided assistance to SDSMAS and HFs to integrate HIV services for pregnant women and also for TB/HIV co-infected patients, adopting the one-stop service delivery model. MCH nurses were trained to test and counsel for HIV, take blood samples and prepare dry blood spots (DBS) for PCR, prescribe ARVs to HIV-positive patients, provide

Option B+, promote Mother-to-Mother activities onsite, and promote male partner involvement in testing in ANC/PMTCT services. Chief medical officers and MCH nurses reported how service delivery in ANC improved with the one-stop model.

The community activities led to an improved number of women attending ANC, from 223,168 in 2012 to 295,267 in 2014, of which 87 percent were tested for HIV in 2012 and 88 percent in 2014. At least

86 percent of the HIV-positive pregnant women were initiated on ART (Figure 2). However, the percentage of children born to HIV-positive mothers who receive a PCR test within 12 months is still low (Table 3).

HCT established at HFs: To increase facility-based HCT services, the project provided logistics for counseling, testing and referral at all points of entry at the HFs. Between 2011 and 2014, 966,155 individuals were tested at HFs, and the 142,081 who tested positive were referred into pre-ART care.

“One-stop approach worked very well; there is no waiting time and treatment of patients improved.” (Nurse, Tete)

“With one-stop approach, our services are integrated at each HF; we nurses test women and babies.” (Medical

Chefe, Nhamatanda)

“We were successful in bringing all pregnant women who are HIV-positive to ART, and babies born are negative.

And since 2013, we provide ART at maternity ward for positive pregnant women.” (MCH nurses in Gondola and

Changara Districts)

43,939 37,094

69,717

107,91313,955 23,922

32,481

47,675

33% 31% 30% 29%

8% 8% 8% 8%

-25%

-15%

-5%

5%

15%

25%

35%

45%

20,000

40,000

60,000

80,000

100,000

120,000

140,000

160,000

180,000

2011 2012 2013 2014

New on ART Already on ART

% Current ART clients who are men % Current ART clients who are children

10 Performance Evaluation of the CHASS-SMT Project

Sixty percent of HCT was through the provider-initiated approach. Health workers emphasized that community mobilization contributed significantly to the increases.

Figure 2: Key PMTCT indicators

A summary of other findings is presented in Chart 1.

Chart 1: Clinical services weaknesses and recommendations

Findings Recommendations

1) Low retention rates (poor quality of counseling due to insufficient number of staff, long distances, HIV status disclosure issues, stigma)

2) Provincial, district and HF stakeholders report insufficient number of staff to manage improved activities, meet increased demands and maintain high-quality services.

For example, MCH nurses in some HFs reported attending to…