ATTACHMENT J.6- CHASS SMT end of project report.pdf

Submitted January, 2016

Clinical HIV/AIDS Services Strengthening Project, Sofala, Manica, Tete Provinces Cooperative Agreement Nº 656-A-00-11-00021-00

The Clinical HIV/AIDS Services Strengthening Project, Sofala, Manica, Tete Provinces is funded by the United States Agency for International Development and implemented by Abt Associates Inc. in partnership with FHI360.

CHASS-SMT

END OF

PROJECT

REPORT

ii

CHASS-SMT

END OF PROJECT

REPORT

DISCLAIMER

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States government.

iii

TABLE OF CONTENTS

Acronyms ..........................................................................................................................vii

1 Introduction

2 Results and Achievements

Key PEPFAR Indicators and Targets.........................................................................22.1 Access 42.2 Quality 52.3 Sustainability .............................................................................................................62.4

3 Implementation and Management

Strategies and Activities ............................................................................................93.1

3.1.1 District Approach

3.1.2 Graduation Path

3.1.3 Sub-agreements

3.1.4 Clinical Tutoring

3.1.5 Re-engineering of Patient Flow

Implementation and Management Challenges and Lessons Learned......................183.2

3.2.1 Start-up and DPS Buy-in

3.2.2 Staffing 19

3.2.3 Changes in Scope, Priorities, Strategy, and Management

4 Remaining Challenges and Recommendations

Pediatric ART and Retention ...................................................................................214.1 Facility-level Patient Data and Registration .............................................................214.2 Laboratory224.3 Community Linkages ...............................................................................................234.4 System Capacity .....................................................................................................244.5

4.5.1 Facility Level

4.5.2 District Level

5 Conclusions

Annex I: District Sub-agreement Indicators

Annex II: List of CHASS-SMT Partners

Annex III: End-of-project Documentation

Annex IV: Study Report - Strengthening District Health Systems and HIV Service Delivery Outcomes in Mozambique: Findings from the CHASS-SMT Project

Annex V: Study Report: Evaluation of Performance-based Incentives Pilot

Annex VI: Final Performance Monitoring Plan Indicators iv

LIST OF TABLES

Table 1. Examples of project achievements by objective Table 2. Summary of life- of-project performance on six key PEPFAR indicators Table 3. Package District Approach Interventions Applied to Priority vs. Non-Priority Districts Table 4. Indicators analyzed in study and results Table 5. Improvements in PCR Turnaround Time v

LIST OF FIGURES

Figure 1. Number of ART Sites by Province Figure 2. Patients Active on ART by Province Figure 3. Improvements in Key Service Quality Indicators from Year One to Year Five Figure 4. Improvements in 12-month ART Retention Rates Figure 5. Average Graduation Path Scores by Characteristic for Each AssesSment (number of districts) 7 Figure 6. Percent of Disbursed DPS Sub-Agreement Funding Liquidated vii

ACRONYMS

AOR Agreement Officer Representatives

ANC Antenatal Care

ART Antiretroviral Therapy

CBO Community-Based Organization

CD4 Cluster of Differentiation

CHASS-SMT Clinical HIV/AIDS Services Strengthening Project in Sofala, Manica, and Tete provinces

COP Chief of Party

DPS Direcção Provincial de Saúde (Provincial Health Directorate) eMTCT Elimination of Mother-to-Child Transmission

EP Equipa Polivalente (Polyvalent Team)

EPTS Electronic Patient Tracking Systems

GAAC Grupo de Apoio e Adesão da Comunidade (Community Support and Adherence Group)

GBV Gender-based Violence

HIV/AIDS Human Immunodeficiency Virus/Acquired Immunodeficiency Syndrome

MOH Ministry of Health

PCR Polymerase Chain Reaction

PEPFAR President’s Emergency Plan for AIDS Relief

PIMA CD4-analyzing machine

SDSMAS Serviços Distritatis de Saúde Mulher e Acção Social (District Health Directorates)

SIMAM Sistema de Informação de Medicamentos e Artigos Médicos (Pharmaceuticals and Medical Commodities Information System)

SMS Short Message Service

SOPs Standard Operating Procedures

TB Tuberculosis

USAID United States Agency for International Development

VMMC Voluntary Male Medical Circumcision

1 INTRODUCTION

The Clinical HIV/AIDS Services Strengthening Project in Sofala, Manica, and Tete (CHASS- SMT) in Mozambique was a five-year project (2011-2015) funded by the United Stated Agency for International Development (USAID) and implemented by Abt Associates (Abt) and partner FHI360. The goal of the project was to improve HIV clinical services in Sofala, Manica, and Tete within a strengthened comprehensive primary health care system. CHASS-SMT was the follow-on to a project that provided intensive clinical support to HIV and AIDS services in the provinces of Sofala and Manica. USAID/Mozambique tasked CHASS-SMT with pivoting support to the HIV and AIDS response in the Mozambique’s central provinces from an emergency response under the President’s Emergency Plan for AIDS Relief (PEPFAR) I, to focus on improving sustainability by strengthening local health systems under PEPFAR II. The project’s three original objectives were:

Strengthening Mozambican health systems and institutional capacity to provide high-1.

quality services and ultimately receive and manage direct support from the United States Government

Improving integration of HIV and related primary health care services and linkages2.

between the community and the health system

Increasing demand, use and provision of high-quality HIV services3.

In Year Two of the project (2012), USAID requested that CHASS-SMT support the Mozambican Ministry of Health’s (MOH) strategy of decentralizing HIV and AIDS services to peripheral health facilities. In response to this request, the project added a fourth objective:

Increasing coverage of services for HIV care and treatment and elimination of mother-to-4.

child transmission (eMTCT)

Originally, the project consisted of two distinct components: (1) a health systems strengthening component applied mostly at the provincial level and aimed at objective 1, and (2) a clinical component, applied mostly at the health facility level, and aimed at objectives 2-4. In 2013, the Mozambique MOH’s National HIV Prevention, Care and Treatment Acceleration Plan targeted districts with the highest unmet need for HIV treatment to intensify efforts to expand access to and quality of treatment. Based on lessons learned during the first years of implementation, and to better respond to the MOH’s Acceleration Plan, CHASS-SMT shifted its implementation strategy in 2013. Under the new strategy, the District Approach, CHASS-SMT integrated its health systems strengthening and clinical activities and focused assistance at the district level, to build the capacity of districts to provide quality expanded HIV services. The District Approach and its elements are described in detail in other CHASS-SMT End-of-Project (EOP) documentation1. This report focuses on the results, challenges, lessons learned, and recommendations of the CHASS-SMT project.

1 Please note that the Results Brief distributed at the EOP event in September, 2015 in Maputo included only data until July 2015. The results described in this document are updated with data through the end of the project (September 2015).

2 RESULTS AND

ACHIEVEMENTS

Project data provide good evidence that CHASS-SMT succeeded in achieving its four objectives. Table 1 provides examples of indicators that demonstrate project achievements under each objective. The following sub-sections highlight the most significant life-of-project results by four categories: key PEPFAR indicators and targets; access, quality, and sustainability.

TABLE 1. EXAMPLES OF PROJECT ACHIEVEMENTS BY OBJECTIVE

Objective Achievements

1. Strengthening Mozambican health systems and institutional capacity to provide high-quality services and ultimately receive and manage direct support from the United States Government

Priority districts improved their capacity to manage health services functions from 38% to 83% compliance with MOH standards

This improvement showed a positive, statistically significant association with improvements in key service-delivery indicators

District capacity to manage direct funding improved from an average of 47% to 80% execution of planned activities in 21 months

2. Improving integration of HIV and related primary health care services and linkages between the community and the health system

One Stop Model for tuberculosis (TB)/HIV and eMTCT introduced in 215 health facilities

41,223 lost-to-follow-up patients placed back on anti-retroviral therapy (ART) through community outreach

3. Increasing demand, use and provision of high-quality HIV services

Percent of HIV+ pregnant women receiving eMTCT increased from 59% to 92%

Percent of TB/HIV+ patients initiating ART improved from 22% to 81%

Adult ART 12 month retention rate improved from 65% to 71% Pediatric ART 12 month retention rate improved from 64% in to

71%

4. Increasing coverage of services for HIV care and treatment and elimination of mother-to-child transmission (eMTCT)

129 new ART sites opened 162,707 new patients enrolled in ART 101,253 new HIV+ pregnant women received antiretrovirals for eMTCT

KEY PEPFAR INDICATORS AND TARGETS2.1

Table 2 below summarizes CHASS-SMT’s life-of-project performance on key PEPFAR indicators. Adult and pediatric anti-retroviral therapy (ART) 12-month retention rates (indicators 1 and 2) improved slowly but steadily over five years. However, by Year Five, both rates were 71percent, still short of the target of 85 percent. This is consistent with national-level trends in retention rates, which are notoriously difficult and slow to improve. New enrollment on ART also increased steadily, with Year Five values reaching three times or more the Year One values. By Year Two CHASS-SMT consistently achieved greater than 90 percent performance against adult enrollment yearly targets through the end of the project. Pediatric enrollment remains the biggest challenge. In Year Five the project reached 64 percent of its new pediatric enrollment target. Other provinces in Mozambique face the same challenge in accelerating new pediatric enrollment. Section 4.1 discusses the remaining challenges with retention and pediatric enrollment.

CHASS-SMT showed consistently high performance on the eMTCT indicator (indicator 6). After a slow start in Year One, the project maintained the improvement achieved in Year Two, surpassing its PEPFAR target from then on. The TB/HIV indicators also show positive life-of-project trends. The number of HIV positive patients screened for TB (indicator 5) increased steadily, reaching 151 percent of the PEPFAR target in Year Four and 102 percent in Year Five.

Finally, the number of TB/HIV+ patients on ART (indicator 7) also increased with every year, as well as achievement of targets. In Year Three CHASS-SMT reached 85 percent of the PEPFAR target, rising to 110 percent in Year Four and 102 percent in Year Five.

TABLE 2. SUMMARY OF LIFE- OF-PROJECT PERFORMANCE ON SIX KEY PEPFAR INDICATORS

Indicator

Performance (% yearly PEPFAR target achieved)

Year 1 Year 2 Year 3 Year 4 Year 5

1. Proportion of ART patients <15 years known to be alive and in treatment after 12 months of treatment (Indicator: T-

ARV.04.)

64% (80%)

64% (80%)

66% (77%)

68% (80%)

71% (84%)

2. Proportion of ART patients 15+ years known to be alive and in treatment after 12 months of treatment (Indicator: T-

ARV.04.)

65% (81%)

64% (80%)

73% (86%)

67% (79%)

71% (84%)

3. Number of patients <15 years with advanced HIV infection newly enrolled on ART during the reporting period (Indicator: T-ARV.02.)

1,344 (53%)

2,357 (88%)

4,960 (99%)

4,077 (27%)

3,987 (64%)

4. Number of patients 15+ years with advanced HIV infection newly enrolled on ART during the reporting period (Indicator: T-ARV.02.)

12,611 (74%)

21,392 (119%)

29,709 (90%)

43,197 (148%)

40,963 (99%)

5. Number of HIV-positive patients who were screened for TB at last visit in HIV care or treatment setting (Indicator: C-

CLC.04.)*

137,136 (285%)

140,062 (175%)

98,187 (69%)

216,360 (151%)

433,156 (102%)

6. Number of HIV-positive pregnant women who received ART to reduce risk of mother-to-child-transmission (already on ART or initiated ART in ANC) (Indicator:

P-MTCT .06.04.)

13,806 (64%)

21,975 (127%)

22,495 (109%)

20,729 (114%)

22,990 (102%)

7. Number of HIV-positive TB patients who are known to have started ART (Indicator:

C-TB.04.)

1,774 (34%)

2,972 (63%)

2,904 (85%)

5,205 (110%)

5,453 (102%)

* The tracking methodology for this indicator changed in the third quarter of Year Two, likely explaining the apparent sharp decrease from Year Two to Year Three.

Jan-Mar 2011 EOP

Jan-Mar 2011 EOP

ACCESS2.2

The achievement of project objectives 1-4 contributed towards increasing access to high quality HIV and AIDS services in Sofala, Manica, and Tete. The main activities that CHASS-SMT implemented to expand access to HIV and AIDS services included:

Supporting the opening of new ART and eMTCT sites by:

o Providing training and clinical tutoring to health facility staff on ART and eMTCT protocols, including use of registries and data reporting o Supporting critical infrastructure improvements and providing basic equipment and materials

Providing clinical tutoring and technical assistance to health facility staff to improve the delivery of services, including improving patient information systems and data flow

Improving the linkages between communities and facilities by supporting community-based organizations (CBOs) to work with facility staff on demand creation of HIV services and the active outreach of ART patients lost to follow up

The above activities led to the following results:

129 new ART sites opened

162,707 new patients placed on ART, of which 16,700 (10%) are children

101,253 new HIV+ pregnant women placed on eMTCT

41,223 lost-to-follow-up patients recovered back into ART through active outreach

1,952,461 people tested for HIV, of which 10 percent (195,857) were positive:

o Community counseling and testing tested 268,371 people, referring the 19,740 (7%) who were positive to health facilities for care o Monthly average of people tested through provider-initiated counseling and testing increased from 39,604 in the first year of the project to 193,697 in the last year

Introduction of voluntary medical male circumcision (VMMC) services, with 19,628 VMMCs conducted over five quarters

Introduction of health services for gender-based violence (GBV) survivors in 75 new facilities; provision of post-GBV care to 4,509 people

FIGURE 1. NUMBER OF ART SITES BY

PROVINCE

FIGURE 2. PATIENTS ACTIVE ON ART BY

PROVINCE

QUALITY2.3

Project objectives 2 and 3 also sought to improve the quality of HIV and AIDS and related health services. Project activities for improving health service quality were sometimes the same, or closely linked to the activities for improving access:

Providing clinical tutoring, technical assistance, and targeted interventions at the health facility level to improve provider knowledge and skills and systems and processes

Strengthening district health managers’ capacity to manage health programs, laboratory services, and community engagement

Supporting critical infrastructure improvements and providing basic equipment and materials

Improving the linkages between communities and facilities, with a focus on community adherence support groups (GAACs) to improve ART retention

These activities contributed to the following improvements in service quality from Year One to Year Five:

Proportion of pregnant women in antenatal care (ANC) with known HIV status improved from 88 to 96 percent

Percent of HIV+ pregnant women receiving antiretrovirals (ARVs) for eMTCT increased from 59 to 92 percent

Percent of HIV+ pregnant women receiving Option B+ increased from 14 to 83 percent

Percent of TB/HIV+ patients on ART improved from 22 to 81 percent

FIGURE 3. IMPROVEMENTS IN KEY SERVICE QUALITY INDICATORS FROM YEAR ONE TO

YEAR FIVE

12-month ART retention:

o Adult ART retention rate improved from 65 percent to 71 percent o Pediatric ART retention rate improved from 64 percent to 71percent o 12-month pre-ART retention:

92 81

Known HIV status in ANC HIV+ PW on eMTCT TB/HIV+ on ART

Year 1 Year 5 o Adult pre-ART retention rate improved from 28 percent2 to 70 percent o Pediatric pre-ART retention rate improved from 36 percent2 to 80 percent

FIGURE 4. IMPROVEMENTS IN 12-MONTH ART RETENTION RATES

Referral completion rates for community-facility referrals improved from 37 percent to 67 percent

5,293 new GAACs were formed with 20,659 members

SUSTAINABILITY2.4

In addition to expanding access to and quality of HIV and AIDS services, CHASS-SMT was also tasked with strengthening Mozambican systems to ensure sustainability of service provision and capacity to receive direct funding. This was the aim of project objective 1. Although true sustainability can only be demonstrated over the long term, CHASS-SMT achieved significant, measurable milestones that strengthened the Mozambican health system at different levels and contributed to sustainability. Key project activities to strengthen the health system included:

Providing sub-agreements to all three Provincial Health Directorates (DPS) and 36 District Health Directorates (SDSMAS) in Sofala, Manica, and Tete

Providing intensive technical assistance to DPS and SDSMAS on sub-agreement management

Strengthening SDSMAS management capacity through the Graduation Path

Supporting Human Resources for Health by funding pre-service training of new health workers and providing gap-year funding for their immediate deployment

Providing financial and technical support for the expansion of essential MOH systems such as electronic patient tracking system (EPTS) and pharmaceuticals and medical commodities information system (SIMAM)

Directly procuring essential equipment, materials, and repairs for health facilities

In the second year of the project.

65 64

71 71 70

Adult ART Pediatric ART Adult pre-ART Pediatrict pre-ART

Year 1/2 Year 5

Key results from these activities include:

Priority districts improved their overall management capacity from 45 percent to 79 percent compliance with MOH operating standards

There was a positive, statistically significant relationship between improved district capacity to manage services functions (health programs, laboratory, and community mobilization), and the following access and quality indicators3 (five out of six indicators analyzed):

o Number of new ART enrollees o Number of new pre-ART enrollees o New enrollees in pediatric HIV care o Percent of pregnant HIV+ women in ANC on ART for eMTCT o Percent of TB/HIV+ patients on ART

FIGURE 5. AVERAGE GRADUATION PATH SCORES BY CHARACTERISTIC FOR EACH

ASSESSMENT (NUMBER OF DISTRICTS)

More than US$16 million provided to health facilities in essential materials and equipment

US$4.5 million provided to DPS’ through sub-agreements to support essential functions

Results of the study “Strengthening District Health Systems and HIV Service Delivery Outcomes in Mozambique:

Findings from the CHASS-SMT Project”. The study report was submitted to USAID on October 30 th , 2015.

57%

66%

48%

48%

38%

39%

34%

40%

21%

12%

19%

12%

31%

30%

39%

20%

3%

4%

14%

5%

15%

8%

6%

14%

8%

10%

12%

3%

5%

3%

11%

0% 25% 50% 75% 100%

Planning

Information Systems

Financial Mgmt

HRH

Supply & Logistics

Health Programs

Laboratory

Community Mob.

1st (18) 2nd (18) 3rd (12) 4th (7)

Improved DPS capacity to manage direct funding: the percent of disbursed sub-agreement funding successfully liquidated by the DPS increased steadily every year, reaching 100 by the end of the project

FIGURE 6. PERCENT OF DISBURSED DPS SUB-AGREEMENT FUNDING LIQUIDATED

Over US$1.3 million channeled to 36 districts through innovative results-based sub-agreements to support essential service delivery functions

Improved SDSMAS capacity to manage direct funding: the percent of planned quarterly sub-agreement activities executed increased from an average of 47 to 80 in 21 months

352 new nurses, lab technicians, and pharmacists graduated with project support

193 new health workers deployed to health facilities with gap-year funding, of which 172 (89%) were successfully transferred to the DPS payroll

EPTS expanded to 29 new ART sites, contributing to improved data quality

SIMAM coverage expanded from 8 percent of districts to 100 percent, contributing to improved commodities supply

Over 2,000 health workers provided with in-service training on clinical topics to improve HIV and AIDS services

85 86

94 97 99

90 92

Year 1 Year 2 Year 3 Year 4 Year 5

Sofala Manica Tete

3 IMPLEMENTATION AND

MANAGEMENT

STRATEGIES AND ACTIVITIES3.1

3.1.1 DISTRICT APPROACH

As mentioned earlier, the District Approach is described in detail in the District Approach technical brief from the EOP materials. In this report we focus on analysis and lessons learned.

CHASS-SMT implemented a full version of the District Approach in the 18 priority districts from Sofala, Manica, and Tete. The MOH’s Acceleration Plan4 identified the districts, targeting those with the highest unmet need for ART. The remaining districts received a less intensive form of the approach. Table 3 summarizes the main differences in District Approach implementation between priority and non-priority districts. The objective was to focus efforts on the districts that needed the most improvement, and therefore have greater impact on them.

TABLE 3. PACKAGE DISTRICT APPROACH INTERVENTIONS APPLIED TO PRIORITY VS. NON-

PRIORITY DISTRICTS

Intervention Priority Districts

(PD)

Non-Priority Districts

(NPD)

Technical assistance visits from multidisciplinary teams

Monthly Quarterly

Graduation Path intervention ✔

Sub-agreements ✔ ✔

District Profile ✔ ✔

Monitoring and Evaluation support (data collection and reporting)

Clinical Tutoring ✔ ✔

Re-engineering of patient flow ✔

Quality improvement initiatives ✔ ✔

Community-level interventions ✔ ✔

As part of the study that examined the relationship between improved district management capacity and service delivery outcomes, CHASS-SMT assessed differences in service delivery outcomes between priority and non-priority districts5. The project analyzed six indicators, three

The MOH Acceleration Plan assigned “priority status” to districts based on their unmet need for ART. However, CHASS-SMT had to make slight adjustments to this assignment to accommodate for factors such as political instability in certain districts. There were also changes in the MOH assignment through the course of the project.

By the end of the project, the priority districts receiving the full version of the District Approach were: Barue, Chimoio City, Gondola, Machaze, Mossurize and Sussundenga in Manica Province; Buzi, Chemba, Beira City, Dondo and Nhamatanda in Sofala Province, and Changara, Chifunde, Tete City, Moatize, Mutara and Tsangano in Tete Province.

For a detailed description of the study design, methodology, and results please see the full study report:

“Strengthening District Health Systems and HIV Service Delivery Outcomes in Mozambique: Findings from the CHASS-SMT Project”. Study report submitted to USAID on October 30 th , 2015.

related to access (A) and three related to quality (Q). Table 4 displays the indicators analyzed in the study and the results. At first glance, results seem mixed.

TABLE 4. INDICATORS ANALYZED IN STUDY AND RESULTS

Indicator (A = access; Q = quality)

Relationship detected with “Priority District” status

1.Number of new enrollees in ART (A) None 2.Number of new pre-ART enrollees (A) None 3.Number of new enrollees in pediatric HIV care (A) Negative 4.Percentage of pregnant women in antenatal care (ANC) who knew their HIV status after first consultation (Q) Negative

5.Percentage of HIV-positive pregnant women in ANC who received anti-retrovirals (ARVs) for eMTCT (Q)

Positive

6.Percentage of co-infected TB/HIV+ patients that initiated ART in the TB ward (Q)

Positive

However, when looking only at the quality indicators (4, 5, and 6), two of the three – indicators 5 and 6 – had a positive, statistically significant relationship with district priority status. Indicator 4 had a negative relationship; however, the baseline for this indicator was already very high, especially in priority districts (89% versus 85%). These are encouraging results regarding the potential effect of the full District Approach on service quality, especially considering two factors: the study only examined an 18-month time period; and it was not a “pure” intervention versus non-intervention design (non-priority districts also received many of the District Approach interventions, including sub-agreements).

For the access indicators, only indicator 3 had a statistically significant relationship with district priority status, and it was negative. However, results for these indicators are more difficult to interpret due to the ongoing decentralization of ART services over the course of the study (an unknown number of patients were transferred from sample facilities to non-sample facilities, possibly artificially decreasing patient volumes unevenly between priority and non-priority districts).

3.1.1.1 LESSONS LEARNED

The roll-out of the District Approach marked a turning point for the CHASS-SMT project. The most important feature of this approach was the integration of the project’s technical areas.

Under the new approach, project staff shifted from providing discrete technical assistance to district and health facility counterparts on specific topics, to providing comprehensive, integrated technical assistance. This integration had several positive effects:

Allowed for greater synergies across technical areas, including between health systems strengthening and clinical areas

Improved accountability of project staff for results, since teams were now clearly assigned to and responsible for particular districts

Decreased the burden on the MOH district and health facility counterparts during technical assistance visits

Enabled stronger capacity building of district and health facility managers, by focusing on their ability to see, analyze, and problem-solve the “full picture” affecting the delivery of health services.

These effects were frequently cited by project staff during project meetings. Project staff noted improvements in their own work and motivation as a result of the integration and the District

Approach. District and health facility staff also expressed their satisfaction with this new integrated approach during key informant interviews6:

“The team doesn’t look at just one area, which is very beneficial to the achievement of the different targets of the different health programs” – Dondo District Health Directorate Staff, Sofala Province.

“The integrated approach brings to our attention attitudes that need to be changed” – Gorongosa District Health Directorate Staff, Sofala Province.

The integration of technical areas in the District Approach was facilitated by three main factors:

The Equipas Polivalentes (EPs): multi-disciplinary teams that provided integrated technical assistance to districts. Each EP was formed by four to five project staff comprising health systems strengthening, clinical, M&E, and sub-agreement areas. They planned, conducted, and analyzed their field visits together. Although each EP member had his/her own area of specialty, they were each trained to address the basic requirements of all technical areas using the Graduation Path instrument as a guide.

The Graduation Path: discussed below, it was the main instrument used by EPs to guide their technical assistance visits at district level.

The District Sub-Agreements: also discussed below, they funded activities across all clinical areas and their successful execution was dependent on key health system capacities (planning, financial management, monitoring).

Another key feature of the District Approach was the systematization of project assistance.

CHASS-SMT was a complex project with a large breadth and depth of technical areas covering a large geographic area. Determining which assistance to give when and where was a major operational challenge in the beginning of the project, especially when each technical team worked separately. Hence the first step in overcoming this challenge was the integration of the teams. The second step was the introduction of tools and systems that provided standardized guidance to the teams on how to prioritize, provide, and monitor assistance. These tools and systems included the standard operating procedures (SOPs) for the EP district visits, the Graduation Path, the District Profile, and the Clinical Tutoring Tool (all of which are described in the EOP materials). Once again, project staff expressed great satisfaction with the systematized feature of the District Approach methodology. Technical staff frequently cited how their jobs had improved after the introduction of the approach. They noted how they had an increased understanding of their roles and what they needed to do in the field, as well as how all the different project pieces fit together to improve health services.

The introduction of the District Approach presented some challenges, mostly because it marked a significant change from the way the project was originally structured. CHASS-SMT had to reconfigure its original staffing plan to fit the new approach, and in some cases the fit was not optimal. The successful implementation of the approach also required a major change in attitude of project staff. They needed to move from seeing themselves as discrete, parallel teams who were only responsible for their own technical areas, to integrated teams who were collectively responsible for the overall improvement of a district. Although all project staff eventually came to embrace and appreciate the new approach, there was an initial resistance to this shift and it took time to materialize.

CHASS-SMT carried out these key informant interviews as part of the study “Strengthening District Health Systems and HIV Service Delivery Outcomes in Mozambique: Findings from the CHASS-SMT Project”.

3.1.2 GRADUATION PATH

The Graduation Path methodology is described in the Graduation Path technical brief from the EOP materials. The main goal of this methodology was to inform, guide, and monitor the project’s efforts to strengthen SDSMAS management capacities. The methodology proved to be successful in achieving these goals. Results of the Graduation Path assessments described above show consistent improvements in districts’ management capacity with every assessment.

District staff noted specific improvements in SDSMAS operations as a result of the Graduation Path7. For example: stronger culture of documentation and reporting, greater quality and emphasis on data, and better maintenance of laboratory equipment.

Most importantly, results from the study carried out by CHASS-SMT showed that improvements in certain Graduation Path functions had strong, positive associations with improvements in access to and quality of HIV and AIDS services8. This is important evidence regarding the potential relationship between district management capacity and service delivery performance.

The functions that showed an association with improved service delivery were the services functions: health programs, laboratory services, and community engagement. The remaining systemic functions (planning, information systems, financial management, human resources, and supply and logistics) showed mixed results.

A total of 18 districts participated in the Graduation Path methodology. The strategy was originally intended for use only in the 12 priority districts. Because there were changes to the assignment of priority districts, the final number of participating districts was higher. Districts launched the Graduation Path at different points in time. Of the 18 districts, they all carried out at least two assessments, 12 carried out three assessments, and seven carried out four assessments. Assessment frequency varied from every four-six months.

3.1.2.1 LESSONS LEARNED

CHASS-SMT designed the Graduation Path to be ultimately a district-led quality improvement process. Before rolling it out, CHASS-SMT advocated with the three DPS’ to obtain their buy-in into the strategy. All three DPS’ supported the introduction of the Graduation Path; in fact, the Sofala DPS also expressed its interest in adopting the methodology as part of the Province’s regular processes. As expected, there was wide variation on the implementation experiences across different districts.

On one end of the spectrum were districts where EPs and SDSMAS staff worked hand-in-hand to carry out the Graduation Path assessments. In these districts SDSMAS staff had high appreciation for the value of the self-assessment as part of a greater, continuous quality improvement process. Their assessments included rich, detailed action plans to address weaknesses identified, and the SDSMAS took increasing ownership of the process with each assessment round. They clearly saw the link between the Graduation Path and other project support such as technical assistance and the sub-agreements.

On the other end of the spectrum, were districts that still saw the assessment process as a “project requirement,” with little value to their day-to-day activities. In these districts, assessment instruments and action plans tended to be minimally, sometimes partially completed, and SDSMAS staff ownership levels remained low. The level of SDSMAS understanding and ownership of the Graduation Path was at least in part related to the corresponding levels in their

Information obtained through the qualitative component of the study “Strengthening District Health Systems and HIV Service Delivery Outcomes in Mozambique: Findings from the CHASS-SMT Project”.

Results of the study “Strengthening District Health Systems and HIV Service Delivery Outcomes in Mozambique:

Findings from the CHASS-SMT Project”.

assigned EPs. That is, it reflected varying levels of understanding of the methodology across the different EPs. As with the overall District Approach, it took some time until the EPs fully absorbed the purpose of the methodology, how it fit within the District Approach, and how to implement it. Ultimately some EPs were better at this than others.

DPS and SDSMAS ownership is possibly the area with the greatest room for improvement in the Graduation Path. Given that the project introduced the methodology relatively late in the life of the project (end of 2013), project staff may have felt pressed to rush its introduction to have enough time to show results. As a result, the project may have underestimated the time and effort needed to obtain full buy-in from districts. Taking the time to introduce the methodology jointly with the DPS would have probably strengthened SDSMAS ownership and provided a stronger foundation for eventual institutionalization.

However, CHASS-SMT did not expect to have the methodology fully institutionalized within MOH processes in the short time between its introduction and the closing of the project.

Successful adoption of strategies such as the Graduation Path takes time, and happens gradually. Overall, there was a clear positive pattern in terms of increasing SDSMAS and DPS ownership, with the potential to lead to full ownership and sustainability.

In terms of the instrument itself, results of the study showed that as expected, the services functions, which are more closely related to the delivery of health services, had a much stronger and more positive association with improvements in service delivery. However, the importance of the systemic functions should not be minimized by these findings. Systemic functions play an important role in supporting the services functions, and likely take more time than 18 months to show an association with service delivery outcomes.

3.1.3 SUB-AGREEMENTS

CHASS-SMT provided sub-agreements to three different types of institutions: DPS, SDSMAS, and training institutions. Sub-agreements to the DPS’ were “classic” cost-reimbursable grants to cover basic DPS operating expenses. The project designed the sub-agreements for the training institutions to pay for the pre-service education of new health care workers. In this report we focus on the district or SDSMAS sub-agreements due to their innovative nature and the impact they had on project results.

CHASS-SMT provided fixed-price sub-agreements to all 36 districts in Sofala, Manica, and Tete. The purpose of the sub-agreements was to provide the districts with the essential resources they needed to function, such as fuel and per diem for supervision visits and transportation of commodities. CHASS-SMT designed the sub-agreements as fixed-price or output-based grants with two goals in mind:

Reducing the administrative burden on district management; and

Using the grant mechanism as an incentive for districts to perform their functions

The sub-agreement design was based in part on the lessons learned from a performance-based incentives pilot that CHASS-SMT implemented in Manica in 2012-20139. Results from the evaluation of this pilot showed no clear impact of performance-based incentives on service delivery. However, the qualitative component of the evaluation found that resources for improving working conditions in health facilities may create a stronger incentive to health workers than individual monetary incentives.

The activities covered by the grants came directly from the district annual plans (Plano Económico Social e Orçamento Distrital – PESOD). CHASS-SMT worked with district teams to

Draft evaluation report submitted to USAID on May 22, 2014.

select indicators for the grants and assign quarterly targets (see Annex I for the district sub-agreement indicator list). Each quarter, project staff and district teams assessed district performance against the indicators, and districts received a quarterly payment based on their performance. CHASS-SMT staff provided continuous technical assistance to SDSMAS administrative teams on sub-agreement budgeting, execution, and verification. Over the seven quarters of sub-agreement implementation, districts’ ability to execute their planned activities for the quarter improved from 47 percent to 80 percent.

The introduction of the district sub-agreements had a profound positive impact on the effectiveness of project interventions. The first and most obvious effect was the availability of essential resources for the provision of health services. There is a limit to the impact that technical assistance can have on service delivery when facilities and districts do not have the minimum conditions necessary. Once those resources were available, district and facility staff was able to better perform their functions and implement solutions identified through the technical assistance. The level of motivation and collaboration with project staff also increased significantly. Key informant interviews revealed that the sub-agreements were perceived by district and facility staff to have had the greatest impact on improvements of district management and service delivery10. It is important to note that in addition to the sub-agreements, CHASS-SMT also provided more than US$16 million worth of equipment and materials to health facilities and districts through a major procurement effort (kit básico) and a donation from partner Project CURE. This additional material support was also critical to enabling improvements in service delivery, increasing motivation of health workers, and facilitating a collaborative partnership with the DPS’ and SDSMAS’.

In addition to providing essential resources for improved district functioning and service delivery, the sub-agreements proved to be a powerful health system strengthening tool. By placing the districts in charge of managing their own resources, their capacities visibly improved as they “learned by doing” with mentoring from CHASS-SMT. The sub-agreements served as a convergence point between service delivery and health system strengthening, where clinical and financial and administrative teams were “forced” to work together on the planning, budgeting, executing, and reporting of activities. Both project and district staff reported notable improvements in the integration and collaboration within and across teams, as well as improved communication and coordination between districts and health facilities.

The sub-agreements also improved transparency and accountability. Clinical staff had to advocate for funding for their programs and activities. All district and facility staff knew how the funding was eventually assigned, who was responsible for spending what, and when the money was received. Therefore all staff was motivated to work together to ensure that planned sub-agreement activities were executed and reported on time.

“(…) with the direct sub-agreement from CHASS-SMT, the only funding we don’t receive during the quarter is for activities that we weren’t able to execute. That is, if we don’t have funding it’s our own fault.” – Gorongosa District Health Directorate Staff, Sofala Province

“Before the sub-agreements, the activities were carried out (well or not), but were not reported, no one did the reports; now with the sub-agreements everyone is concerned with reporting their activities so that they can be converted into products.” – Dondo District Health Directorate Staff, Sofala Province

Information obtained through the qualitative component of the study “Strengthening District Health Systems and

HIV Service Delivery Outcomes in Mozambique: Findings from the CHASS-SMT Project”.

3.1.3.1 LESSONS LEARNED

Planning

It is common for implementing partners to approach sub-agreements from only an administrative and contractual perspective. The most successful sub-agreements programs obtain maximum impact when the following conditions are in place:

There are clear objectives

Stakeholders are identified

Rules are set

Risk is managed

Work breakdown structures are used

Schedules are realistic

Estimates are accurate

Communications are clear

Progress is measured

Management is organized

In this sense CHASS-SMT managed to strike the balance among the various tasks mentioned by employing a team planning approach with technical officers across the various areas of the project and the MOH staff, so that priorities could be set and support structures would be suitable.

Task-Based Budgeting

Managing a sub-agreement with USAID funding can be a daunting task for organizations with little or no USAID experience. Utilizing task or milestone based budgeting and reporting, which are designed to achieve deliverables and important events, simplifies sub-agreement management for the sub-grantee. It also provides clarity on where efforts must be focused. By providing focus to the sub-grantee, they do not get overwhelmed by the sheer volume of rules and regulations and can direct their efforts towards implementation.

Right-Sizing Compliance

Sensitive items or items that are often the subject of noncompliance can be provided “in-kind” to the grantee by the implementing partner, to ensure that the proper approvals are in place and disallowances are avoided. Examples of such items are international travel, vehicle acquisitions, and the purchase of durable goods. In this regard, CHASS-SMT complemented sub-agreements with in-kind purchases to support the sub-grantee. Examples of in-kind purchases and support provided include: the delivery trucks provided to each DPS; 36 motorcycles provided for districts; and furniture and equipment provided in the basic package to improve district facility functionality.

Shadowing and Capacity Building

When implementing sub-agreements, after the sub-agreement signing, the sub-grantee is often left alone to implement the sub-grant, as the donor or partner moves on to the next priority. To build capacity of the sub-grantee in technical and administrative areas, the CHASS-SMT project maintained constant contact with the sub-grantee throughout all phases of implementation (planning, implementation, reporting, close-out), and provided significant training and human resources support in all of these areas. Technical staff met with their counterparts as frequently

Clinical Tutoring Tool Components Observation checklists:

• General conditions

• Pharmacy

• Laboratory Clinical observation and tutoring checklists:

• Antenatal care - HIV counseling and testing

• Antenatal care - HIV care and treatment

• Adult HIV counseling and testing

• Adult HIV care and treatment

• Pediatric HIV counseling and testing

• Pediatric HIV care and treatment as monthly, to ensure activities were being implemented, and to assist in managing those activities that were falling behind schedule or would not occur altogether. The close link between technical and administrative staff, including the inclusion of a sub-agreement manager in the EPs ensured that obstacles and challenges were shared and resolved in an integrated manner.

3.1.4 CLINICAL TUTORING

As part of the District Approach, CHASS-SMT developed and introduced a Clinical Tutoring Tool to be used by EPs during technical assistance visits to health facilities.

This Excel-based tool covered all key components that affect HIV and AIDS service delivery in the health facilities (Box 1), including general conditions such as facility infrastructure and supply of basic materials. The tool consisted of a checklist for supervisors to complete by observing clinical consultations and general facility conditions. Its main purpose was to structure and guide the joint EP-SDSMAS supervision visits to heath facilities.

SOPs associated with the tool provided clear guidance on how to apply it while assessing the performance of health staff and determining ways to strengthen their clinical skills through on-the-spot mentoring and tutoring.

In addition to tutoring, technical assistance teams discussed and helped mitigate any observed weaknesses and recommended targeted interventions at the facility. These interventions included: formal training; quality assurance/improvement initiatives; thorough reviews of patient files; and/or improvement of basic infrastructure or acquisition of basic materials. SDSMAS could implement these interventions using funding from the district sub-agreements.

Overall, use of the tool was critical in standardizing and systematizing the supervision provided by SDSMAS’ and CHASS-SMT to health facilities. District supervisors had a clear roadmap on how to proceed during supervision visits, while health facility staff knew what to expect from the visits. The district/facility teams were able to clearly identify areas in need of improvement and provide ways in which issues could be quickly addressed. Additionally, they engaged in consistent follow up during subsequent visits to ensure that facilities implemented and sustained corrective actions and/or interventions. By using the same tool and process each time, district and facility staff could also identify improvements that needed to be made and track progress against these indicators.

Additionally, CHASS-SMT applied the tool to district supervisors to ensure that their clinical skills were up-to-date and to reinforce the correct use of the tool. District and health facility staff reported widespread improvements in the quality of supervision visits and in the quality of care as a result of this tool. Enhancements reported by facility staff included better compliance with clinical protocols, and more consistent and correct use/filling of registries and patient files.

3.1.4.1 LESSONS LEARNED

As with every new strategy or tool that is introduced, it took some time until the EPs and their district counterparts became familiar with the tool and its SOPs. There was also variation on the comfort level with using the tool. Some EPs easily adopted the concept and the tablet. Others felt that use of the tablet could be intrusive, that the tool was too rigorous, or that they were focusing too much on the tablet rather than the consultation. CHASS-SMT senior technical staff worked closely with the EPs to understand and overcome these challenges. By the end of the project, all 11 EPs were using the tool across the three provinces. As clinical staff became more comfortable using a tablet-based technical assistance guide, they found that it was a useful strategy for guiding their clinical tutoring. The tool made it easier to target the weakest areas, and provide immediate technical assistance following each observed consultation to strengthen the quality of HIV and AIDS prevention, care and treatment services. For those older staff that never became comfortable using a tablet-based tool, they were still able to take advantage of the tool’s checklists and prioritization function by printing the different sections out on paper. As PEPFAR shifted towards more detailed monitoring of the type of support each health facility receives, this tool also helped to facilitate the accurate tracking and reporting of clinical tutoring, as well as enable project managers to identify best practices and trends to maximize impact.

Although CHASS-SMT did not formally evaluate the impact of the tool, the improvements in quality indicators and feedback from SDSMAS and health facility staff were encouraging. The tool also offered the possibility of adding another layer of quantitative monitoring of service quality, in addition to facility-level service delivery indicators. Given more time, the project would have migrated the tool over to DHIS2 to facilitate data aggregation, analysis and reporting, thereby enabling EPs and project managers to look beyond a specific consultation room or clinical sector to analyze data at the health facility, district, province, or project level.

3.1.5 RE-ENGINEERING OF PATIENT FLOW

In Year Four, CHASS-SMT introduced a methodology for re-engineering patient flow in high-volume health facilities.