Attachment_L_SDSH_Final_Evaluation_-_Nov_2011.pdf

USAID/HAITI: SANTE POUR

LE DEVELOPPEMENT ET LA STABILITE

D’HAÏTI (SDSH) PROJECT

EVALUATION

NOVEMBER 2011

This publication was produced for review by the United States Agency for International Development. It was prepared by Gerard Bowers, Rachelle Cassagnol, and Williams Ojo through the Global Health Technical Assistance Project.

Attachment L - SDSH Final Evaluation

USAID/HAITI: SANTE POUR

LE DEVELOPPEMENT ET LA STABILITE

D’HAÏTI (SDSH) PROJECT

EVALUATION

DISCLAIMER

The views of the authors expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

This document (Report No. 11-01-584) is available in printed or online versions. Online documents can be located in the GH Tech website library at http://resources.ghtechproject.net.

Documents are also made available through the Development Experience Clearinghouse

(http://dec.usaid.gov). Additional information can be obtained from:

The Global Health Technical Assistance Project

1250 Eye St., NW, Suite 1100

Washington, DC 20005

Tel: (202) 521-1900

Fax: (202) 521-1901 info@ghtechproject.com

This document was submitted by The QED Group, LLC, with CAMRIS International and Social

& Scientific Systems, Inc., to the United States Agency for International Development under

USAID Contract No. GHS-I-00-05-00005-00.

USAID/HAITI: SDSH PROJECT EVALUATION i

CONTENTS

ACRONYM LIST ............................................................................................................. III

EXECUTIVE SUMMARY ............................................................................................... VII

The Project ............................................................................................................................................ vii

Evaluation Purpose and Methodology ............................................................................................. vii

Key Findings and Recommendations................................................................................................ vii

Project Monitoring .............................................................................................................................. viii

Project Implementation........................................................................................................................ ix

Capacity Building .................................................................................................................................... x

Lessons Learned .................................................................................................................................... xi

Next Steps .............................................................................................................................................. xi

I. INTRODUCTION

Summary Project Description

Purpose of the Evaluation

Methodology

2. FINDINGS AND RECOMMENDATIONS

Project Monitoring

Project Implementation

Capacity Building

Lessons Learned

Needs Assessment

ANNEXES

ANNEX A: SCOPE OF WORK

ANNEX B: EVALUATION DESIGN AND METHODOLOGY

ANNEX C: LIST OF PERSONS INTERVIEWED

ANNEX D: LIST OF DOCUMENTS REVIEWED

USAID/HAITI: SDSH PROJECT EVALUATION ii

FIGURES

Figure 1. Map of Haiti Showing Spatial Distribution of

SDSH-Supported Sites .................................................................................................. VIII

Figure 2. Ideal National HMIS Integrated Data Flow

TABLES

Table 1. Proposed Schedule

Table 2. SDSH Evaluation Design and Methodology

USAID/HAITI: SDSH PROJECT EVALUATION iii

ACRONYM LIST

AEADMA Association d‘Entr‘Aide des Dame-Mariens

AIDS Acquired immunodeficiency syndrome

BCC Behavior change communication

CA Cooperating agency

CBO Community-based organization

CDC Centers for Disease Control and Prevention

CDS Centres pour le Développement et la Santé

CHW Community health worker

CIDA Canadian International Development Agency

CM Community mobilization

CONASIS Comité National du Système d'Information Sanitaire

(National Committee for the Health Information System)

COP Chief of party

COR Contracting Officer Representative

COTR Contracting Officer Technical Representative

DQA Data quality assessment

DRI Direct Relief International

DSF Direction Santé de la Famille (Division of Family Health, MOH)

FA Financial advisor

FBO Faith-based organization

FONDEP Fondation pour le Développement de Petit Trou de Nippes

FOSREF Fondation pour la Santé Reproductive et l‘Education Familiale

FP Family planning

GH Tech Global Health Technical Assistance Project

GHI Global Health Initiative

GOH Government of Haiti

GUC Grants-Under-Contract

HCF Health care financing

HIV Human immunodeficiency virus

HMIS Health Management Information System

HR Human resources

HS 2004 Haiti Health Systems 2004

HS 2007 Haiti Health Systems 2007

USAID/HAITI: SDSH PROJECT EVALUATION iv

HSS Health systems strengthening

ICC International Child Care

JHU-CCP Johns Hopkins University‘s Center for Communications Programs

JHPIEGO Johns Hopkins Program for International Education in Gynecology and Obstetrics

LAPM Long-acting and permanent methods (of family planning)

MCH Maternal and child health

M&E Monitoring and evaluation

MOH Ministry of Health

MOU Memorandum of understanding

MSH Management Sciences for Heath

MSPP Ministère de la Santé Publique et de la Population

(Ministry of Public Health and Population)

MSSF Monthly Statistics Summary Form

MYAP Multi-Year Assistance Program

NGO Non-governmental organization

PADESS Projet d‘Appui au Développement du Système de Santé

PBF Performance-based financing

PDI Plan Départemental Intégré (Integrated Departmental Plan)

PEPFAR President‘s Emergency Plan for AIDS Relief

PLWHA Persons living with HIV and AIDS

PMP Performance monitoring plan

PMTCT Prevention of mother-to-child transmission (of HIV/AIDS)

POI Plan Organisationnel Intégré (Integrated Operational Plan)

PPP Public-private partnership

PPS Point de prestation de services (service delivery point)

PWW Pure Water for the World

RH Reproductive health

RNDI Réseau national de distribution des intrants (national health sector supplies distribution)

SADA Service and Development Agency

SDMA Service Delivery and Management Assessment

SDSH Santé pour le Développement et la Stabilité d‘Haïti

SOG Soins obstétricaux gratuits (free obstetrical care)

SOW Scope of work

TA Technical advisor

TB Tuberculosis

USAID/HAITI: SDSH PROJECT EVALUATION v

TBA Traditional birth attendant

UCS Unité Communale de Santé (Communal Health Unit)

UNFPA United Nations Population Fund

UNICEF United Nations Children‘s Fund

UPE Unité de Planification et d'Evaluation (Planning and Evaluation Unit, MOH)

USAID U.S. Agency for International Development

WHO World Health Organization

ZC Zone ciblée (targeted area)

USAID/HAITI: SDSH PROJECT EVALUATION vi

USAID/HAITI: SDSH PROJECT EVALUATION vii

EXECUTIVE SUMMARY

THE PROJECT

The Santé pour le Développement et la Stabilité d’Haïti (SDSH) Project is the U.S. Agency for International

Development/Haiti (USAID/Haiti) flagship health sector activity. Awarded to Management Sciences for

Health (MSH) in August 2007 for an initial period of three years (2007–2010) at a cost of $42.5 million, the contract was amended four times between 2007 and 2009 to increase its cost to $81.4 million, and to extend the activity to September 30, 2012.

The SDSH Project builds on two predecessor projects managed by MSH—Haiti Health Systems 2004

(HS 2004) and Health Systems 2007 (HS 2007)—and continues the earlier activities‘ support for the health delivery programs of (currently) 28 non-governmental organizations (NGOs). The most important new component of the project is the inclusion of a focus on 31 target zones or zones ciblées

(ZC)—areas identified by the Ministry of Health (MOH) as seriously underserved by MOH resources.

Health services available through those networks are expected to reach approximately 50% of the population by the end of the project in 2012. MSH manages the SDSH project with two subcontractors:

The Johns Hopkins Program for International Education in Gynecology and Obstetrics (JHPIEGO) and the Johns Hopkins University‘s Center for Communications Programs (JHU-CCP).

EVALUATION PURPOSE AND METHODOLOGY

In October/November 2011, USAID/Haiti sponsored an evaluation conducted by a consultant team recruited by the Global Health Technical Assistance Project (GH Tech). The key purposes of the evaluation were to (1) assess the performance of the SDSH project and, drawing on the findings of that assessment, (2) propose, for USAID‘s consideration, recommendations regarding the shape and objectives of a follow-on project.

The project evaluation was carried out in four departments (West, Artibonite, Central Plateau, and

Nippes), where the team observed service delivery programs conducted by seven NGOs and two MOH

ZCs. The team analyzed quantitative and qualitative data collected at NGO and MOH service sites;

interviewed SDSH, NGO, MOH, and other donor personnel in Port-au-Prince and at the four MOH health departments (departments sanitaires); and visited one grantee undertaking work further to the project‘s Grants-Under-Contract (GUC) initiative. (See Figure 1 for a map of Haiti with information on SDSH-supported sites.)

KEY FINDINGS AND RECOMMENDATIONS

The evaluation team collected, analyzed, and arranged its findings in general conformance to categories suggested by USAID/Haiti, namely project monitoring, project implementation, and capacity building.

USAID/HAITI: SDSH PROJECT EVALUATION viii

Figure 1. Map of Haiti Showing Spatial Distribution of SDSH-supported Sites

Note: SDSH currently supports service delivery activities in 78 health facilities (NGO_PPS = service delivery points, operated by 28 NGOs; ZCs_PPS = 79 MSPP public sector sites in 31 target zones)

PROJECT MONITORING

Key Findings:

Although the evaluation team did not undertake an assessment of the validity of data reported in the project‘s performance monitoring plan (PMP), an informal review of those data indicate that the project is meeting or surpassing most of its performance objectives.

The SDSH project has developed and utilized a robust performance monitoring system that effectively tracks the performance of the project‘s participating NGOs and MOH ZCs. That monitoring system generally exists in parallel, however, to other MOH (and President‘s Emergency Plan for AIDS Relief

[PEPFAR]) data collection systems in use at the MOH department level. No meaningful efforts are currently under way to harmonize the collection, distribution, or use of those separate data flows.

The disconnect between these systems is at least partly a consequence of (1) MOH delays in extending its Health Management Information System (HMIS)—itself developed with assistance from SDSH and the two predecessor projects—down to the MOH department level; (2) the absence of a weak data culture at the department level, where little priority is given to data-related training, coordination among persons responsible for data collection and reporting, or ensuring the validity of collected data; and (3)

USAID/HAITI: SDSH PROJECT EVALUATION ix inadequate efforts by SDSH (technical and financial) advisors at the department level to promote better integration of the SDSH data system into the other systems.

Key Recommendations:

The SDSH project should seek to align its data collection and reporting system with other systems in use at the MOH departmental level by promoting closer interaction between the project‘s on-site

SDSH advisors (i.e., the technical and financial advisors) and MOH department counterparts.

SDSH should encourage the MOH to provide additional training in data management and performance-based financing (PBF) procedures to MOH department-level staff.

PROJECT IMPLEMENTATION

Key Findings:

SDSH got off to a slow start because of USAID-mandated changes in the contract‘s scope of work and was beset with delays and disruptions throughout its four-year implementation period (e.g., temporary deactivation of subcontractors; high turnover of key staff, including three chiefs-of-party;

the hurricanes of 2008 and subsequent political instability; the January 2010 earthquake and subsequent cholera epidemic). The project nonetheless remained generally on track and continued to achieve most of its performance targets. The project‘s inability to retain key personnel, however, prompts some concern over its ability to maintain a stable project management and oversight capability.

SDSH has developed several management and oversight tools that it applies with consistent vigor.

These include a PMP and the creation of specific performance targets for each participating NGO and ZC; close monitoring of partners‘ performance by SDSH department-level personnel (technical and financial advisors) and SDSH headquarters personnel; annual reviews (service delivery and management assessments, or SDMA) of project partners; and annual meetings of the partners, to name a few. Although several NGO managers noted their preference for less ―rigidity‖ in the manner in which SDSH assigns and enforces project targets, the rigor with which SDSH applies its tools and enforces its targets contributes to the project‘s effectiveness.

SDSH has successfully expanded the scope of the predecessor projects (HS 2004 and HS 2007) by incorporating the MOH ZCs into its health service delivery program.

The project‘s success in expanding service delivery at the MOH department level is not matched by its more lackluster performance in improving the executive capacity of the MOH at the central level, where SDSH efforts to help develop a HMIS and financial management tools have yet to produce durable results.

The project‘s success in expanding service delivery is due in good measure to its adoption and application of a sound service delivery model with several interlocking elements (community health workers (CHWs), trained technical and support personnel at service delivery points and at the

MOH departments, engaged SDSH advisors at the department offices, attentive SDSH headquarters staff, and successful application of the PBF approach to project contracting). The model, however, is not currently sustainable from either a financial or a management perspective, as participating NGOs and the MOH are financially dependent on project resources and the MOH lacks any meaningful capacity to implement or monitor a PBF system.

Long-term sustainability of the service delivery program will require (1) development of program management skills at MOH central and departmental levels and (2) continued donor and MOH provision of the financial resources needed to implement the service delivery program.

USAID/HAITI: SDSH PROJECT EVALUATION x

The project succeeded in responding—to the extent it could within the constraints of its contract— to the cholera epidemic.

Some elements of the project, although generally successful and useful in their own right, are not contributing substantively to the attainment of the project‘s primary objective: the expansion of integrated health services to underserved populations.

USAID/Haiti has not adequately explored closer linkages between the SDSH project and the

Mission‘s PL-480 Title II program.

Key Recommendations:

USAID should consider encouraging bidders for the follow-on project to propose a Haitian national for the chief-of-party position in order to (1) better reflect Haitian ownership of the activity. and (2) improve the stability of a key position that has been subject to high turnover over the past four years.

The follow-on project should include provision for a comprehensive effort to develop and install professional-level capacity at both the central and departmental levels of the MOH to effectively install capacity to design, execute, monitor, and support (though not necessarily fund) performance-based contracts.

The design of the follow-on project should call for a lean activity (in terms of its various components) focused on the extension of basic health services, plus the capacity-building efforts required to support that expansion. Extraneous elements that do not directly support that objective should not be included in the project.

USAID should expand and formalize NGO-cooperating agency (CA) arrangements by encouraging its food assistance partners in Haiti to reach out to NGOs that participate in the SDSH project, and to include their plans for collaboration in the CAs‘ multi-year assistance plans.

CAPACITY BUILDING

Key Findings:

Although capacity building of local NGOs was not a central aspect of SDSH, the project helped strengthen the capacity of local organizations, mainly through mentoring and technical assistance for service delivery. SDSH is overly involved, however, in the routine oversight of NGO service delivery points, a practice that weakens the leadership of NGOs and hinders their path toward graduation from donor reliance.

SDSH participation on various MOH technical committees contributed to the development of norms, standards, and procedures for selected service delivery activities. However, the project‘s participation did not result in significant transfer of knowledge to reinforce the capacity of MOH partners at the central level.

MOH departmental directorates have only a limited role in the management of performance-based contracts signed between NGOs and SDSH in the department.

SDSH training and support for several thousand CHWs and traditional birth attendants(TBAs), plus the project‘s role in establishing mothers‘ clubs, are helping communities improve access to basic health care services. However, the project has not realized its goal of establishing local health task forces at the community level.

USAID/HAITI: SDSH PROJECT EVALUATION xi

Key Recommendations:

SDSH should utilize its annual Service Delivery and Management Assessment exercise to analyze the level of organizational capacity of each participating NGO.

The follow-on project should include provision for (1) continuation of on-site technical and financial advisors at MOH departmental levels, and MOH central-level access to advisors who would assist in the development of improved MOH executive capacity; (2) a focus on the development of local health task forces; and (3) efforts to increase access by youth to health services.

The follow-on project should help selected local NGOs develop innovative ways to increase funding, including the exploration of prospects for community health insurance to increase users‘ financial access to health services.

LESSONS LEARNED

Two general lessons emerge from the SDSH project evaluation:

1. The design and implementation of health assistance projects in Haiti needs to take place over a long planning horizon—beginning several decades ago, and continuing at least 10 years into the future.

2. The SDSH project has succeeded in its primary goal to expand the availability of essential health services, but it has not played a strategic role in helping to ensure that such services are and remain financially and managerially sustainable. A more strategic orientation would position the project to begin the transfer of responsibility for project management and oversight from the contractor to the

Government of Haiti (GOH)/MOH.

Other, more specific lessons include the following:

A project focused almost exclusively on the achievement of performance targets runs the risk of giving inadequate attention to measures that track quality of care, ethical issues in the delivery of health services, and progress toward the organizational strengthening of partners.

Similarly, a project heavily invested in its own performance monitoring system can inadvertently contribute to the emergence of separate, parallel reporting systems that compete for the time and attention of scarce data management personnel.

Haiti lacks a functioning patient referral system, and there are no patient tracking systems in place that can be used to follow up on patients who are nominally referred to other health facilities.

No long-term plan is in place to effectively respond to the lack of long-acting and permanent methods (LAPM) of family planning in rural Haiti.

NEXT STEPS

Short-term adjustments that SDSH can adopt in the remaining 10 months of the project include the following steps:

Undertake an assessment, as part of the project‘s SDMA exercise, of (1) NGO and ZC capacity to implement a more methodical response to the cholera epidemic; and (2) track the progress of participating NGOs toward a higher level of institutional readiness for ―graduation‖

Finalize the MOH HMIS

Begin the process of aligning data reporting systems at the MOH department level

Ensure that SDSH documents best practices applied during the project and reports these practices to USAID/Haiti

USAID/HAITI: SDSH PROJECT EVALUATION xii

Consult with prominent NGOs to develop human resource (HR) policies designed to minimize staff resentments and turnover

Re-examine the use of SDSH patient registers at health facilities

For the longer term, USAID should consider incorporating the following factors into the design of the follow-on to the SDSH project:

Retention of key components of the current project (the service delivery model; performance-based contracting; NGO partnerships; and so forth)

Elimination of extraneous activities (the Strategic Partners and Grants-Under-Contract initiatives;

marginally productive efforts at the MOH central level)

Transition of project management responsibilities from the contractor to the MOH, with an end-of-project goal of at least two departments capable of negotiating and managing performance-based contracts with NGOs

USAID/HAITI: SDSH PROJECT EVALUATION 1

I. INTRODUCTION

SUMMARY PROJECT DESCRIPTION

The Santé pour le Développement et la Stabilité d’Haïti (SDSH) Project is the U.S. Agency for International

Development/Haiti (USAID/Haiti) flagship health sector activity. Awarded to Management Sciences for

Health (MSH) in August 2007 for an initial period of three years (2007–2010) at a cost of $42.5 million, the contract was amended in 2009 to extend the project to September 30, 2012, and to increase project funding to $81.4 million.

The SDSH Project builds on two predecessor projects managed by MSH: Haiti Health Systems 2004 (HS

2004) and Health Systems 2007 (HS 2007). All three projects were designed to increase access to and use of a package of integrated basic health services that covers maternal and child health (MCH), nutrition, family planning (FP), HIV and AIDS, and tuberculosis (TB). All three projects stressed close cooperation and collaboration with Haiti‘s Ministry of Health (MOH), the Ministère de la Santé Publique et de la Population (Ministry of Public Health and Population, or MSPP). With some minor changes, all three projects supported essentially the same group of (currently) 28 non-governmental organizations

(NGOs)—two were dropped for various reasons in 2008 and 2009; one joined in 2006; and the most recent addition joined in 2010. Finally, the SDSH Project continues the HS 2007 Project‘s reliance on performance-based financing (PBF) as a basis for MSH‘s relationships with participating NGOs—and more recently as an assistance mechanism to support MOH target zones, or zones ciblées (ZC).

This is not to say, however, that the SDSH project does not include changes, some of which are significant. The most prominent changes include (1) a heightened focus on measures to increase community involvement in project implementation; (2) the inclusion, at the MOH‘s request, of the ZCs in the project; (3) a shift in the focus of capacity-building efforts from the MOH central level to the departmental level; and (4) a more robust application of the PBF model as a basis of SDSH relationships with NGOs and the MOH. Notwithstanding those changes, the SDSH project serves as a relatively seamless transition from the predecessor activities to the current one.

The project is currently supporting service delivery activities in 81 health facilities (points de prestation de services, or PPS), operated by the 28 participating NGOs, and through 79 MSPP public sector sites in 31

ZCs. The project‘s primary beneficiaries are (1) children and youth under 25 years of age; (2) women;

and (3) groups with special concerns, including persons living with HIV and AIDS (PLWHA) and TB.

SDSH currently covers 42% of the Haitian population, or 4,254,166 people.

Other new components of the project include a Grants-Under-Contract (GUC) program that was slow getting under way for a variety of reasons (discussed in detail below), and a Strategic Partnership initiative designed to leverage health-related contributions from other partners, especially in the private sector.

MSH manages the SDSH project with two subcontractors: The Johns Hopkins Program for International

Education in Gynecology and Obstetrics (JHPIEGO) and the Johns Hopkins University‘s Center for Communications Programs (JHU-CCP).

PURPOSE OF THE EVALUATION

In several important respects, USAID/Haiti views the SDSH project as an antecedent to a more direct agency relationship with selected, institutionally capable NGOs, and most especially with the MOH. The

Mission‘s interest is driven in part by the Global Health Initiative (GHI) and USAID Forward, both of which call for increased reliance on and ownership by host-country partners. Looking to the future, and

USAID/HAITI: SDSH PROJECT EVALUATION 2

in the wake of a successful democratic transition in Haiti, USAID/Haiti is embarking on a broad-based effort to help ensure the success and durability of Haitian government institutions and is planning to develop the needed assistance modalities to do so.

The two-fold objective of this evaluation, then, is to provide USAID/Haiti with (1) an assessment of the project‘s performance in laying down a firm groundwork for the next phase of USAID assistance in the

Haitian health sector (i.e., the project‘s relative success in preparing NGOs and the MOH to assume a more mature and direct relationship with USAID under the strategic direction of the Government of

Haiti (GOH)); and in a more prosaic sense, (2) an assessment of the project‘s performance in meeting its contractual requirements.

METHODOLOGY

The project team, consisting of two expatriate and one Haitian consultant and three Haitian data collectors, familiarized themselves with the project by reviewing project reports prepared by SDSH, interviewing SDSH staff, and travelling extensively in Haiti to visit NGO and GOH offices and service delivery sites, where team members interviewed NGO and government program managers and service providers and reviewed project data. The team also attempted to gather qualitative data via discussions with patients, health care providers, and community health workers. (Annex B presents the team‘s evaluation design and methodology.)

Further, to supplement guidance provided to the evaluation team by USAID/Haiti program managers, the evaluation team attempted to focus its data-gathering and reporting efforts on information that is not readily available to the Mission in SDSH project documents.

USAID/HAITI: SDSH PROJECT EVALUATION 3

II. FINDINGS AND RECOMMENDATIONS

PROJECT MONITORING

The evaluation attempted to assess the extent to which the SDSH project achieved the expected results specified in the project‘s performance monitoring plan (PMP), and the extent to which the SDSH project sought to ensure the validity and reliability of performance indicators via the application of the project‘s PBF model.

In addition, the evaluation team was asked to determine whether the health ministry‘s Health

Management Information System (HMIS) was being used to monitor SDSH project performance. This aspect of the evaluation considered the status of the HMIS as a mechanism that all levels of the MOH could use to generate, compile, analyze, communicate, and use health data in decision-making and policy planning.

Finding 1

Despite several disruptions faced by the project, SDSH affirms and project data indicate that the project met or surpassed virtually all of its targets. A notable exception was the TB targets, a shortfall that was due at least in part to an abrupt withdrawal of Global Fund support. The project‘s confidence in making this claim rests on its faith in a rigorously implemented performance monitoring system, which does, in the evaluation team‘s judgment, produce generally accurate and reliable information.

Discussion

Prior to 2008, SDSH used five indicators to determine NGO performance in meeting its contractual targets. The NGOs, however, knew the indicators that SDSH was using to track its performance could theoretically focus on attaining ―incentive‖ targets. In 2008, SDSH established 18 performance indicators; since that time, it has randomly selected one incentive indicator each year for each NGO.

This indicator is not known by the NGO, driving it to achieve all of their performance targets in order to ensure successful achievement of the unknown incentive target.

SDSH, meanwhile, reinforces its partners‘ motivation by carefully scrutinizing each partner‘s monthly performance report. If SDSH reviewers discover any slippage in partners‘ performance against the indicators, project personnel quickly contact the partner (by email, letter, phone call, etc.) to discuss the issue, and examine the steps needed to correct the problem. The project‘s on-site technical and financial advisors also follow up with the partner to ensure that appropriate corrective measures have been taken.

As designed and implemented by SDSH, this performance-monitoring system produces timely, high-quality data that accurately track partners‘ progress—or lack thereof. To be sure, the system has a small number of vulnerabilities. These include (1) health care provider ―data entry fatigue,‖ a possible result of providers‘ obligation to painstakingly enter data for each patient into cumbersome SDSH registers; (2) errors associated with the transcription of data from one register to another; (3) the lack of any regular assessment of the quality of data submitted by the PPSs; and (4) software artifacts in the Microsoft Excel data sheets used by the project. Taken together, however, any errors that these factors might introduce into the overall performance monitoring system would be relatively minor.

SDSH project staff explained the project‘s shortfall in meeting its TB performance targets by noting that when they had elaborated their TB objectives and activities in 2007; moreover, the MSPP National

Tuberculosis Program had substantial support from the Global Fund for TB interventions in Haiti. They were not overly concerned that only 4% of the SDSH budget was planned for TB activities because many of the needed inputs (e.g., drugs, trainings, equipment) were provided via the National

USAID/HAITI: SDSH PROJECT EVALUATION 4

Tuberculosis Program. Unfortunately, Global Fund support for these activities ended abruptly in 2010 and to date neither SDSH nor other parties have been able to fill the funding gap. SDSH advises that it will continue to advocate through ongoing technical discussions with MSPP counterparts for greater collaboration and interaction between the MSPP HIV and TB programs.

Finding 2

Discussions with NGO and MOH PPS managers indicate that they have been motivated by the project to develop innovative ways to meet their contractual targets, and extend their reach to target populations (e.g., use of community ―rally posts,‖ fielding mobile teams of physicians and nurses to remote areas, coordinating health information messages with traditional healers, etc.). However, none of the 18 indicators used to monitor NGO performance provide any gauge of the NGOs‘ progress toward improved organizational/institutional growth or their readiness to graduate from the SDSH project.

Finding 3

The SDSH project has made little progress in strengthening overall data management procedures within the MOH. Only in March 2011 did MOH set up a technical committee on HMIS to review and harmonize the Monthly Statistics Summary Form (MSSFs) produced by the PPS). The HMIS is not yet functioning ministry-wide, and SDSH has not attempted any meaningful integration of the project‘s data management system into the broader MOH data management system.

Discussion

At the central level of the MOH the Unité de Planification et d'Evaluation (Planning and Evaluation Unit, or

UPE) is responsible for compiling health system data. UPE collects data from MOH, NGO, and private

PPS through MOH reporting channels according to the sequence (PPS Commune (where a

Communal Health Unit is functioning) Department Central Level. Although data are reported

―upward‖ to the UPE, there is little feedback downward from the UPE to the PPS, a shortcoming that the UPE attributes to a lack of staff and resources. Factors that might alter this process and facilitate departmental use of data for management purposes remain unlikely in the absence of a national strategic information plan or policy or a single HMIS that could serve all elements of the MOH. Lacking such a unified data management system, the MOH departments continue to rely on parallel and uncoordinated data collection and reporting systems, including the upward-oriented MOH system discussed above, the

SDSH project reporting system, and (in selected departments) the President‘s Emergency Plan for AIDS

Relief (PEPFAR) reporting system. There does not appear to be any intention on the part of the MOH to consolidate or harmonize these disparate systems into one system that could provide management-relevant information to all users.

Recommendation

Since the MOH has a standardized data flow system in place from PPS via the commune to MOH departments, and from there to the UPE, SDSH should align its project data reporting system and data flow procedures with that existing system. Such integration would help strengthen the overall MOH health data reporting system, as shown in Figure 2.

USAID/HAITI: SDSH PROJECT EVALUATION 5

Figure 2. Ideal National HMIS Integrated Data Flow

HMIS Data Flow and Reporting

MOH/MSPP(SDSH embedded)

Department(SDSH TA embedded)

Health Facility (Zone Cibles & NGO PPS)

Unite Communal de Santé / NGO office

Routine data collection of Harmonized MSSF

Coordinate PPS data, Supervision and Feedback

Data Collation, Supervision, Feedback and DQA

Data Consolidation, Planning, DQA, Feedback ,Donor Reports, Recommendation

Prior to the end of the project, SDSH staff should convene a forum of department- and commune-level data producers and users to explore next steps in harmonizing the various data systems in use at those levels.

Finding 4

There is a strong culture of data use at the MOH central level, demonstrated by UPE‘s interaction with other agencies of government, and that unit‘s publication of quarterly and annual reports for GOH and public use (although data are frequently incomplete). That culture is much weaker at the MOH department level, where the team found little interest in data use for strategic planning, for aggregation of data into one health information system, or for staff interaction on data use. At the MOH department level, the SDSH technical advisor is often the only person involved in the collection, monitoring, and reporting of SDSH data. The advisor attempts little mentoring and/or inclusion of MOH staff in those efforts, further contributing to the perseverance of separate, nonintegrated data systems at that level.

Discussion

Most of the MOH and NGO PPSs visited by the evaluation team had designated data management personnel on their staff. All were able to produce a data management manual distributed by SDSH, including an indicators definition guide for reference. Moreover, most of these personnel had good knowledge of the indicators and knew how to transcribe them from client service registers into project reports. Most indicated that they generally met their monthly submission deadlines for SDSH reports, but were usually late in submitting their MOH monthly reports.

Once those reports reached the MOH department offices, SDSH data—processed most frequently by the SDSH technical advisor (TA)—were reviewed and dispatched quickly to SDSH headquarters. MOH data, however, were processed in a more desultory manner and without critical review by data managers or more senior department staff. Some MOH department offices have a departmental

USAID/HAITI: SDSH PROJECT EVALUATION 6

epidemiologist, statistician, and surveillance officer on staff, but such personnel do not generally coordinate their work with each other or with the SDSH technical advisor. Few of the MOH personnel responsible for monitoring and evaluation (M&E) tasks received training in that function, nor have they attended regularly scheduled data review meetings.

Among the more significant consequences of the MOH departments‘ lack of attentiveness to the management utility of data is the non-use of those data to (1) monitor drug and commodity supplies to help prevent stock-outs, and (2) track patients who are referred from a PPS to another health facility.

Recommendation

SDSH technical and financial advisors should try to more fully engage MOH staff in the processing and oversight of SDSH project data to build the capacity of MOH personnel in data management and to promote closer alignment of SDSH and MOH data systems.

Recommendation

The MOH should, with assistance from the follow-on project, explore ways to more effectively utilize MOH data to monitor medical supplies at PPSs and track referral patients.

Finding 5

Service providers‘ entry of data into the SDSH client data registers is excessively time-consuming, and could limit health care workers‘ time and ability to interact with their patients. Moreover, each register is compiled on an annual basis, requiring service delivery personnel to manually transcribe all of the data from prior-year registers into new annual registers. Thus, by the fourth year of the project, service data for a single patient may have been transcribed four times into as many registers.

Recommendation

Managers of the follow-on project should explore the feasibility of more user-friendly, less time-consuming technologies for data entry and retrieval, including the use of electronic tablets if the price for such tools declines in the near future.

Finding 6

The MOH departments‘ capacity to manage their ZC programs is not as well developed as the capacity of the NGOs, since to a significant extent the MOH/ZC programs are still managed directly by SDSH.

Discussion

The successful application of a PBF system requires leadership, capacity to plan and optimize resource allocation, and a functioning health care delivery system. Those management skills, tools, and infrastructure are not yet adequately developed at the MOH department level. Until the MOH departments develop greater managerial strength, it might be appropriate to measure performance in the ZCs against a less demanding standard than might be applied to NGO programs.

Recommendation

As noted in this report, the follow-on project should provide for long-term capacity-building at the MOH department level in the management and implementation of a PBF contracting system.

USAID/HAITI: SDSH PROJECT EVALUATION 7

PROJECT IMPLEMENTATION

The scope of work (SOW) that USAID/Haiti prepared for this evaluation called for, among other things, an examination of factors that positively or negatively affected SDSH‘s ability to deliver the services required to achieve expected results. USAID encouraged the evaluation team to look beyond specific topics noted in the SOW, but also asked the team to assess the role of the project‘s specific characteristics and their role in supporting or impeding the achievement of project goals. These factors included project staffing and structure; project management; USAID management; and the partnership model employed by the project, with a special focus on the project‘s application of the PBF model to relations with its project partners.

Finding 7

The project did not have a smooth start and experienced a number of disruptions during implementation. These disruptions included the following:

Delayed project launch: In March 2007, USAID decided to eliminate several activities that had been called for in the request for proposal and that had been included in MSH‘s successful proposal. Steps to reconfigure the project‘s content and decrease its funding delayed execution of a USAID-MSH contract for approximately six months, to August 2007.

The 2008 hurricanes: The devastating hurricane season of 2008 pummeled Haiti with four huge storms in the space of 30 days, killing hundreds and perhaps thousands of people. The SDSH project, like most of the USAID projects then under way, worked with disaster relief personnel to address population needs after the storms, as directed by USAID. While not in any way comparable to the event‘s tragedy for the people of Haiti, the storms and the donor response also affected the capacity of the SDSH project to continue effective project implementation. Specifically, the influx of donor agencies and NGOs working on hurricane relief produced a bidding contest for Haitian staff, and several SDSH personnel migrated to the better-paying jobs in those organizations.

Political instability: The post-hurricane civil unrest and food riots complicated the project‘s ability to maintain service delivery in high-strife areas of the country. (For instance, project personnel reportedly wore bulletproof vests as they carried out project monitoring.) Instability also impeded

MSH‘s ability to recruit a replacement for their original chief of party, who had moved back to the

United States upon receiving a promotion within MSH.

COP turnover: The project has had three chiefs of party in four years and, at the time this report was being written, was searching for a fourth. Those changes inevitably contribute to disruptions in project orientation, internal and external communications, and ―drift‖ in project implementation.

January 2010 earthquake: The January 2010 earthquake was the most profoundly significant event in

Haiti‘s recent history. Again, as in the instance of the hurricanes of 2008, SDSH personnel were enlisted into USAID‘s broad-based response to the crisis. And again—but on a much larger scale— international NGOs and donor organizations were able to recruit away SDSH staff to an extent that is still affecting the project‘s personnel structure. Both SDSH personnel and key staff members of the project‘s NGO partner organizations and central and department MOH personnel were also drawn away by the relief organizations. Moreover, the higher salary scales of these organizations continue to complicate recruitment of replacement staff at all levels of the SDSH project. The

―cluster groups‖ hosted by various United Nations agencies reportedly serve as job fairs for Haitians seeking to increase their income.

The cholera epidemic: The cholera epidemic that emerged after the earthquake poses a serious long-term threat to public health in Haiti. The SDSH project, although lacking contractual authority to address cholera, continues to support the international effort to control the disease via the project‘s mandate to work on the prevention and management of diarrheal disease.

USAID/HAITI: SDSH PROJECT EVALUATION 8

Contraceptive stock-outs: Parliament‘s unexpected rejection on a GOH decision to procure contraceptive commodities led to major stock-outs of FP supplies at service delivery points, and setbacks in the project‘s ability to meet its and reproductive health (RH) goals.

Temporary loss of subcontractor participation in the project: USAID‘s decision in 2009 to extend the project through August 2012 (later revised to September 30, 2012) called for the termination of

MSH‘s subcontracts with JHPIEGO, JHU-CCP, and the NGO Fondation pour la Santé Reproductive et l’Education Familiale (FOSREF). The agency‘s decision was apparently driven in part by funding constraints, but also by an effort to bring the project into closer compliance with the GHI, as the project‘s reliance on several U.S.-based partners seemed to fly in the face of GHI‘s call for greater reliance on host-country institutions. MSH argued for a continuation of its subcontracts with JHU-

CCP and JHPIEGO, but the USAID-MSH negotiation process took 10 months to complete, during which time the professional staff of the two organizations were effectively immobilized. The project‘s communications director was not able to wait for resolution of the issue and moved on to another job, and her position in the SDSH project remains vacant. The departure of FOSREF from the project (as a subcontractor) negatively impacted the ability of the project to successfully implement efforts to expand and intensify the involvement of youth. If that subcontractual role had been continued, FOSREF would have provided technical assistance and guidance to the other 27

NGOs to strengthen and/or install youth-focused activities at those other NGOs (it did so with seven of the NGOs before the subcontract was terminated). Despite that loss, the evaluation team observed that most of the NGOs‘ field programs include youth-focused elements, even if these are not as extensive as they might have been if the FOSREF subcontract had been continued.

Change of government: A new GOH administration was installed on October 17. Although the incoming administration will in all likelihood be supportive of the project, the change will result in a large-scale turnover of management, professional, and technical personal at all levels (central and departmental) of the government. At the very least, it is likely that many of the project‘s counterparts, and many of the staff who have been trained in project tools (the HMIS, financial accounting programs, logistic systems, etc.) will move on—requiring extensive efforts to train their replacements.

A way to summarize these distractions would be to recall the two assumptions included in the original project design, which noted that achievement of project goals would be contingent on (1) governmental stability and (2) an absence of major natural disasters. The ability of the project to continue project implementation in keeping with its design, and to achieve most of its planned goals despite these and other setbacks, reflects positively on the commitment and resilience of the project and its staff.

Project Staffing and Structure

SDSH project personnel include 50 staff at SDSH headquarters in Port-au-Prince and approximately 20 staff assigned to the 10 departments, with one technical advisor and one financial advisor in each department (the former position is currently vacant in West Department, and is being covered on a part-time basis by a member of the headquarters staff). MSH is the project‘s prime contractor, and is supported by subcontractors JHPIEGO, which focuses on maternal health and family planning/reproductive health, and by JHU-CCP, which focuses on communications/behavior change communications and efforts to increase community involvement.

Many of the 25 technical staff at project headquarters support and backstop the separate health service

―silos‖ (e.g., child health, maternal health, HIV/AIDS prevention and treatment, family planning/reproductive health, nutrition, TB), the elements of which comprise the essential health package promoted by the project. A particular challenge for project management is to ensure that the managers of these nominally separate foci adopt and promote an integrated approach to service delivery, notwithstanding the risk of tunnel vision.

USAID/HAITI: SDSH PROJECT EVALUATION 9

Finding 8

Despite this vertical structure at headquarters, evaluation team observations indicate that the project has been successful in promoting an integrated approach via cross-training of personnel at headquarters and in the field. As discussed above, however, the project has not succeeded in developing a practically useful indicator to measure ―integration‖—an indicator that has eluded health care providers virtually everywhere.

Finding 9

Departures of key personnel and the slow recruitment of their replacements have affected project performance on the margin, but do not appear to have substantively impeded the attainment of project goals. That said, the recent departure of several high-level managers calls into question MSH‘s ability to maintain continuity in project management.

Discussion

During the evaluation team‘s visit, the project‘s third chief of party in four years resigned, as did the project‘s senior financial manager. Those departures came three months after the project‘s deputy chief resigned. Meanwhile, the project has been functioning without a communications director and a formally approved technical director (both designated as ―key positions‖ in the MSH contract) for several months. Six technical advisors in as many departments quit the project for higher-paying jobs with other

NGOs in the wake of the hurricanes and the earthquake. The continuing absence of a technical adviser for the West Department had been previously mentioned.

The reasons for these departures and/or turnover are varied. The chief of party issue was a mixture of the routine (a promotion) and the problematic (internal MSH management issues), with recruitment efforts for previous chiefs of party complicated by unstable conditions in Haiti. The former deputy chief of the project and the chief financial officer left SDSH for senior positions in the MOH and Prime

Minister‘s office. The continuing absence of a communications chief has the potential to reduce the project‘s contractual requirement to produce at least ―one success story per month‖ (according to an

SDSH project manager), but the project has been able to comply with that obligation thus far. The absence of a technical adviser in the West Department is producing some friction with West

Department managers, who informed the evaluation team that the part-time arrangement discussed previously was not serving the office‘s needs well.