MEEPP_Evaluation_Executive_Summary.pdf

MEEPP-II Performance Evaluation

Draft Executive Summary – October 2016

EVALUATION PURPOSE AND EVALUATION QUESTIONS

GH Pro conducted an evaluation of the Monitoring and Evaluation of the Emergency Plan Progress

(MEEPP II), a $17 million USAID-funded activity to support PEPFAR planning and reporting in Uganda.

The purpose of this evaluation was to comprehensively analyze MEEPP-II’s performance towards supporting PEPFAR data and analytical needs, and assess progress in the transition from reliance on a project-managed database, to the Government of Uganda (GOU) systems for reporting PEPFAR interventions. The findings of this evaluation are expected to contribute to United States Government

(USG) decision on the level and type of support to further strengthen the Government of Uganda

(GOU) data system for PEPFAR reporting and other national needs. Specifically, the evaluation provides an opportunity for USG to identify gaps, including strengths and weaknesses, and gather evidence upon which a post-MEEPP strategy can be based.

PROJECT BACKGROUND

MEEPP II is a 5-year contract funded by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID). The project is implemented by Social and Scientific Systems (SSS) Inc. While MEEPP I focused on developing a comprehensive

PEPFAR implementing partner-based performance management system, MEEPP II was asked to shift its focus to strengthen existing national and district-level M&E systems to support data needs of the key

HIV/AIDS related stakeholders. The project was to have significantly contributed to a strengthened national HIV/AIDS M&E systems allowing Government of Uganda (GOU), PEPFAR, the Global Fund and other development partners to meet most of their data needs from these systems.

EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS

Using a cross-sectional study design, the following four questions were investigated:

1. PEPFAR Reporting. To what extent has MEEPP-II efficiently, consistently and routinely collected, analyzed, and reported data for PEPFAR planning, project management and performance reporting?

2. Transition (MEEPP). To what extent have the data collection, data validation, analysis and reporting systems established by MEEPP effectively transitioned to the national systems pivoted around DHIS-2 and OVC MIS?

3. Transition (GOU). To what extent is the GoU prepared to make the transition into becoming the primary source for PEPFAR data, and to what extent will the data be readily accessible by

PEPFAR and other users?

4. Data Quality and Data Use. To what extent has MEEPP-II contributed towards timely acquisition of quality data and use of data by health facilities, district health managers, GOU and USG

Implementing Partners?

The Evaluation team used several methods, including surveys that targeted all biostatisticians and district health officers in all 112 districts, all IPs, and key informant interviews and focus group discussions with the USG, GoU, and district and health facility staff during site visits, in addition to document reviews.

While the design of the evaluation was relatively strong, a limitation is that data collection focused more on the DHIS 2 than OVC MIS.

FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS

Findings

Question 1: PEPFAR Reporting

MEEPP II provided indispensable support to the USG for OGAC reporting and planning. The MEEPP II team successfully ensured regular, timely and accurate data exchanges for PEPFAR planning and reporting both before and after DATIM. The MEEPP II team supported 6 USG/PEPFAR Country

Operating Plan (COP) planning processes, 12 SAPR & APR reports were compiled & submitted to

PEPFAR/OGAC on time and 4 HIV/AIDS burden tables were produced to assist in performance review and program planning for several COPs. In 2015, the MEEPP II team also started supporting quarterly reporting. Over 70 PEPFAR IPs supported and trained in PEPFAR reporting requirements with 100% of

USG prime partners receiving monthly feedback on data reported to HIBRID, DHIS2 and OVCMIS.

Further, 100% of all IPs have had reported data reviewed against targets and achievements. In addition, MEEPP II team responded with speed to any special requests made by the USG team or the GoU for additional analyses or reports to help with decision making.

Question 2: Transition to GoU M&E Systems

MEEPP II’s transition of PEPFAR reliance for data towards national systems is its greatest accomplishment. Significant efforts were made by the MEEPP II team to build GoU capacity to ensure data availability and reporting, and the team also made some progress towards sustainable capacity for the GoU to ensure data reliability, data analysis, or data use. The team found this to be the case for both the DHIS II and OVC MIS, though with the later built-in checks that cannot be overridden has helped to ensure data reliability. In evaluating MEEPP II’s role over time, the team found that it evolved significantly from one focused on collecting data from partners to focusing on improving data availability in the DHIS 2 and OVC MIS so that these two GoU M&E systems could provide the data needed for

PEPFAR reporting. This was accomplished for most requirements outlined in the PEPFAR NGI indicators, though there still remained indicator requirements captured only by partners especially associated with community datasets and the revised (MER) indicators and entered into HIBRID, a

PepfAR specific MEEPP-supported database.

Question 3: Government Preparedness

While overall ownership and a sense of responsibility for the data in the DHIS II and the OVC MIS was evident, key activities along the continuum of data are not led by the Government. In addition, policies and other supportive infrastructure to ensure data reliability, validity, and analysis are in their infancy and still require significant technical assistance. The team concluded that, at this time, the Government of

Uganda is prepared to make DHIS 2 data available to PEPFAR, but data validation and analysis are still areas that will need to be supported if the DHIS II data are to be used for PEPFAR reporting purposes.

Question 4: MEEPP II Contribution to Data Quality and Use

The Team concluded that MEEPP II contributed significantly to ensuring availability of quality data on

HIV in Uganda. MEEPP has contributed to the timely acquisition of quality data for HIV in Uganda, in several important ways: 1. Provision of a clean data set for PEFAR reports which can be used by MoH, MoGLSD, USG, and other donors, setting the “gold standard” for clean data for Uganda. One challenge is that the clean data set does not have an independently accessed repository, limiting the availability of the data for future usage. 2. Number and timeliness of reports from districts significantly improved since

2012, especially for HIV related reports in the DHIS2; 3. IPs reported significant support for PEPFAR reporting and PEPFAR indicators, with some support for other aspects of the data lifecycle. 4. With regards to data use, the Team did not find explicit activities implemented or supported by MEEPP tied to data usage, although the districts and sites are using their data for making important management decisions from their data. However, it should be noted that this was not an explicit responsibility of

MEEPP II, as other IPs are responsible for ensuring data reliability and use at district and health facilities.

For OVC MIS, data usage was more clearly attributed to MEEPP, due to MEEPP-II supported quarterly meetings which greatly facilitated data analysis and usage at the district and sub-county level. MEEPP II’s main contribution to data driven learning environment and decision making within the Uganda HIV/AIDS program community is timely acquisition of quality data, and producing the gold standard dataset.

Conclusion

MEEPP II provided indispensable support to the Uganda PEPFAR Team for OGAC planning and reporting. Through diligent work, they enabled the team to submit high quality quarterly, semi-annual, and annual reports on time to OGAC. They also served an integral role in helping to harmonize indicators between GoU M&E systems and PEPFAR. MEEPP II’s efforts greatly contributed to increasing the availability of data in GoU M&E systems, particularly the DHIS II and the OVC/MIS. However, although the GoU forms part of the team, MEEPP II staff continue to lead the bulk of data cleaning and validation activities with heavy reliance on IPs for follow up with districts and sites. Also not transitioned were ability to analyze or use the data from the DHIS II for decision making. As of now, the GoU is not prepared to ensure that the DHIS II is the primary source of data for PEPFAR reporting. To become prepared, there is a need to improve relevant policy, advocacy capacity, organizational infrastructure, physical infrastructure, staffing, and financial viability.

Recommendations

PEPFAR

General

Continue to focus on improving the quality and strategic use of GoU M&E systems following

UNAIDS and PEPFAR’s Three Ones Policy- “one national plan, one national coordinating authority, and one national monitoring and evaluation system.1

Consider having staff imbedded where the work needs to happen. This may be within the various USG offices responsible for PEPFAR reporting unless this introduces additional limitations, such as flexibility of movement and physical infrastructure that are difficult to surmount Consider embedding staff within the GoU, in particular, the MOH Resource Center

(now Division of Health Information) responsible for ensuring the quality of the DHIS 2, and with the MoGLSD responsible for OVC MIS. Embedding staff within the GOU would likely increase efficiency and provide the opportunity for mentoring and shared learning in order to increase capacity within the GoU to improve all aspects of the data lifecycle. Interviews with a number of stakeholders, particularly donors and Ministry personnel suggested this same

1 http://www.pepfar.gov/about/c19381.htm recommendation be part of a MEEPP II follow on. Additionally, a document review revealed that this same recommendation was initially made in the MEEPP I evaluation.

Clarify roles and responsibilities delineated for all PEPFAR-funded M&E activities.

Hold implementing partners explicitly responsible and accountable for GoU M&E strengthening in districts/sites that they support

Play an integral role in shaping emerging legal and policy framework around edata in partnership with the GoU to secure USG’s continued access to data from GoU M&E systems

Follow-on Programming

Support planning and reporting needs of the USG Uganda PEPFAR Program, including preparation of country operational plans, quarterly and annual reporting. Specifically, new programming should continue with the Interagency coordination and facilitation as directed by the USG.

Ensure coordination between PEPFAR and the GoU and employ responsive management of the ever changing USG/GoU reporting and planning needs.

As an explicit goal for MEEPP II follow on, include a component on learning and data use that will explicitly focus on building data-driven decision making in Uganda’s HIV community. This activity would likely include knowledge management, communication, and collaboration and learning for PEPFAR, the GoU, and partners of the GoU.

Build additional software tools that can better assist with data validation, analysis HIV data for decision making by all stakeholders

GOU

In order for the GoU M&E systems to become reliable, trusted, and sustainable, The GoU should receive support from the USG and other partners to improve the policy environment around health data. Specific recommendations include elaboration of policies, particularly those for cross-national data exchange as well as others listed in the eHealth Strategy Draft; production and housing of the final dataset accessible to PEPFAR and all other GOU partners at the national level; Investments in a sustainable EMR system that feeds into the DHIS 2 summary reports; extending capacity for data into the DHIS 2 by health facilities, and increasing focus on data use and outreach at all levels.