Attachment_J.2_Zambia_HMIS_Assessment_Report_2016.pdf

HMIS Assessment 2016 Copperbelt and Lusaka Provinces

Page 1 of 54 November, 2016

Table of Contents

1. Executive Summary

1.1 Conclusions

1.2 Recomendations

2. Background

3. Conclusions from Past Assessments

4. Akros Facility Field Assessment

4.1 Facility Level Assessment Objective

4.2 Assessment Process and Facility Profiles:

4.3 Suggestion Outline

4.4 The Information Cycle

4.5 Data Collection and Processing

4.5.1 Data collection tools

4.5.2 Data quality

4.5.3 Digital data entry:

4.5.4 Parallel systems found at the facility level:

4.6 Data Extraction and Mining

4.7 Data feedback and Dissemination

4.7.1 Informal dissemination

4.7.2 Formal dissemination

4.8 Data Analysis and Interpretation

4.8.1 Data discussions

4.8.2 Visual tools for data interpretation

4.8.3 Monitoring and evaluation

4.8.4 Facility level performance

4.9 Decision Making

4.9.1 Client care

4.9.2 Facility Management

4.9.3 Systems management

4.9.4 Service delivery

5. Community Data

5.1 Existing Community structures/groups

ATTACHMENT J.2-ZAMBIA HMIS ASSESSMENT REPORT_2016

Page 2 of 54 November, 2016

5.2 Cadres of Community Health Workers

5.3 Community Health Assistants

5.3.1 Data Capture and Reporting

5.4 Suggestions

6. Data Quality District Level Survey and Implications for DHIS2

6.1 Reasons for Poor Data Quality at the Facilities

6.2 Realistic Solutions for Addressing Afore Mentioned Data Quality Problems

6.3 Facility Level Electronic Data Entry and Feedback

6.4 Additional Observations

7. The Current DHIS2 System Usage

7.1 User Logins

7.2 Returning Users:

7.3 User Roles

7.4 Data Collection

7.4.1 Data elements

7.4.2 Number of data elements by organisation unit level

7.4.3 Data elements which are not linked to indicators

7.4.4 Indicators

7.5 HIA1 2016 Reporting Rates

7.6 Recorded Zeros

7.7 Data Quality

7.7.1 Validation rules

7.8 DHIS2 Tool Use

8. The PRISM Framework

8.1 Technical Determinants

8.1.1 HIS design

8.1.2 Computer software

8.1.3 Data collection and reporting forms

8.2 Behavioral Factors

8.2.1 Motivation

8.2.2 Data demand

8.2.3 Data quality checking skills

8.2.4 HIS Competence and confidence levels

Page 3 of 54 November, 2016

8.3 Organisational Factors

8.3.1 Governance and planning

8.3.2 Resources and financing

8.3.3 Training and supervision

8.3.4 Culture of information

9. Best Practice: Rufunsa

10. Recommendations

10.1 Facility Level Data Entry

10.2 Facility Mentorship

10.3 District Capacity Development

10.4 Stimulate Data Use

10.5 Recommendations for national level

10.5.1 Governance and leadership

10.5.2 Data collection policy

10.6 Stimulate Demand for Data

10.7 Research Community HIS

11. Way Forward

12. Annexes:

Annex 1: Past Assessments of Zambia HMIS

Annex 2: Facility Level Assessment

Annex 3: Facility Level Performance Assessment

Annex 4: District Data Quality Questionnaire

Annex 5: Number per Data Element Group

Annex 6: Percent of Data Elements that are Zeroes by Program and Facility Type

13. References

Page 4 of 54 November, 2016

Abbreviations

ANC – Antenatal Care

ART – Anti-Retroviral Treatment

CBD – Central Business District

CDC – Centre for Disease Control

CFR – Case fatality Rate

CHA –Community Health Assistant

CHAI – Clinton Health Access Initiative

CHW – Community Health Workers

CIDRZ – Centre for Infectious Disease Research in Zambia

CPs - Cooperating Partners

CSO – Central Statistics Office

DART - Decentralised, Action-oriented Responsive and Transparent.

DEHO – District Environmental Health Officer

DHIO –District Health Information Officer

DHIS2 – District Health Information System 2

DIM – District Integrated Meetings

DMS - District Management Specialists

DQA – Data Quality Assessment

EGPAF – Elizabeth Glaser Pediatric AIDS Foundation

EHT – Environmental Health Technician eLMIS – Electronic Logistics Management Information System

EPI – Expanded Programme on Immunisation

EU – European Union

FAMS – Financial Administration and Management System

GIS – Geographic Information System

GMP – Growth Monitoring and Promotion

GRZ – Government Republic Of Zambia.

HIA – Health Information Aggregation form

HIS - Health Information system

HIV/AID – Human immunodeficiency virus infection and acquired immune deficiency syndrome

HMIS –Health Management Information System

HMN - Health Metrics Network

HR – Human Resources

ICCM – Integrated community case management

ICT – Information Communication and Technology.

IDSR – Integrated Disease Surveillance and Response

IMAM –Integrated Management of Acute Malnutrition

JSI – John Snow Incorporated

Page 5 of 54 November, 2016

M&E –Monitoring and Evaluation

MCDMCH – Ministry of Community Development Mother and Child Health

MCH –Maternal and Child Health

MDGi – Millennium Development Goal Initiative

MOH – Ministry Of Health.

NDHO – Ndola District Health Officer

NHC – Neighbour Health Committee

NHSP – National Health Strategic Plan

OiC – Officer in Charge

OPD – Out-Patients Department

PA - Performance assessment

PIM – Project Implementation Manual

PMTCT –Prevention of Mother-To-Child Transmission

PRISM – Partnership for Integrated Social Marketing

RDT -Rapid Diagnostic Test

REMS –Routine Enhanced Monitoring Systems

RHIS – Routine Health Information system

SMAG - Safe Motherhood Action Group

SMS – Short Message Service.

SoP – Standard Operating Procedure.

UNICEF – United Nations Children's Fund

UPS – Uninterruptible Power Supply/Battery

USAID – United States Agency for International Development

WHO – World Health Organisation.

ZEM – Zambia enrolled midwife

ZPCT – Zambia Prevention Care and Treatment

Page 6 of 54 November, 2016

1. Executive Summary The Government of the Republic of Zambia (GRZ), the European Union (EU), and the United Nations

(UN) have launched the “Millennium Development Goal Initiative (MDGi) -- Accelerating Progress towards Maternal, Neonatal and Child Morbidity and Mortality Reduction in Zambia.” This programme will be carried out over a four year period and aims at improving the availability and quality of health and nutrition services in 11 districts in Lusaka and Copperbelt provinces, covering over 30% of the

Zambian population.

In lieu of the above mentioned initiative, the GRZ will embark on an ambitious two year project that seeks to upgrade the existing Health Management Information System (HMIS) while at the same time building the capacity of staff, and strengthening ICT infrastructure, especially on the district and sub-district levels. The goal is to further solidify data collection, quality control, and data use in order to support decision making processes which will positively impact individual beneficiaries (clients and patients) to ensure that they get better quality of care. This will also enhance facility management such as staff, finances, logistics, maintenance, and support systems.

Central and local level decision makers using information generated through the upgraded HMIS system, will be in a better position to monitor the health status (including nutrition), medical stocks, impending epidemics (including Ebola and Cholera), and use the data as the basis for status reports, timely interventions, and medical supply logistics.

In order to ensure that the above mentioned facility level data entry is successful an assessment was carried out by Akros in collaboration with the GRZ. The assessment closely examines the current

(August2016) situation on the ground, specifically regarding the facility and district levels but also the current state of the DHIS2 system.

For this assessment three sub-assessments were carried out.

1. A facility level assessment with over 30 selected health facilities in the MDGi districts

2. A district level assessment with District Health Information Officers (DHIOs)

3. A DHIS2 system assessment of both national and MDGi data

Details and findings for each assessment can be found in the chapters below. The facility level assessment focuses on the data cycle taking decision and action, collection and processing, data mining and extraction, feedback and communication, and analysis and interpretation into consideration.

The district level assessment focuses on reasons for low data quality at the facility and how this can be improved. The DHIS2 system analysis examines the current use of the system along with elements and indicators in the system.

Each of the assessments have suggestions after the listing of the findings in order to demonstrate what efforts are necessary to improve data quality which ultimately improves service delivery. Overarching components such as technical, organizational and behavioral factors are discussed in Chapter 8.

Lastly, recommendations for implementation are given in Chapter 10. These are based on the suggestions from the three sub-assessments and the overarching component findings.

Page 7 of 54 November, 2016

1.1 Conclusions

The problems facing the HMIS in Zambia, noted in this assessment, are not new and are not unique to

Zambia. Zambia’s HMIS is at a better state of use than most countries in the region. Overall there is a good, working data “pipeline” in place, with facilities using standard data collection tools (registers, tally sheets and forms), collecting required data, checking it for quality and sending monthly HIA 1 and 2 forms to the District health Information Officer (DHIO), who enters data onto DHIS2.

There are however at least six parallel paper systems in place, each with their own paper forms and database. Data, most of which is already in the HIA 1 &2 mentioned above, is sent to different

“coordinators” at district level and bypasses the DHIO. There is also a parallel electronic medical record system (SmartCare) that is time consuming, produces no data compatible with DHIS2 and adds little value to the routine Health management information system (RHMIS)

DHIS2 itself is currently grossly under-utilised. The available data quality assessment tools are not used, very little data mining and extraction is done, meaning that pivot tables, visualiser and GIS functions are hardly used. Similarly, the data dissemination tools are hardly used, very few dashboards are made and minimal use made of analytic and interpretation tools. There are some “discussions about data” held at various levels, but these are mainly ad hoc, donor funded and poorly structured.

There is a system in place to do annual plans and to assess performance of each level. Both of these systems have standardised tools, but these tools hardly mention HMIS and neither of them mentions using data from the national data warehouse, DHIS2.

The Community health Information System (CHIS) is still in its infancy and needs to be further reviewed and the community health system that it measures better understood before it is possible to collect coherent data for it.

From a technical standpoint, the HMIS is basically sound with a good design, solid database (DHIS2), relatively good ICT infrastructure (computers and internet) and a proven set of data collection and reporting tools. However very few health staff have access to the DHIS2 database, the management and maintenance of the database is poor, resulting in excessive numbers of data elements, lack of definitions of data elements, non-use of most data quality tools and a general under-utilisation of the system.

Staff are surprisingly motivated by HMIS and see it as important, with much effort put into data quality assurance at facility level. However demand for quality data from higher levels is limited and DHIS2 problem solving ability and skill levels are low.

Organisation and management is the biggest obstacle to effective implementation of the HMIS in the districts and facilities assessed. Governance is weak, most resources are from donors, training is limited at district level and below and the culture of information for action is extremely weak.

Page 8 of 54 November, 2016

1.2 Recomendations

1. Decentralised data entry at facilities of HIA1 and HIA2 data onto the DHIS2

2. Pilot necessary to assess community level data aggregation done by CHAs or assigned community health volunteer using the HIA4 form.

3. Facility Mentorship, through which district level staff sustainably support facilities to improve information collection, analysis and use, in order to strengthen integrated health service delivery.

4. Develop SOPs for data use at each level to stimulate data use and demand for high quality data for decision making. These would include guidelines for facility level data entry, data quality assurance, quarterly mentoring and district level discussions about data, data use in DIMs etc.

5. District capacity development for practical mentorship, using an integrated approach that will promote data quality by stimulating increased demand and use. These trainings will target

DHIOs and district program managers.

6. Improve DHIS2 performance at all levels by using performance tools, eliminating and organising data elements and indicators, add follow-up options into the system to enable and demonstrate improvements made.

7. Governance and leadership that converts existing draft policies and strategies to official documents and then implements them. These policies include Health Information Systems

Policy, Monitoring and Evaluation Policy and Framework, Data Access Guidelines, Routine Data

Management Guidelines.

8. Data Collection policy to be developed and implemented to eliminate parallel reporting systems.

9. Research Community HIS to identify the best mechanisms to strengthen community health services

2. Background The Government of the Republic of Zambia (GRZ) has a long history of relatively successful Health

Management Information System (HMIS) implementations, going back to the 1990s. The following is a brief overview of the HMIS implementation to date.

The 1996 HMIS was implemented using the Decentralised, Action-oriented Responsive and

Transparent (DART) principles. A custom database system was developed by the Zambia Central

Board of Health.

In 2006, the District Health Information System (DHIS) 1.4 system was implemented in Zambia.

This was an improvement, but persistent technical and organisational problems were

Page 9 of 54 November, 2016 experienced at national1 and sub-national levels, with slow reporting times and poor data quality.

In 2013 the Ministry of Health (MOH) deployed the DHIS2 web-based platform to all districts with support from the Global Fund and PATH. During this move to DHIS2, much of the mandate for HMIS-related data collection moved from the MOH to the Ministry for Community

Development and Mother and Child Health (MCDMCH). In this time, an additional 33 districts were created in Zambia, increasing the number of districts from 72 to 103.

In 2015 the MCDMCH was dissolved and responsibilities for HMIS returned to MoH.

HMIS is not typically the center of any health system intervention, but a strong information system is necessary to ensure other components of health delivery are having a positive effect on service delivery and system functioning. A 2005 assessment of HMIS in Zambia at the time looked at capacity devilment, decentralization, integration and technical strengthening, and staff retention, as key goals for the HMIS.2 An assessment conducted of the Rwandan HMIS in 2005 revealed human capacity, change management alongside decentralization, and compatibility of several systems to be the key themes in the HMIS.3 Similar themes are present elsewhere, even today: an assessment of the HMIS in Ethiopia found the bottlenecks of the same origins.4

As time passes, resources develop and change, and technology advances, the key challenges to maintaining a solid HMIS remain, and the political, managerial and human resource (HR) uncertainties still have the most considerable negative impact on overall HMIS stability. As in the work in Zambia mentioned above, to date, there is still confusion within the districts regarding the correct procedures for submitting data.

In order to strengthen the HMIS within the MoH, Akros has been contracted to target certain identified weaknesses and strengths in the 11 MDGI districts (Lusaka, Kafue, Rufunsa, Chilanga, Chongwe, Kitwe, Ndola, Luanshya, Masaiti, Chingola, and Mufulira). HMIS is just one component in the MDGi goal to improve healthcare delivery, addressing gaps in infrastructure at and across many levels. Within HMIS there is a comprehensive strategy to address three objectives, namely:

1. Strengthening human resource (HR) capacity for sustaining the consistent, timely flow of quality data and feedback through the health system hierarchy,

2. Broad DHIS2 support, ranging from hardware support to software configuration and HR capacity, to enhance functionality and maximize utility in the context of programmatic decision making, and

3. HMIS system-wide support to increase demand and use of quality data for program, planning, and intervention management throughout the health system hierarchy.

1 Inconsistent metadata versions were applied throughout the phased implementation approach, which led to problems reconciling datasets from multiple periods.

2 Heywood et al., 2005.

3 RTI International, 2006.

4 Belay et al., 2013.

Page 10 of 54 November, 2016

This assessment will guide the project implementation process and ensure that the decisions made are based on facts presently (September 2016) found within the national, provincial, district, facility and sub-facility levels.

3. Conclusions from Past Assessments

The problems facing the HMIS in Zambia, noted in this assessment, are not new. The problems faced are also not unique to Zambia, they can be seen throughout Africa. However, in comparison to most countries in the region, Zambia’s HMIS is at a better state and use.

Detailed assessments were made in 2005, 2008, 2009 and 2012. These have been sensible and practical and the detailed plans made have been feasible and well-funded, yet implementation levels are low. The main findings from all assessments show that what is needed is not more money, but better management of the money, more strategic leadership, with authority and targeted resources given to support decentralised implementation at provincial and district levels. A more detailed overview and summary of all the assessments made to date, can be found in Annex 1.

However, even though the assessments have all made reference to the above mentioned shortcomings not much has changed from 2005. Most recommendations that have been made in the assessments, have not been addressed or implemented. This is most likely due to shifting governments, changing ministries and responsibilities, and the inclusion of thirty-three new districts.

The Akros consortium is not going to re-hash the findings of the many competent assessments that have been done previously and re-make feasible plans that have been made but not implemented. This assessment will build on those evaluations and harness the rapid evolution of internet access and mobile technology to attempt a paradigm shift in decentralised HMIS management to improve the situation using a systems strengthening approach.

All the problems identified above have their roots in weak health systems and inconclusive HMIS management, procedures, and leadership. The Akros consortium feels that over the years these three issues have not been adequately addressed by either MoH or cooperating partners, and it is hoped that, together with the MoH leadership, the HMIS system can be strengthened and an on-the-ground information management system can be established to significantly improve the HMIS in the MDGi districts. This will lay the groundwork for a possible scale up to further districts in Zambia.

Page 11 of 54 November, 2016

4. Akros Facility Field Assessment In order to gain a deeper understanding of the situation at the facility, community and district levels, two surveys were carried out: one at the facility level, and one with all eleven District Health Information

Officers (DHIO). The facilities were visited with a district officer and an officer of the Ministry of Health headquarters. Details and findings of both assessments can be found in the chapters below. Both rural and urban facilities were visited.

4.1 Facility Level Assessment Objective

The primary objective of facility level assessment was to determine current data collection, reporting, and use practices, in surveyed facilities. The results of this assessment will be used to reveal needs, gaps, and best practices currently in place to inform future planning for updates to the Zambian HMIS.

The assessment was carried out over a period of one month (August – September 2016) using a data collection form (Annex 2) developed for the purpose of gathering data about the facility and community level.

4.2 Assessment Process and Facility Profiles:

The HMIS facility assessments were concentrated in the Lusaka and Copperbelt provinces, in the Lusaka, Chongwe, Kafue, Chilange, and Rufunsa districts and the Ndola, Kitwe, and Luanshya districts, respectively. Thirty two (32) facilities were visited: of these facilities, fifteen (15) of which were considered urban in location, and seventeen (17) of which, peri-urban or rural. In total interviews were conducted with over 50 facility staff health staff.

At each facility, the in-charge nurse, or acting in-charge nurse was the main informant. Each assessment included a visit by a team including one Ministry of Health staff, a district staff member, and two Akros employees. This team sat with the in-charge and reviewed a detailed questionnaire that focused on data collection practices, data use, and facility planning, organization, and capacity. When possible, the team physically examined existing reporting tools, including paper registries and computers.

The Mother and Child Health (MCH) unit was the focus of the assessment, and the services offered at these clinics were in kind. The majority of facilities surveyed offered outpatient services, immunizations, antenatal care, postnatal care, family planning, HIV/AIDs treatment including ART supply and PMTCT services, tuberculosis, under-five and paediatric care, and treatment for malaria, diarrhoea and respiratory infections. Most clinics did not offer inpatient services, though select hospitals did.

4.3 Suggestion Outline

Conclusions were made using two frameworks, the information cycle and the PRISM framework. Both are outlined below and the findings of the facility level interviews are applied to the methodologies.

Conclusions relevant for an HMIS upgrade will be given after findings are reported for each component of the information cycle as suggestions.

4.4 The Information Cycle

The information cycle consists of five main components (See image 1 below):

1. Decision and action,

2. collection and processing, Page 12 of 54 November, 2016

3. data mining and extraction,

4. feedback and communication,

5. and analysis and interpretation.

Image 1: The Information Cycle

The current HMIS focuses largely on data collection in order to send data to the next level (from facility level district province national level). Other components of the cycle need considerable strengthening. The following chapters, starting with component 2 of the cycle, and ending with component 1, will highlight the strengths and weaknesses of the current system.

Page 13 of 54 November, 2016

4.5 Data Collection and Processing

Overall there is a well-designed and functional data pipeline in place with facilities using standard data collection tools (registers, tally sheets and forms), collecting required data, checking it for quality and sending HIA 1 and 2 forms to the DHIO, who enters data onto DHIS2. The current monthly HMIS reporting protocol mandates that all paper-based HIA1 and HIA2 forms must reach the District Health

Office by the 7th of each month. District teams then have 15-30 days to conduct data reviews, follow-ups, and validation activities before the finalized facility-data is manually entered into DHIS2. Forty-five days after the end of each month, central and provincial level teams expect to have data ready for report generation and use in planning.

The process of filling in all relevant tally sheets, registers, and aggregate forms is clear and well established at the facilities. However, the assessment revealed that in-charges cited a need for an improved understanding of reporting forms and the needed calculations in order to better understand the relevance of the data.

In addition, short-staffed clinics are forced to use clinical personnel to do reporting, when they should be providing health care services. Due to the low health professional to patient ratio, nurses/midwives will prioritize patient care over reporting which can make the data results questionable and incomplete.

Health personnel are faced with the dilemma that they face negative consequences both when not submitting data and when not attending to as many patients as possible.

4.5.1 Data collection tools

All facilities initially gather data on paper, utilizing a system of individual records, tally sheets, registers and report forms

Individual patient records are usually an exercise book for Outpatient Department (OPD) or, for

HIV clients, and a specially designed file with loose sheets. This seems to work well, although paper forms did get lost on occasion.

Tally sheets are used to aggregate and track simple counts, such as the number of cases, number of visits, number of pregnancies, etc. Using Tally sheets, some data from these are put into registers that are used for patient continuity of care.

A set of over thirteen registers have been produced by MoH, covering all priority programs. The registers are very cumbersome and health workers tend to re-enter clients each time they come, rather than finding the old records. Thirty-one (31) or 97% facilities are actively using at least ten registries, most of which are paired with tally sheets and report forms that also require documentation.

These data collection tools are distinct for each of the many reporting areas, which correspond to different diseases tracked, treatments given, and other programmatic areas. For example, Antenatal care visits are tracked using an ANC-specific tally sheet and registry.

Tracking anti-retroviral therapy (ART) involves a report form, a tally sheet, and a registry.

Surveillance data on notifiable diseases is submitted to the district weekly.

Data is aggregated from all forms monthly to be sent to the district on Health Information Aggregation

Forms known as HIA forms of which there are four:

Page 14 of 54 November, 2016

HIA1 is the health center disease aggregation form

HIA2 is the health service delivery aggregation form

HIA3 is the hospital level aggregation form

HIA4 is the recently developed community level reporting form which has been rolled out to select districts

The whole facility team spends three to five days at month-end, aggregating data into the above mentioned relevant HIA forms. The data is then entered into weekly and monthly aggregates. This pipeline also incorporates some data collection from the community, which is also aggregated by the facility and sent to the district. Staff at facilities that are not entering data electronically, spend up to five days at month-end, aggregating data; the average amount of time spent aggregating data by staff at the seventeen (17) peri-urban facilities was just over three days. Once completed, the forms are sent to the district.

The transport to the district is a process that often takes the Officer in Charge (OiC) away from the facility for an entire day. Image 2 reviews the overall generation and flow of data. Note that not all facilities have each of these reporting pathways.

Suggestions

Image 2: Information flow diagram

Page 15 of 54 November, 2016

Data aggregation into the DHIS2 at the facility level would be an optimal way to ease the workload of the in-charge. Facility level DHIS2 entry would negate the need to develop weekly and monthly data aggregates as the system can calculate these based on the daily aggregation forms. Internet access at the facilities would allow for near real-time, online data submission.

4.5.2 Data quality

Most facilities do not have a true data quality process in place. A procedure for signing off on the quality of the data could not be found. Almost all facilities cited controlling data through “good supervision,” but fewer than half (15) of the facilities had a true process in place for reviewing data quality. The de facto quality review for those who did do them involves cross-checking tally sheets and registers while aggregating at month end. Three facilities mentioned sitting together doing aggregation as a team to check arithmetic.

One major data quality constraint is timeliness. The paper-based HIA1 and HIA2 forms need to be physically transported to district offices, which requires time and transportation resources and has been found5 to result in up to a 45 day reporting delay. The estimated cost to physically transfer data to the district office rural districts ranges between $34,000 and $42,000 USD per year.6 In urban settings, this cost reduces to $8,270 per district per year. Furthermore, the current paper-based system is prone to errors and, as practiced, lacks data validation (e.g., double data entry, thus affecting data quality). The heavy dependence of paper forms can lead to bottlenecks in reporting, such as instances when forms are not readily available or distributed. Planning and decision-making suffer as a result.

Suggestions

Data quality is key to data use. If the data is not trusted, then it will not be used for decision making.

DHIS2 has various automated data quality checks. These need to be trained and implemented by all levels of stakeholders in the HMIS. Further suggestions on the improvement of data quality can be found in Chapter 6. This section focuses specifically on data quality and harnesses the experience of the DHIOs.

4.5.3 Digital data entry:

Currently, data is entered onto DHIS2 at the district. In order to bring the digital data entry to the facilities the following findings are relevant:

a. All but two facilities had computers, but only 72% of facilities had functional computers and still fewer had staff adequately trained to use them. MCH Units in facilities in urban locations, as in

Lusaka district in Lusaka province, had on average 2 computers, with at least one of these working and having been acquired within the last five years. Facilities outside of large urban centres also had one or two computers, but the majority were bought before the staff could remember and were rarely functional.

b. All facilities have decent cellphone coverage, even those that do not have internet set up at their facility.

c. Facilities commonly experience power outages, whose frequency and duration is highly variable.

With the exception of one (1) rural clinic running on solar power, all facilities experience at least eight hours a week of outages, with the seventeen (17) peri-urban facilities averaging five hours

5 In an assessment done by Akros in 2015 6 Figures captured through interviews with district and provincial-level staff during March 2015

Page 16 of 54 November, 2016 a day. Some facilities have outages for up to eight hours at a time. While some facilities have back-up generators, these are by no means ubiquitous, and even those facilities with generators at times run out of fuel to run the generators.

Suggestions

As mentioned above, facility level data entry can play a key role in timely data entry and use by the facility. Laptops and internet access will be required by the facilities as desktop computers lose power regularly. Trainings tailored to the facility and a GRZ integrated mentorship program are necessary for a sustainable decentralization of the digital data entry process.

4.5.4 Parallel systems found at the facility level:

There are a number of parallel systems in place. The tallies and register sheets used to enter the data when seeing the patient are aggregated weekly or monthly not to one report, but to several aggregates corresponding to different donor programs or initiatives that are run by different district program managers. Each of these managers uses their own database or excel file system. Many of these forms are redundant, utilizing the same raw data and even indicators. On average the attending/nurse/midwife writes aggregate data into six forms other than the HIA forms required by government.

Below is a list of the most common forms, other than the HIA forms, which are filled in by the facilities at month end and sent to different parts of the district health Office:

(1) HIV: The LPHO/CDC CoAg Monthly Report form collects over 200 data elements, most of which are on HIA1

(2) MCH:

a. Integrated Static/Out-Reach Monthly Report: The outreach form collects 50 elements already in HIA2, disaggregated to outreach sites

b. Community data: ICCM forms and CBD forms

(3) Nutrition: collects

a. Nutrition data with 10 data elements for each client.

b. Monthly Integrated management of acute malnutrition form (80 data elements)

(4) Environmental Health Technician (EHT): collects weekly Disease Surveillance report with data

150 data elements (32 diseases with 5 categories):-cases, lab confirmation confirmed cases, deaths and a calculated CFR

(5) Pharmacy: collects REMS report, MLIP and specialised ARV forms

(6) HR: collects quarterly HR returns and HR data sheet

These parallel systems need to be made into one national system, as they mean redundant effort for both data collectors and for district staff. Most use exactly the same data as is on the HIA 1 and 2 forms and are a human resource-draining duplication of efforts. Parallel data collection forms must be prohibited by national level and data must be instead incorporated into DHIS2, if it is indeed useful for program management.

SmartCare

SmartCare, the nationally implemented electronic medical record system, has a presence in almost all facilities, though due to several issues, operates in parallel to the paper based register system. Ideally, Page 17 of 54 November, 2016

SmartCare works with an electronic patient card that corresponds to patient data stored on the computer at the facility. In both Lusaka and the Copperbelt provinces the DHIO did mentioned that the electronic card was available and used directly when seeing patients to enter data. However, power outages and computer issues necessitate manual data capture at the time of most visits, followed by only somewhat consistent entry into SmartCare at the end of the day. When data is entered, it is done mostly by staff employed by Centers for Disease Control (CDC), Center for Infectious Disease Research in

Zambia (CIDRZ), or EGPAF. The main constraints of the SmartCare system are:

a. Does not work without electricity (does not provide a backup plan for data entry)

a. Training of MOH officers and staff (those not funded by donors) is inadequate

b. Does not link to DHIS2

Suggestions

The harmonization of these parallel information systems should be a high priority for the MoH and the project needs to work closely with the HMIS units to try to bring these into DHIS2 in the two project provinces.

Hand-in-hand with proper training on data collection tools is the need to standardize and maintain consistency in these tools. In-charges cited high programmatic turnover and computer literacy as reasons for inconsistent or incomplete reporting. As new programs with different funding come and go, reporting requirements keep changing, making it impossible for staff to stay trained and current on procedures..

4.6 Data Extraction and Mining

Data extraction and mining is done in DHIS2 through pivot tables, data visualiser and GIS functions. This allows health workers to analyse data from different perspectives and summarize it into useable information. This, however, is only minimally done, even at the district level. The main reason stated by district officers was that they often did not have the correct user access rights to DHIS2 to be able to conduct these analyses or to put the data into tables and graphs automatically using DHIS2. The combination of limited access and minimal training means that:

There is almost no use of DHIS2 tools (validation rules, trend analysis, max/min values etc.) to assess data quality

Districts do not use DHIS pivot tables, visualiser or GIS but instead export data to Excel to make graphs

No dashboards are produced outside of the MoH Headquarters

Suggestion

Train district officers and facility level staff to use the tools available in DHIS2. Ensure access is given to all MoH staff including those of sub-district levels.

Page 18 of 54 November, 2016

4.7 Data feedback and Dissemination

Data dissemination is the process of communicating the extracted data to a group of identified stakeholders. It also consists of alerting authorities if there are irregularities and initiating feedback loops to engage stakeholders in the further analysis and interpretation of the data. At facility level, data seems to be shared both formally and informally.

4.7.1 Informal dissemination

Informal dissemination is done at irregular intervals, and on demand. Data and information is shared with stakeholders across the district.

Facilities share their data not only with the district office, but occasionally with neighbouring health facilities and community health personnel. Feedback focuses on the trends in their catchment populations, better data collection practices, community participation, health promotion sensitization, care delivery, as well as the positive contributions of their work.

Facilities also engage stakeholders at the community level by meeting with schools, churches, local government, and traditional leaders. This engagement is normally about giving (verbal) information about service delivery (e.g measles outbreaks malaria cases and maternal deaths), and organisational issues (e.g. planning for campaigns) and rarely uses analysed data

NGO sponsored programs, get some data through the district reporting system, but will still visit the clinic directly. Where the NGOs are not satisfied with the formal reporting structures, they often develop the parallel reporting systems referred to above

4.7.2 Formal dissemination

Formally, there is a procedure in place to share data upwards through the HIA form “data pipeline” that is largely adhered to. The information generated from this data is usually not fed back down to the facility on a regular basis. At irregular intervals facilities receive feedback from their district offices through a number of channels.

The District may send direct feedback on a certain month’s HIA1 and 2 reports via a phone call or memo.

District Program Officers may visit facilities to conduct quarterly Data Review Meetings (these are often funded by donors) and “mentorship” sessions (e.g. Rufunsa District is funded by

MDGi). Mentorship involves the Officers providing technical support and conducting performance assessments with personnel who collect and report data.

The DQA guidelines were not performed in any district visited, and no written reports of data quality assessment were found at any level.

District-level Program Officers (program managers) are not seen as stakeholders in DHIS2. Informally a

DHIO may contact a Program Officer if they spot datasets they may interest them. In some districts, Program Officers are given the HIA1 & 2 forms when they are sent from the facility, but do not receive any extracted data, graphs or tables with which they can perform analysis. This interaction doesn’t involve any documented feedback loops or dashboards and Program Officers remain outside of the

DHIS2 process. Because the program officers are not integrated into the data collection and dissemination structure, they do not use the valuable facility level data to make decisions.

Page 19 of 54 November, 2016

Existing procedures can be strengthened by providing DHIS2 access to all levels of health professionals and empowering them with the skills to analyse and use the data within DHIS2. This is especially relevant for Program Officers in order to make evidence-based decisions in their areas of expertise.

4.8 Data Analysis and Interpretation

Facility staff see patients, note their disease, prescribe medication and hear about shortages which currently is the main source of information on how they are performing or where bottlenecks are. There is little analysis or feedback reaching facility staff and most information is offered ad-hoc and not based on comprehensive data. Thus, central-level MOH staff discourages facility staff from basing decisions on impressions.

For this reason, a formal data analysis and interpretation protocol needs to be developed at both the facility and the district levels. Formal data analysis and interpretation is currently restricted to those programs that provide paper templates for graphs (EPI, MDGi) and ongoing supportive supervision.

However, when projects and support ends the facilities often do not keep up the procedures.

Additionally, the different projects support different thematic areas such as HIV or mother and child health, so during the project these specific areas are analysed but others with less funding are overlooked.

Some effort is made to analyse and interpret data, mainly program specific data, using cumulative graphs at facilities and Excel at the district level. DHIOs and Program Officers do not use DHIS2 for analysis, and the “sharing interpretation” functions are hardly used. Listed below are findings from the assessment concerning data analysis and interpretation.

4.8.1 Data discussions

Discussions about data are intended to be held monthly at facility level, but several facilities acknowledge that these do not happen consistently. All districts conduct quarterly activities to review data with facilities. Some gather personnel from facilities together for a meeting, and others will conduct visits by district staff to facilities. These visits involve a review of reporting forms and usually are spent explaining how to do reporting to facility staff. To date these activities lack some critical measures:

a. Appropriate guidelines for data use or interpretation for facilities or districts,

b. Linkage of graphs to monitoring of annual plans

c. Written or computerised self-assessment

4.8.2 Visual tools for data interpretation

Almost all facilities had visual displays of data in the in-charge office. These were generally hand drawn graphs, of the below listed content. Half of the facilities had up-to-date data, the remaining did not have data beyond 2015. Charts found were related to:

a. Expanded program for Immunisation (EPI) - cumulative coverage

b. Mother and child Health – trends of key interventions such as Antenatal Care (ANC), deliveries

c. Outpatient department (OPD) top 5 diseases

Page 20 of 54 November, 2016

In addition to being out of date, graphs were not readily interpreted by facility staff. For example, targets as provided by the district and the Central Statistics Office were noted on the graphs, and once a target threshold was met, there was no data graphed beyond this point.

DHIO use of data within DHIS is almost nonexistent. Each representative from the district described an arduous process of entering data as received into DHIS, and then copying the aggregate figures from DHIS to Excel to make graphs. Additionally, in two (2) district offices, there were graphs displayed using non-sensicle scales.

4.8.3 Monitoring and evaluation

In terms of data use, monitoring and evaluation of plans are weak. There is no data use in making plans and no M&E framework in the planning structure

a. The “culture” of using data for planning, monitoring and evaluation of programs is not well developed (though being pushed by some DHIOs, donors)

b. Indicators are not well understood or widely used, with most staff using raw numbers

c. Targets are set at district and given to facilities, but are never monitored or adjusted based on performance. For example, the facility sets immunization goals to fully immunize 3.5% of the population. If the catchment area of facility has a population of

30,000 people, the facility will aim to fully immunize and track the immunization of the first 950 individuals, but beyond that target, may not carefully track immunizations.

4.8.4 Facility level performance

Performance assessment does not use DHIS2 data

a. The current GRZ performance assessment tool has over 200 components. Even services that are not delivered at the facility are assessed. For example number of births attended by a doctor. In facilities where there is no doctor the facility would receive a negative assessment because 0 births were attended by a doctor.

b. No linkages to DHIS2 were found – all seems to be done without hard data.

c. Quarterly mentoring by integrated district team to prepare for District Integrated

Meetings (DIM) (funded by CDC, MDGi)

d. Quarterly DIM meetings attended by all facilities. Unfortunately, projects are not coordinating with one another. This results in an average of 3 meetings per quarter, where almost the same data is discussed. This is a huge drain of personnel to the already understaffed facilities. Quarterly meetings with facility level staff are currently funded by:

The district themselves at irregular intervals

MDGi at regular intervals for mother and child health

CIDRZ at regular intervals for data related to HIV

Global Fund at regular intervals for data related to TB

Page 21 of 54 November, 2016

DHIS2 can be used to support performance assessments at each level, and provide graphs, tables to facilitate the (mainly service delivery) areas of assessment where data is available within the DHIS2.

However, facilities need basic guidelines to strengthen the existing facility data quality assessment process, district and provincial information meetings (DIMs and PIMs), and develop draft templates to allow program managers to interpret data.

In addition, interventions by donors need to be coordinated and integrated so that all funds are going into one integrated data support activity, rather than the current haphazard process. The current redundant quarterly meetings are taking the already understaffed facility personnel away from there posts where they are most needed.

4.9 Decision Making

Decision making was generally limited at facility and district level, as little decentralisation of decision making has occurred to empower action by facility management. There are no guidelines to help health workers know what decisions can be made, or how to use data to inform them. Additionally, facilities receive very little funds from the government to upgrade their facility. Facility grants are extremely small (for example, Ndola Mile 21 HPA facility receives only 600ZMK per month) so they do not have much flexibility about how to spend their money.

Decision making is assessed at three levels, client care, facility management, and systems management.

4.9.1 Client care

Paper data is collected for all clients using the system of individual records, tally sheets and registers described above. This is the basis for most decisions made on individual clients.

Further, though an incredible amount of data is captured on all these paper forms, it cannot be leveraged for decision making at the facility level. This data is aggregated and sent to the district on monthly forms but does not seem to benefit the facility itself.

The electronic SmartCare system is deployed for HIV in many facilities assessed, and in a few selected facilities, for MCH as well. Service providers say the system is beneficial when it works, allowing them to access client data immediately and providing continuity of care. However, the system seldom works due to nonfunctioning computers, failure of electricity and short lifespan of the UPS(20 minutes).

4.9.2 Facility Management

All facilities make annual plans, following the MoH guidelines for planning at each level of the system.

These plans make no reference to the DHIS, More often facility level workers go back to their HIA 1 and

2 report forms to get data. There was minimal reference to these plans for routine monitoring and evaluation during the interviews conducted.

Performance Assessments (PA) are done quarterly or semi-annually using MoH templates. The performance tool was updated in early 2016. However, there are still close to 200 aspects of facility performance, most of which are outside of the control of the managers at facility level. Again, the PA

Page 22 of 54 November, 2016 makes minimal reference to the HMIS and no reference at all to DHIS. The updated performance tool can be found in Annex 37.

Further, performance assessments do not follow up on past assessments. This means that there is little incentive to actually improve or follow up on sub-par findings before the next assessment.

4.9.3 Systems management

System management looks at the six WHO building blocks (Information, Service delivery, Human

Resources, Finance, Drugs, Governance), and the data collected to measure these.

The HIS itself is reasonably well designed, with a standard set of national indicators and data elements, the data is systematically collected and transmitted upwards. The computer software (DHIS2) is adequate, customizable, and open source but IT infrastructure (computers and internet) needs to be improved, at the facility level.

Implementation at the district is not well followed through and thus, the benefits at the facility level are not yet visible. There has been an uncontrolled increase in the number of data elements, quality is not well controlled and use of the DHIS2 is sub-optimal. There is little effort to monitor performance of the

HMIS itself.

In addition to DHIS2 there are a number of other information systems in place that are not streamlined with DHIS2. These have been put in place for a number of reasons:

a. The quality of data coming in from the facilities was not sufficient as it lacked completion, timeliness, accuracy and relevance.

b. Data was not made accessible to the cooperating partner.

c. Certain data elements only need to be reported on for the cooperating partner and are not found in the national data collection structure.

Facilities are mainly reporting…