Attachment_J.3_Zambia_HMIS_Evaluation_Report_2012.docx

The European Union’s

EDF

Programme for Zambia

EVALUATION OF THE HMIS

Letter of Contract N° 2011/283159

EVALUATION

REPORT

Prepared by Arthur HEYWOOD Paul CHISHIMBA Steven LANJOUW June 2012

The project is implemented by IBF International Consulting In collaboration with EHG (Denmark) The project is financed by the European Union

“The contents of this publication are the sole responsibility of the author/contractor/implementing partner and can in no way be taken to reflect the views of the European Union.”

ATTACHMENT J.3_ZAMBIA HMIS EVALUATION REPORT_2012

Table of content

ACRONYMS AND ABBREVIATIONS 5

6. HMIS Improvement Matrix 48

1.

Acronyms and Abbreviations

2. EXECUTIVE SUMMARY

The evaluation team was asked by European Union (EU) to evaluate Programme Estimate (PE) 3 and spent 30 days reviewing background documents, interviewing HMIS stakeholders at national, provincial, district and facility levels and providing feedback to a number of different fora.

0. To evaluate whether the revised Health Management Information System (HMIS) is integrated, flexible and responsive to the needs of all users

1. Determine the extent to which staff capacity has been built at all levels to ensure sustainable functioning of the revised HMIS

2. Evaluate strengthened Information Communication Technology (ICT) infrastructure for effective operations of the new HMIS

3. Determine improvements in the utilisation of data and information generated by the revised HMIS

4. Evaluate the effectiveness of the adopted project implementation structure

5. Provide recommendations and formulate a plan of action to address gaps for developing a sector wide HMIS

The EU has funded three programme estimates (PE) to support the Ministry of Health (MoH) from 2006 to 2011 to strengthen HMIS. The HMIS has been extensively evaluated over the past 10 years (2003, 2005, 2008, and 2009 [twice]) and all have found that the fundamentals are in place with

1. A solid “pipeline” of information starting with a paper system at facility level, computerised at district and sent to national level

2. Indicators and related data elements are in place but these are fragmented and little care is given to ensuring data quality, so nobody trusts HMIS data. As a result data is variably used

3. Human resources are in place and have been trained at all levels, but their HMIS skills are weak and they do not adequately use the data that they have

4. ICT infrastructure is present and (largely) adequate at all levels Management of HMIS is weak, with multiple stakeholders setting up parallel (competing) systems that place unnecessary burdens on overworked health care providers and fragment financial oversight.

The mission has adopted a systems approach to this assessment and has developed a number of criteria of what a sustainable Zambian-led HMIS should look like, analysing them using a framework that looks at technical, organisational/ environmental and behavioural determinants within the context of the six World Health Organisation (WHO) building blocks

Findings The HMIS has achieved some good results, particularly in the technical aspects where the HMIS is well designed, with a standardised set of forms for data collection, a computer programme in place for entering data and adequate computer and ICT infrastructure in place. There are also only two major databases functioning: District health Information Software (DHIS) collecting aggregate data and SMARTCARE collecting electronic patient records. On the downside, technical aspects are adversely affected by various organizational and behavioural factors such as the limitations of paper forms at facility level, the limited synchronicity of databases between districts and the financial constraints imposed on data audits that affect the quality of the HMIS.

From an organisational and environmental perspective the picture is mixed. Whilst an organizational structure and culture exists and is steering the HMIS at all levels, the current arrangements are fragile in terms of governance, management and implementation. Elements that characterize this are:

1. There is no encapsulating Monitoring and Evaluation (M&E) plan and no one directorate responsible for M&E, which is spread across 7 directorates in 3 different ministries

2. There is no real champion of the HMIS at the MoH and the M&E Technical Working Group (TWG) does not represent all directorates or ensure participation of all stakeholders

3. The relationship between donors and the ministry are strained because of the corruption scandal which has weakened the Sector Wide Approach (SWAp) coordination mechanisms

4. Roles and responsibilities for HMIS are not clear at any level and there are no mechanisms for pooling of resources and expertise for M&E

5. HMIS activities are centrally driven, with no real administrative, managerial or financial decentralisation of HMIS activities to provinces or districts

6. There is no fulltime and dedicated HMIS programme management team at any level

7. Integration of different M&E activities is weak, with vertical programmes and their funders setting up parallel data collection processes independent of the HMIS. Each different M&E system (Programme monitoring, Joint Annual Review (JAR), Performance Assessment (PA) and Planning) uses separate indicator sets

8. The HMIS lacks a problem solving culture which reflects in a failure to solve relatively simple DHIS problems, limited desire to improve data quality, poor feedback to lower levels and weak monitoring of annual plans

9. Available data is not easily accessible, either on the web or in pivot tables and procedures for accessing data are cumbersome.

10. Cooperating Partners and donors have short–term programme funding cycles that do not support systems development and most do not subscribe to the “one plan, one M&E framework, one funding source” philosophy of the Paris declaration

Behavioural determinants are influenced by both organisational and technical factors and are accordingly rather weak

1. There is no culture of information use and little linkage between good data and resource allocation or other incentives

2. There is little demand for quality data and there is no standardised schedule for data analysis or programme monitoring. Data audits that are done are ad hoc and based on individual programmes rather than for the HMIS as a whole.

3. Programme and district managers do not use data regularly and therefore do not effectively communicate data needs to the HMIS unit

4. HMIS skills are low and this reflects in inability to perform simple HMIS tasks, solve HMIS problems or use the DHIS computer programme effectively

5. Motivation is low due to absence of paper based data collection tools, inadequate funding to decentralised structures and lack of feedback from higher levels.

With the long history of a functional HMIS, the adequate financial resources and skilled health workers available over the past seven years, it is felt that Zambia can do better than it is doing currently. Many, largely similar recommendations have been made in the past, but the majority of them have not been addressed or implemented, due to poor leadership and lack of a systematic approach to HMIS strengthening by both MoH and Country Programmes (CP).

Recommendations There are two major sets of recommendations that will be further elaborated in the plan of action: short term “quick wins” that complete activities already started in PE 1, 2 and 3 and longer-term system strengthening. In summary, the HMIS needs

1. A clear strategic direction based on a HMIS strategic plan and ICT strategy, pooling of HMIS resources and decentralized, performance based management and finance

2. Strengthened institutional framework with a strong HMIS programme management team, decentralised management and finances and incentives for producing quality information. A Directorate of Information, Research and ICT would be an ideal scenario to coordinate M&E activities,

3. Improved staff capacity based on a problem-solving national training plan, decentralised training capacity and improved pre-service and in-service training

4. A web-based DHIS for aggregated data that has adequate metadata, produces necessary reports and is interoperable with all other systems, particularly a strengthened SMARTCARE that is expanded to collect in-patient and death diagnoses

3. BACKGROUND

Document Outline This document presents the findings of a 3-man mission to review the routine HMIS in Zambia, specifically looking at the implementation of the third PE, funded by the EU from May 2009 to November 2011.

The document first presents a historical overview of the Poverty Reduction Strategy, National Health Policy (NHP) and the National Health Strategic Plan (NHSP) 2011-2015 as well as the 2004 closure of the Central Board of Health (CBoH) and the 2009 corruption scandal.

Past assessments of the HMIS in 2003, 2005, 2008 and 2009 are summarised and overall conclusions made.

The methodology used by the current assessment is described, outlining the three key analytic frameworks of Systems thinking, the Performance of Routine Information System Monitoring (PRISM) framework and the M&E and Review of National Health Strategies document.

Findings are presented, first an assessment of PE3, then looking at the HMIS using the PRISM framework, then from the perspective of desirable characteristics of the HMIS and finally identifying HMIS problems and their possible solutions. A number of conclusions are presented and recommendations made which will be further elaborated in the Plan of Action (PoA).

Annexes include a situation analysis, terms of reference and the inception report, timetable and list of persons contacted, detailed references and some more details of past assessments (2003, 2008, and 2009)

Historical Overview Currently the Government of the Republic of Zambia (GRZ), within the National Statistical System (NSS) is implementing a Health Information System (HIS) that is monitoring performance of the health sector investment plans as part of the National Poverty Reduction Strategy Paper (NPRSP). As described in both the NHP 2011 and NHSP, 2011-2015, the MoH together with Cooperating Partners are jointly developing support systems for improved health service delivery, such as the HMIS, which is the routine part of the overall HIS.

In order to make the HMIS responsive to the current sector needs for information and effective reporting on poverty reduction programmes, the MoH with support from the EU has since July 2005 embarked on a rigorous process of revising the HMIS system. In this regard, three EU-supported PEs have been implemented. The first two (PE1 and PE2) were evaluated in 2009 and this mission has evaluated PE3, in conjunction with relevant parts of PE1 and PE2.

Poverty Reduction Strategy In July 2002, the Zambian Government officially launched its first PRSP [footnoteRef:1] for the period 2002-2004. In the health sector, in line with the NHSP, the PRSP has the following programmatic priorities: [1: Zambia Poverty Reduction Strategy Paper 2002-204, Ministry of Finance and National Planning, Lusaka, March 2002, http://www.cboh.gov.zm/documents/PRSP%20Final%20Document.pdf]

1. Provision of the basic health care package

2. Allocation of financial and human resources to districts through the SWAp

3. Restructuring of the procurement system

4. Improving access to health care in hard to reach and under-served areas The End of the Central Board of Health Most of the changes seen today at the MoH are rooted in the decision made by the government in 2004, to abolish the CBoH arguing that the CBoH was undertaking similar functions to those of the MoH.

The CBoH was set up (through the National Health Services Act of 1992) to ensure delivery of an efficient health delivery system country wide, while the MoH dealt with policy issues and only played supervisory and regulatory roles. The duplication of functions, cited as one of the reasons behind the dissolution was caused by failure of the MoH, to transfer operational functions to the CBoH and thereby creating parallel structures. A typical example was the creation of a unit responsible for health statistics when the CBoH was already handling this function.

In 2005, Parliament amended the National Health Services Act, which saw the immediate abolition of CBoH. The Board’s roles reverted to the MoH and consequently to a centralised bureaucracy with a lack of accountability for performance. At the same time most powers of districts were removed, their funding was cut to 1/3 and their ability to hire and fire staff was removed.

This has meant that many activities carries out by districts have been reduced in scope, the authority of the District Health Management Team (DHMT) has been drastically reduced and the need for good quality data has been eliminated, as there are no longer functional links between information and resource allocation.

The 2009 Corruption Scandal In 2009, investigations by the law enforcement agencies revealed that some resources in the MoH were alleged to have been misappropriated over a period of years. A forensic audit undertaken by the Office of the Auditor General (OAG) provided an opinion on the funds that were misappropriated. The audit revelations raised serious fiduciary concerns among the CPs leading to the withholding of contributions to the Expanded and Human Resource (HR) baskets. The Global Fund for the fight against AIDS, Tuberculosis and Malaria (the Global Fund) and the Global Alliance for Vaccines Initiative (GAVI) also withheld funding to the MoH due to similar fiduciary concerns.

In recognition of the foregoing, the GRZ and the CPs agreed on a Governance Action Plan[footnoteRef:2] to address the fiduciary risk concerns at the MoH and to strengthen future fiduciary controls, systems and structures in the MoH with a view to rebuilding confidence among the partners providing support to the health sector. Central to public sector accountability and responsibility are issues related to financial management which include processes by which funds are allocated, disbursed, and monitored. [2: MoH (Draft 2012) Governance and Management Strengthening Plan, MoH, Lusaka.]

This scandal had enormous implications on the relationship between MoH and donors, and led to cessation of many donor-funded activities and slowing down of many others as funds dried up. The HMIS was also seriously affected by these changes as the entire SWAp mechanism slowed down, disbursement of funds was slowed, communication between MoH and donors was reduced and, most importantly the trust between partners to put funds into pooled mechanisms was lost.

National Health Policy 2011 & National Health Strategic Plan 2011-2015 The NHSP 2006-2011 addressed the HMIS, stating that the system displayed weaknesses and faced obstacles, amongst others:

· Lack of evidence based decisions leading to weak demand for data

· Difficulty of monitoring progress and lack of reliable data on inequalities

· Duplication of effort

· Weak capacity to analyse existing data and underuse of data

The NHSP 2006-2011 identified the following strategies for HMIS:

1. Strategy 1: Create a harmonized framework for the national HMIS

2. Strategy 2: Strengthen country health information systems

3. Strategy 3: Improve access to and use of health information The NHSP 2006-2011 has now been superseded by a new draft NHP 2011 and a new NHSP 2011-2015 which likewise address Health Information, M&E and HMIS.

The draft NHP 2011, anchored in Zambia’s Vision 2030, sets out policy measures that will guide strategies and programmes in the health sector. The policy provides an overarching NHP framework that amalgamates existing policies in the health sector. Prioritizing Primary Health care (PHC) services, hospital referral services, human resources development and management, medical supplies and logistics, infrastructure development, legal framework and health financing, the policy mentions health information as a key health system building block, one of the six WHO health systems building blocks. The HIS has as its objective to ensure availability of relevant, accurate, timely and accessible health information to support planning, coordination, M&E of health services.

The NHP 2011 has specific policy measures linked to health information:

1. Provide for timely production and dissemination of health information, education and communication materials to the public

2. Ensure effective use of ICTs in delivery of quality health care services

3. Develop an ICT strategy to ensure that hardware and communication needs are matched with information needs

4. Build ICT capacity and infrastructure

5. Strengthen financial management by implementing the Integrated Financial Management Information Systems (IFMIS)

6. Strengthen human resources management by implementing the computerized MoH human resources management information system and by ensuring that a Health Professionals register system is developed and maintained.

The NHP further recognizes that a National Health Strategy should be based on strong M&E and review of progress and performance as the basis for information, results and accountability. A sound National Health System must be based on a thorough understanding of the health needs, priorities and risks within the country. This requires a logical framework for M&E that is comprehensive, addresses the selection of a core set of indicators: identifying data sources for each indicator and specifies plans for filling data gaps, conducting analysis, data quality assessment, communication and dissemination of results.

The MoH has been implementing significant health sector reforms to attain these goals include alignment to the Millennium Development Goals, successive NHSPs, National Development Plan and PA Framework.

The NHSP 2011-2015, is organized around the WHO conceptual framework of health system building blocks rather than on disease or target group programmes. Implementation of the plan will be within existing policy and regulation, institutional and coordination frameworks as well as on a M&E framework. The M&E framework will be conducted through appropriate existing and new systems, procedures and mechanisms and will be supported by the M&E Sub-committee working under the Sector Advisory Group (SAG).

Under the Plan’s fourth Strategic Direction, a HMIS will ensure availability of relevant, accurate, timely and accessible health care data to support the planning, coordination and M&E of health care services. Key strategies (page xiii) include:

a) Strengthening and capacity building of health information cadre at all levels

b) Strengthening data capture capacity of HMIS and to include other important conditions e.g. non-communicable diseases and eye diseases

c) Roll-out and strengthen the HIMS to all private and public hospitals at community level

d) Strengthen the harmonization and co-ordination of health information systems

e) Support the use of research evidence The national HIS is to be integrated, comprise both routine and non-routine information, will be institutionalized among stakeholders and coordinated as part of the national M&E framework. Routine data sources include the HMIS, the Integrated Disease Surveillance and Response (IDSR), the Human Resource Information System, (HRIS), the Drug and Logistic Management Information System (DLMIS) and the Financial and Administrative Management Information System (FAMS).

Zambia has also developed the SMARTCARE electronic health record system to capture and store individual patient data records. The system is designed to capture data on Anti-Retroviral Therapy (ART), Tuberculosis (TB), Voluntary Counselling and testing (VCT) and Prevention of Mother to Child Transmission (PMTCT) and is operational in 552 health facilities. The system has the potential to feed into the HMIS and act as the national patient data base.

Non-routine sources of health information include population based and household surveys, sentinel surveillance as well as health system assessments. Other sources of information include annual planning processes, JARs, PA reviews, programme progress reports, health facility census, surveillance of causes of death and national census of population and households. Health research, along with a draft health research bill, is under development

In order to make the HMIS responsive to the current health sector needs, key strategies defined (page 63) include:

1. Ensure smooth functioning of all routine and non-routine health information systems

2. Improve the usage of the new HMIS/DHIS system, by reviewing and updating the indicators sets, to ensure adequate coverage of all programmes and levels

3. Improve hospital and community level health information

4. Expand the SMARTCARE system, by ensuring that more health facilities have a fully functional SMARTCARE system

5. Strengthen staff training and capacity building in ICTs and health information systems, particularly HMIS and SMARTCARE systems, through tailoring pre-and in-service training

6. Strengthen linkages and integration of the existing health management systems To monitor and evaluate the current health services delivery and facilitate the development of the health information system, 23 NHSP key performance monitoring indicators (Annex 2 page 76) need to be regularly monitored.

Health Information System Strategic Plan (HISSP) 2009 Zambia, like many other countries, has shown an increased demand for health information and the MoH has elaborated a Health Information System Strategic Plan (HISSP) 2009-2015 following broad consultation and participation of key stakeholders with funding and technical support provided by the WHO and the HMN. The HISSP is a product of a HIS assessment conducted in 2007 and covers all the components of the HMN framework i.e. HIS Resources, Data Sources, Indicators, Data Management, Information Products, and Dissemination and Data Use. The main focus of the plan is to ensure continuous improvement, availability and utilization of information through planning, M&E, and decision making to achieve the Millennium Development Goals.

4.

5. PAST ASSESSMENTS OF THE HMIS

The issues facing the HMIS in Zambia are similar to HMIS throughout Africa and the developing world and have not changed much over the past 18 years, as shown by the following brief summaries of various assessments and related plans done in 1996, 2003, 2005 and 2009.

2003 Assessment of the 1996 HMIS Plan An assessment of the 1996 HMIS plan carried out in 2003[footnoteRef:3] (Annex 7) found a number of achievements and constraints which are remarkably similar to the findings of this assessment. [3: Based on 1) 2000 RHINO presentation : Establishment of an information culture: achievements and challenges in Zambia by Alex Simwanza and Mary Church and 2) November 2003:The HMIS in Zambia – A trace on the implementation steps by Paul Chishimba et al ]

1. Policy weakness imply that there is no clear framework for the HMIS

2. The lack of an overall M&E framework means that there is no clear overview of the interaction between the various components of the health M&E process and there is confusion of the scope of the various systems and confusion over the roles and responsibilities of the HMIS, the HIS and overall M&E

3. A number of subsystems – human resources, drugs and assets and the community based information system remains uncompleted. The Hospital management information system for second and third levels has not got off the ground, and remains weak at primary level. There are no links with FAMS or Human resource data to ensure assessment of efficiency

4. The computer programme has numerous weaknesses... data is often not collected electronically because of limitations in skills and infrastructure. The linkages between the subsystems and links to other “vertical” programmes have not been finalized. Email and internet at district level have not worked well

5. The programme reviews of the system have not occurred and the result is a system that largely does not answer to current problems. The system is not responsive to newly emerging issues and new programmes such as Millennium Development Goals, the Global Alliance, HIV/AIDS and others

6. Fragmentation remains a problem with vertical programmes setting up their own parallel systems to get data more frequently and accurately. These systems do not relate to the HMIS and it appears that the HMIS does not have the capacity to incorporate even the relevant data. Vertical programmes appoint “focal persons” to collect data to make up for perceived shortcomings in the HMIS

7. Leadership has not been consistent or strong from provincial or national level. There has been a decline in respect for the administrative structures and an unclear public information agenda... poor team work, with little participation of senior and programme managers in data use

8. The national working group proposed to consolidate initial national support for the HMIS never developed into an institutional support framework ...; there was no top-level policy support and on-going planning for the post-implementation phase

9. The private and non-profit sector were hardly involved in the HMIS development or implementation process

10. Documentation of the system has been poor, with few guidelines available. A user reference guide and training manuals for hospitals are still lacking

11. There is a shortage of trained HMIS staff at all levels, in numbers and in skills, and the turnover of staff is high, leading to a poor institutional memory. Many HMIS posts are not filled, and staff that are in post do not have skills to perform their roles, and have many other responsibilities... Most trainers have moved on to other activities... so support and supervision of the HMIS are weak

12. Data quality and use remains generally poor, 90% due to human failure. The quality of data varies from place to place and there are reports of falsification of data, particularly at higher levels[footnoteRef:4]. This results in lack of faith in the system, and data is little used for decision making at any level. [4: ibid p 19]

13. The non-routine data collection processes have not been implemented, meaning that there is no cross-checking of the validity of routine data through surveys or record reviews.

Of particular concern to this mission are the many organisational and behavioural challenges to the HMIS described clearly below, which have still not been addressed nine years later.

This assessment was not followed through with an operational plan but was a major stimulus and inspiration for the 2005 assessment and plan described below.

2005 HMIS Assessment and Plan The 2005 consultancy, sponsored by the EU[footnoteRef:5], assessed the existing HMIS in Zambia and found a number of achievements, again similar to the findings of this consultancy: [5: Heywood, A, Nielsen E. and Orzeszyna, S. (2005). Assessment of the Zambian Health Management Information System, (July-August)]

1. The HMIS is basically functional and compares favourably with many African HMISs. It produces and reports on data according to a defined indicator set and uses it to make annual plans right down to facility level

2. Good 1996 plan with excellent principles that deserve to be implemented – namely Decentralised, Action-oriented, Responsive and Transparent (DART), uniform pipeline, links to NHSP - and should be followed in any future HMIS

3. “Data Pipeline” and staff are in place with data collected from most facilities and sent in a systematic way to districts, provinces and national level. A great deal of data exists that could potentially be used to monitor health priorities

4. In spite of rapid turnover, most staff appears able to use the paper tools to collect and collate data. DIOs are able to enter data onto the computers and to report it to higher levels

5. The national indicator set is defined and there is a related dataset linked to it that is well known and used on a regular basis for action plans, performance appraisal and supervision

6. The Procedures Manuals from 1996 are in place at all levels and are used by new HMIS users as reference files. The manuals are not updated for features introduced since 1996

The weaknesses identified were also similar to the current assessment, summarised by the following quote:

“There have been serious slippages in terms of implementing the 1996 plan due to poor leadership and technical support, … weak capacity development, a low demand for quality information (particularly related to local monitoring of MDGs and the NHSP) little use of information and poor ICT use.”

Identified threats included

· Weak top level political commitment to HMIS change and creation of a culture of information

· Loss of District Information Officers (DIO)s and District Management Specialists (DMS)s to greener pastures

· Vertical programme inflexibility and donor funding of narrow interest data collection

· Weak GRZ Funding and commitment of resources to HMIS The consultancy team prepared a work plan for HMIS strengthening, to improve the efficiency and effectiveness of health care delivery through the strengthening of HMIS. The plan called for a strengthened, revised and comprehensive HMIS, focused on improving quality and coverage of essential health services and health status, open to integration with other components. ‘They identified key strategic issues to strengthen the HMIS:

1. Capacity development of all cadres of staff, including intensive skills development through in-service and pre-service training programmes, upgrading of manuals and study of best practice

2. Return to the 1996 HMIS principles of DART health information system, and introduction of the information pyramid

3. Information and communication technology strengthening through making the database more flexible and strengthening of decentralised information centres linked by internet to a central data warehouse

4. Effective use of information through integration of vertical systems, with improved central coordination between stakeholders and sectors so that the information from HMIS can be used to assess output-oriented performance

5. Improved action research capacity is to improve feedback and dissemination and reduce overlap and duplication

6. HMIS staff retention, particularly DIOs, is needed by improving skill and status and ensuring sustainability of systems, procedures and staff They recommended a return to original 1996 HMIS principles “With minimum effort and disruption of the current system, the original 1996 principles of DART, a single information pipeline, integration of vertical systems into HMIS umbrella could be easily achieved. In each district a District Information Office should be established, where all data from all subsystems relating to the district is electronically available in one place, and from where all data is disseminated.

A regular review process should be set up, led by a representative steering committee that will provide appropriate leadership of the HMIS throughout the MoH. This should be complemented by strong, decentralised information management that focuses on action-led, output-oriented performance assessment that responds to local needs and provides real incentives for well-performing districts and facilities.”

The PoA formed the basis for two EU funded PEs 1 and 2 that were carried out from 2006 to 2009 and assessed by the 2009 mission described below.

Assessment of PE 1 and PE2 and Action Plan (2009) This 2009 assessment covered the two EU funded PEs (PE 1 and PE2) and had Terms of Reference very similar to those of this 2012 mission. Most of their findings (See Annex 6) are corroborated by this mission. In summary the evaluation showed the following findings categorised by desired outputs:

Revised HMIS The new HMIS has interesting potential. However, both the paper tools and the electronic application are not completed and not fully functional.

(1) Two major factors hampered smooth implementation of the project:

a. Substantial changes in the organisational environment of the MoH and fragmentation of M&E responsibility within the Ministry

b. Expected capability of the MoH Unit was not realistic

(2) PoAs have been developed for Community, Hospitals and FAMS subsystems but no actions were taken on implementation

(3) The minimum data set was adopted, and workshops were conducted, but indicators and targets are not integrated into DHIS and reports are not produced by users

(4) Parallel reporting systems still exist which means that paper tools are too many

(5) Data quality assessment is still mixed with the quarterly PA.

Capacity of staff developed

(1) Training was trimmed down from five to three days, but in reality staff had two and often only one day exposure rather than a proper training

(2) Little attention was given to the Training of Trainers (ToT) resulting in badly prepared trainers

(3) There was no monitoring and follow up of the cascade training

(4) The curricula for pre-service training were developed but the manuals are too bulky

(5) Quality assurance of the PA uses a separate set of indicators to the HMIS

(6) Paper based information for self-assessment is not fully exploited.

(7) Performance indicators are imposed by the hierarchy, with little analysis or use at lower level

(8) Support of the District Health Information Officer (DHIO) is lacking Strengthened ICT infrastructure of the HMIS

(1) At the time of the evaluation (2009) the database (DHIS) was not yet completed. ... the system is not able to provide the data and information envisaged

(2) Establishment of provincial, district and hospital information centres is the area where the most satisfactory results were achieved ... almost all DMSs and DHIOs went for training and all offices are sufficiently equipped

(3) For the hospitals, in reality nothing has been done

Increased use of available information

(1) Result-based performance rewarding system was not in place

(2) The DHMTs and Provincial Health Management Teams (PHMT) have serious problems using DHIS data for performance appraisal

(3) HMIS, JAR, annual planning and the PA proceed along parallel pathways and each system uses their its own set of indicators

(4) It is advisable to revise the current Planning handbook, Health Sector Performance Monitoring Framework (HSPMF) and PA tools so that all use the HMIS as their basic data source

(5) [There is a need] to upgrade the technical and managerial capacity of M&E Unit to focus on data quality assurance, analysis, reporting and dissemination

(6) At hospitals, only primary health care indicators can be included

Project management structure within MoH

(1) The project was anchored in the Directorate of Planning and Policy (DPP). However the position of Project Administrator was not established. Instead a project administration unit was established comprising junior positions from the ministry with no executive power

(2) The short-term technical assistance (TA) strengthened capacity by building on existing capacities, thereby contributing to long-term sustainability

(3) Two committees were appointed;

(i) The Technical Implementation Committee (TIC) was constituted by the Project Management Unit (PMU)

(ii) The Project Steering Committee (SC) was taken up by the M&E Sub-committee

(4) Project managers were more focused on technical aspects of the project than on management issues. Project management was not effective and the project did not achieve the expected results

(5) Financial control did take place and accountability of the expenditures has been transparent. However, the budget monitoring was not regular, consistent and timely and did not take into account the financial results achieved for re-allocation of available financial resources

(6) Provincial and district HMIS committees were never established, no decentralised HMIS management is taking place: and local quality assurance of HMIS data is non-existent The project has achieved considerable results on technical areas of the HMIS but has had very limited real impact on the management of the HMIS, Recommendations

The evaluation made recommendations (see annex 6) which formed the basis of PE3, which basically were to do the things not achieved in PE 1 and 2.

“The overall recommendation is to speed up the implementation of the activities described in the PE2 and to apply the suggestions contained in the consultant report of December 2008[footnoteRef:6].” [6: Heywood et al 2007 Hospital, Community and FAMS report ]

Mid-Term Review of NHSP (2008)

A major Mid-term review[footnoteRef:7] of the NHSP 2006-2011 in 2008 also addressed the HMIS, amongst many other aspects. Challenges identified include [7: Chabot et al 2008 Mid-term Review ]

(1) The weak quality assurance system

(2) Software compatibility problems, with different versions floating around

(3) The inconsistent naming of health facilities

(4) Need for automated feedback, simple graphical and data analysis automated modules

(5) Conformance to the data flow policy

(6) The new HMIS manual shows gaps concerning self-assessment and analysis of data at health facility level The review found that “Some of the delays and problems could have been avoided by a less fragmented operationalization of the programme, strong leadership with committed team work at MoH. ..... The new system, as it is now, is not reliable, not complete and not sustainable. The DHIS database is hardly populated with data, paper formats are waiting to be entered and for some districts still to be collected. The back-up technical support system is fragile and relies heavily on external sources. The new DHIS has the risk of becoming as inflexible as the old system, pushing for separate information systems and remaining a tool for computer specialists rather than for health care managers”.

Again, most of these findings are similar to those of this mission.

HIS Assessment and Strategic Plan (2009) This review, conducted by a HMN and MoH team using the HMN Health information system assessment framework, with wide participation, did an in-depth situation analysis. Their findings relevant to the HMIS are broken up according to the 5 outputs of the HMIS PEs. (See Annex 9) The assessment identified a number of key strategic, organisational and management issues that need to be addressed for the HIS to succeed, particularly around existing legal and administrative frameworks governing the collection, processing, analysis and dissemination of health information, and policy guidelines and direction on effective coordination of the sector, human resource capacity and material and financial resources.

The situation analysis notes

1. The lack of integration between various constituencies

2. The lack of implementation of existing guidelines

3. The uncoordinated proliferation of data collection tools

4. The poor capacity and inadequate training of HMIS staff

The objectives for the Strategic plan are listed below, and specific interventions proposed to achieve them are in annex 9.

I. HIS Policy, Legislation, Governance, Leadership and Resources

1. To improve the HIS functions in the country with an updated legislation

2. To strengthen the national role, reputation and authority of the HIS

3. To increase the availability, distribution and use of resources for HIS

4. To increase the HIS data completeness and timely reporting among all stakeholders, including the private sector

II. HMIS

5. To strengthen the data quality by further integrating and harmonizing the HMIS and data from other institutions and agencies

6. To strengthen analysis and use of data for decision-making at every level

7. To strengthen the HIS performance by expanding the availability and use of ICT in the health sector III. Other Information systems

8. To improve the data quality and disaggregation in the HR information sub-system

9. To improve the Finance health information sub-system providing the decision-makers with an integrated database IV. Vital Registration

10. To provide reliable, timely and complete mortality and cause of death data by expanding the vital registration coverage

This HISSP has not been formally accepted by the MoH and most of its recommendations have not been implemented.

Conclusions from Past Assessments The problems facing the HMIS in Zambia, noted in the assessments above, are not new. These problems are not unique to Zambia and are seen throughout Africa, where HMIS are in general not as good as in Zambia.

These assessments have been sensible and practical and plans have been feasible and well-funded, yet implementation levels are low. The assessments show that what is needed is not more money, but better management of the money, more strategic leadership, with authority and resources given to support implementation at provincial and district levels With the long history of a functional HMIS, the adequate financial resources and skilled health workers available over the past seven years, it is felt that Zambia can do better than it is doing currently. Many, largely similar recommendations have been made, but the majority of them have not been addressed or implemented, due to weak governance structures, poor strategic leadership and lack of a systematic approach to HMIS strengthening by both MoH and CPs.

This mission is not going to re-hash the findings of the many competent assessments that have been done previously and re-do feasible plans that have been made but not implemented. We build on these evaluations and attempt a paradigm shift to get a “birds-eye” picture of the situation using a systems approach that uses the following systems-based frameworks for our analysis.

1. Systems Thinking for Health System Strengthening

2. PRISM

3. M&E and Review of National Health Strategies approach used by the Commission on Information and Accountability (CoIA)

All the problems identified above have their roots in weak health systems and inadequate HMIS management and leadership. The mission feels that over the years these two issues have not been adequately addressed by either MoH or CPs.

6.

7. 2012 ASSESSMENT

Currently the GRZ, within the NSS is implementing a HIS that is monitoring performance of the health sector investment plans. Described in both the NHP 2011 and NHSP 2011-2015, the MoH with Cooperating Partners are developing support systems for improved health service delivery, such as the HMIS.

The HMIS is a routine facility-based HIS system operating in all 72 districts and 9 provinces of Zambia, paper-based at facilities and computerised from districts upwards. It operates at all levels of the health system, and extends to public, mission and private health facilities.[footnoteRef:8] Since 1996 when the HMIS was first rolled-out, it has contributed towards improved M&E of health sector performance, and documenting disease epidemiological trend. The HMIS has sought to support evidence-based planning, resource allocation and policy implementation at all levels of health care. [8: Health systems can be understood in many ways. The World Health Organization (WHO) defines health systems as “all the organizations, institutions, and resources that are devoted to producing health actions.” This definition includes the full range of players engaged in the provision and financing of health services including the public, nonprofit, and for-profit private sectors, as well as international and bilateral donors, foundations, and voluntary organizations involved in funding or implementing health activities. Health systems encompass all levels: central, Provincial, district, community, and household.]

To make the HMIS responsive to the current sector needs for information and effective reporting on poverty reduction programmes, the MoH with support from the EU has since July 2005 embarked on a rigorous revision of the HMIS system. These processes were assessed by the mission.

Methodology and Approach The methodology used was in the line with the terms of reference (Annex 2) where the global objective of the assessment is “to improve the efficiency and effectiveness of health care delivery system through the strengthening of HMIS, seen as a combination of people, equipment and procedures organised to provide health information enabling concerned stakeholders to make timely and informed decisions and use of information at various levels”.

The assessment was based on the expected outputs of PE3 and alignment to PE1 and PE2. Given that two crucial outputs (data use and project management) were excluded from PE3, the assessment mission was extended (see Inception Report Annex 2) to…