Attachment_J.4_Zambia_HMIS_Review_FINAL_Report_2009.pdf

European Development Fund

PRBS/HMIS Project

(9 ACP ZA 005)

Ministry of Health

Zambia

Evaluation Report on the performance of the project against the two executed programme estimates

Erik Nielsen

Bruno Piotti

Maurice Coenegrachts

June 2009

ATTACHMENT J.4 ZAMBIA HMIS REVIEW FINAL REPORT 2009

End evaluation of the Zambia PRBS/HMIS project performance against programme estimates I & II

Table of Content

Table of Content

Abbreviations

1. Executive summary

1.1 Background and introduction

1.2 Revised HMIS that is integrated, flexible and responsive to the needs of all user

1.3 Capacity of staff developed to ensure sustainable functioning of the HMIS

1.4 Strengthened ICT infrastructure of the HMIS

1.5 Increased use of available information is facilitated

1.6 Project management structure set up within MoH with Zambian leadership, decentralised management and facility based implementation

2. Evaluation Report

2.1 Background

2.2 Methodology used during the evaluation

2.3 Revised HMIS that is integrated, flexible and responsive to the needs of all user

2.2 Capacity of staff developed to ensure sustainable functioning of the HMIS

2.4 Strengthened ICT infrastructure of the HMIS

2.5 Increased use of available information is facilitated

2.6 Project management structure set up within MoH with Zambian leadership, decentralised management and facility based implementation

3. Annexes

Annex 1 - Terms of Reference

Annex 2 – Workplan

Annex 3 – People met

Annex 4 – Bibliography

Annex 5 – Tools in use at health centres in Zambia (over last 15 years)

Abbreviations

ACP

African, Caribbean, Pacific Group of States

ADB African Development Bank

AFP Acute Flaccid Paralysis

AIDS Acquired Immuno Deficiency Syndrome

ARI Acute Respiratory Infection

ART Anti-Retroviral Treatment

ARVs Anti-Retroviral Drugs

AU African Union

AWP Annual Work Plan

CARITAS Catholic Health Initiatives

CBHMIS Community Based Health Management Information System

CBHG Community Based Health Groups

CBoH Central Board of Health

CBHIS Community Based Health Information System

CHAZ Christian Health Association of Zambia

CHW Community Health Worker

CMAZ Churches Medical Association of Zambia

CP Cooperating Partners

CRS Catholic Relief Service

CSO Central Statistical Office

CSP Country Support Paper

DANIDA Danish International Development Agency

DART Decentralised, Action-oriented, Responsive and Transparent

DFID Department for International Development

DHB District Health Board

DHIO District Health Information Officer

DHMT District Health Management Team

DHO District Health Office

DHS Demographic and Health Survey

DPT Diphtheria, Pertussis, Tetanus

EC European Commission

ECHO European Commission Humanitarian Aid Office

EDF European Development Fund

EPI Expanded Program on Immunisation

EU European Union

FAMS Financial & Administrative Management Systems

GIS Geographic Information System

GRZ Government of the Republic of Zambia

HAHC Hospital Affiliated Health Centre

HAST Integrated HIV, AIDS, STI and TB Program

HBC Home Based Care

HC Health Centre

HIS Health Information System

HIU Health Information Unit

HIV Human Immunodeficiency Virus

HMB Hospital Management Board

HMIS Health Management Information System

HSPMF Health Sector Performance Monitoring Framework

HRD Human Resources Development

HSR Health Systems Research

HSSP Health Sector Support Program

ICT Information and Communication Technology

IDA International Development Agency

IFMIS Integrated Financial Management Information System

IDSR Integrated Disease Surveillance and Response

IP In-Patient

JAR Joint Annual Review

KISS Keep it Simple and Sustainable

LAN Local Area Network

LDC Least Developed Countries

LIMS Logistic Information Management System

M&E Monitoring and Evaluation

MCH Maternal and Child Health

MDG Millennium Development Goals

MoFED Ministry of Finance and Economic Development

MoH Ministry of Health

MoU Memorandum of Understanding

MTR Mid Term Review

NDP National Development Plan

NGO Non-Governmental Organisation

OPD Out-Patient Department

ORT Oral Rehydration Therapy

PEMFAR Public Expenditure Management And Financial Accountability Review

PEPFAR Presidents Emergency Plan for AIDS Relief

PLWHA People Living With HiV/AIDS

PPAZ Planned Parenthood Association of Zambia

PRSP Poverty Reduction Strategy Paper

STI Sexually Transmitted Infection

SWAP Sector Wide Approach to policy/programme

TA Technical Assistance

TB Tuberculosis

TBA Traditional Birth Attendant

UCI Universal Childhood Immunisation

UN United Nations

UNCTAD United Nations Conference on Trade and Development

UNDP United Nations Development Programme

WAN Wide Area Network

UNICEF United Nations Children’s Fund

WHO World Health Organization

ZMK Zambian Kwacha

ZNBTS Zambia National Blood Transfusion Service

1. Executive summary

1.1 Background and introduction

According to the Terms of Reference (TOR) the consultancy should produce two integrated results, namely

(1) a detailed Evaluation of the project impact on the specific objectives of the project, including an assessment of the current state of the country health management information system,

(2) a Plan of Action for development of a sector-wide HMIS for poverty reduction monitoring and evaluation.

This document presents the findings and the recommendations of the evaluation of the project impact.

As laid down in the TOR the consultancy should evaluate the performance of the PRBS/HMIS project against the following five outputs:-

(i) Revised HMIS that is integrated, flexible and responsive to the needs of all users;

(ii) Capacity of staff developed to ensure sustainable functioning of the HMIS;

(iii) Strengthened ICT infrastructure of the HMIS;

(iv) Increased use of available information for action and policy; and

(v) Project management structure set up within MoH with Zambian leadership, decentralised management and facility based implementation

With these tasks in mind the evaluation team reviewed the background documents for the project, e.g. the Assessment Report and the Plan of Action that formed the basis for the development of the two Programme Estimates, was taken through a guided tour of the software application that was developed during the project, reviewed the forms, standard operating procedures and manuals that have been developed, worked out a questionnaire for a field visit to selected provincial offices, district health offices, and health facilities, carried out the field visits, and interviewed information officers, managers, cooperating partners and other stakeholders.

In summary the evaluation showed the following results:

1.2 Revised HMIS that is integrated, flexible and responsive to the needs of all user

The new HMIS has interesting potentials: When completely developed the new HMIS will be an excellent tool to provide the managers with flexible, responsive and integrated routine data that contribute to the assessment of the performance of the health sector, to the planning cycle and monitoring and evaluation of the objectives, and quick action and response. However, both the paper tools in the health centres and hospitals and the electronic application are undermined by several bottlenecks, are all, but not completed and fully functional.

The National set of performance indicators adopted with active participation of stakeholders.

The HMIS team created trust, partnership and commitment among stakeholders. However the bad aspect was a heavy delay in the implementation, instead of October 2006, Copper Belt

Province’s pilot ended after mid 2007 and the paper tools’ roll-out occurred country-wide

Arthur Heywood, Erik Nielsen, Stanislaw Orzeszyna: Assessment of the Zambian Health Management

Information System, July-August 2005, Euro Health Group

2 Arthur Heywood, Erik Nielsen, Stanislaw Orzeszyna: Strengthening of the Zambian Health Management Information System, Plan of Action, July-August 2005 through a provincial phased approach from October 2007 to May 2008. Two major among other factors hampered a smoother implementation of the project: (1) substantial changes in the organisational environment of the MoH and fragmentation of M&E responsibility within the

Ministry: the Sub-Directorate of M&E is the main depositary of the M&E activities, supervises the HMIS and its development and the regular review of the National Set of Indicators, but it still sharing functions with the Sub-Directorate of Planning, the Directorate of the Technical Service Support (Performance assessment-PA). Separate set of indicators are still run by vertical programmes under the Directorate of Public Health &Research and by the Directorate of Clinical Care for Drugs, procurement and logistic; (2) expected development capability of the MoH Unit not realistic: staff too often charged of routine tasks instead to be fully devoted to the development duties and a deficit of leadership has been aggravated by the absence of a permanent advisor, national or external TA.

The minimum data set was adopted, and the relevant consensus workshops were conducted, but multiple and iterative adjustments still on-going for quite some time and the final version of the paper tools and of the Procedures Manual, version 1.4, dated December 2008. This manual contains also the initial data dictionary defining the nature and the best use of 220 indicators include in the tools; a more expanded and complete data dictionary, including metadata, still to be produced. Targets are part of the usual 3-year Operational Plans, at province and district level, yet they are not integrated in software application (DHIS, last version 2008) and both these targets and the standard reports are not easy to be produced by the users in the current application (Pivot Tables).

Collection and collation tools have been substantially revised, yet paper tools are too many, recording, tallying and reporting too cumbersome, parallel reporting systems still exist, according to the small sample of staff interviewed, during the significant but not representative districts and facilities visited. Analysis tools revised and introduced, Data Quality Assurance

Manual elaborated, however the staff has not been not trained in the procedures and the data quality assessment are still mixed with the quarterly performance assessment conducted by the province to districts and facilities.

HMIS is not yet comprehensive, largely because of factors that are not under the direct control of the MoH, e.g. data on mortality and cause of deaths are scarce, irregular and unreliable due to the well known weaknesses of the national Civil Registration services. Needs assessed and plan of action for three subsystems which are under the influence of the MoH have been developed (Community Based, Hospitals, FAMS), however for unknown reasons the Human resources sub-system not considered in PE2 and no action were taken on implementation of the other three subsystems. It is important to establish the Community health information sub-system and urgently integrate data in the HMIS database, to update the Hospitals plan of action produced in 2007 and pilot a set of tools and applications before country roll-out, to extend immediately a FAMS countrywide, as an interim solution, applying the recommendation of the 2007 Plan of Action, and before the introduction of IFMIS and to promote the improvement of the existing HR database.

1.3 Capacity of staff developed to ensure sustainable functioning of the HMIS

The training process

Several project resources including infrastructure development have been shifted to training.

Travels and training and workshops absorbed high costs. In spite of this training has been trimmed down from the planned five to the centrally decided three days, but in reality staff had two and often only one day exposure rather than a proper training. Staff knows how it could be better but is frustrated because the skills to apply the HIS were not transmitted, and for that reason cannot work SMART.

Little attention was given to the TOT with as a result badly prepared trainers. There was no systematically internal monitoring and follow up of the cascade training to be applied.

The curricula for pre-service training medical and paramedical institutes were developed to high standards but the manuals are too bulky to be realistically integrated in the running courses.

Quality assurance and assessment

Quality assurance exists in the context of the Performance Assessment (PA) but the quality systems proposed in the tool developed for this project have not been divulgated or trained to the users. The PA is done at all levels but uses a separate and additional set of indicators then the HMIS.

The lack of clear targets and indicators is a major handicap for service assessment. Few facilities and district apply innovative systems, which both affects staff dynamics, enthusiasm and proactive community links. The HMIS is handcuffed by the hard and soft ware lags.

Meanwhile paper based information to maintain self assessment is not fully exploited as could be.

A number the bottlenecks in the HMIS management would be identified with improved self assessment skills at central HMIS.

The performance indicators are those imposed by and transferred to the hierarchy, with little use and analysis at own level, because of lack of training and insight. Support of the DHIO is often lacking because of the process of civil service reform, blurred retention schemes and subsequently demotivation. Self assessment skills can be introduced in the Health Facilities and with district support on paper based information, moving away from technical approach of HMIS towards daily institutional use of the information.

1.4 Strengthened ICT infrastructure of the HMIS

Strengthened ICT structure

At the time of the evaluation the development and design of the database (DHIS) that was expected to be complete in September 2008, was not yet concluded. The serious delays that have been the order of the day rather than the exemption have many explanations, however the delays have had a negative impact on the development of the system that is not yet able to provide the data and information envisaged at the start of the project.

Although the expected result of establishing a database in all provincial and district offices has been achieved, the functionality of the database is critically lagging behind what was intended, e.g. platform independency, lack of uniformity, no report formats except pivot tables, local capacity not yet satisfactory, no integration with subsystems, Some of the features expected to be developed have actually been created, e.g. adaptation issues, data entry simple and easy, links to spreadsheets and pivot tables, to local realities.

Strengthen provincial, district and hospital information centres…

This activity is the area where the most satisfactory results were achieved although there are considerable variations in the rate of success. The best results have been obtained in the provincial offices where almost all DMS went for training in the new HMIS. However the software literacy varies considerably. All provincial offices are sufficiently equipped with computers and other ICT equipment however maintenance is still a major problem.

All district offices have information officers who have been basically trained in the new system.

There are, though, still problems, e.g. training was too short, new DHIOs are not properly trained, new information officers will replace the existing ones, thus institutional memory may be lost, Although all districts appear to have a sufficient number of computers, distribution of the computers is an issue as well as maintenance.

For the third category that was expected to benefit from the project, i.e. the hospitals, in reality nothing has been done.

Develop an integrated, web-based data warehouse….

Development of a web-based data warehouse has only just started at the central level however some people claimed that no decision so far had been taken on the accessibility for users from the ministry, the provinces and the districts.

1.5 Increased use of available information is facilitated

The introduction of the new HMIS is too recent (December 2008) and not yet fully operational

(see the limits of the DHIS, which is not yet able to produce easy reports), therefore, in May 2009, the evaluation team can not conduct a full appreciation and measure the impact of the new package on the improved use of information at different levels.

As far as the district Operational Plans are concerned they reflect the NHSP priorities, but there is no clear reference to the MDG progress report. However, the activities on the same areas of the major health MDG are targeted in the plans.

Concerning linking the information to output-oriented management performance assessment and incentive scheme the status during the time of the evaluation was that the result-based performance rewarding system was not in place. The DHMTs and PHMTs interviewed declared to have serious problems using new HMIS data for performance appraisal due to DHIS1.4 limits (see above the difficult reporting function). It was evident that the planning at various levels is a common and well established practice countrywide and since 1996-1997

(introduction of the old HMIS) there is a well established habit of the self-assessment on quarterly basis using the “Triple A approach” (Assessment, Analysis, Action). However, it was evident that the two processes, planning and the PA, proceeded along parallel pathways: the Planning Handbook and Guidelines are not yet updated and do not orient officers on the utilisation of the HMIS data and set of indicators during the elaboration of the plans, in addition the Joint Annual Review (JAR) team uses its own set of indicators during the annual process, and finally the two processes are under the responsibility of two different Directorates in the

MoH.

There are several institutional arrangements that hopefully the MoH is going to undertake in order to enhance coordination and to transform the current fragmented and parallel processes of data use for planning, monitoring and performance assessment in a unified and coherent process at provincial, district and facility levels. On the one side, it advisable to revise the current

Planning handbook and tools and revise the current HSPMF and PA tools and start the rewarding scheme. On the other side, it is urgent to improve the HMIS database and to make it able to produce reports with more and well integrated data from budget/expenditure, human resources distribution and drugs/logistics supply and to upgrade the technical and managerial capacity of M&E Unit within the MoH. Finally, it should be focusing on data quality assurance, analysis, reporting and dissemination.

As can be expected the hospitals, especially mission and (previous) mine hospitals have well developed health information and management systems. The main problem is that there are no standardized forms and that MDG and disease programme indicators cannot or can only be included in primary health care forms.

The other subsystem to be developed for health information is the community. Although a large range of activities is there, they are not standardized. The collected information is not used and not transmitted. Before standard information can be gathered and used, the Community Health

Worker (CHW) curriculum and CH strategy needs to be developed, which is planned to move ahead this year.

1.6 Project management structure set up within MoH with Zambian leadership, decentralised management and facility based implementation

Project Management

In line with the recommendations in the plan of action the project was anchored in the

Directorate of Planning and Development (DPD) however the position as Project Administrator was not established. In stead a project administration unit was established comprising junior positions from the ministry with no executive power which left the Imprest Administrator (IA) in charge of all (petty or more significant) administrative matters.

The organisational framework incorporated two technical coordinators with four technical sections but no administrator in charge of the administration unit which in our opinion caused a less successful implementation than envisaged.

The provincial and district HMIS committees that were supposed to be the local strategic bodies were never established. The implementation plan further envisaged short-term technical assistance (TA) to the project on specific activity areas where in-house capacity could not been expected. This took place. It is also worth acknowledging the overall goal of the project to strengthen the capacity by building on existing capacities as having been achieved and thereby contributing to long-term sustainability.

Two committees were appointed; (i) the Technical Implementation Committee (TIC) and (ii) the Project Steering Committee (SC). In compliance with the overall strategy of the project to build on and work within the existing structures of the ministry the TIC was in essence constituted by the PMU and the responsibilities of the PSC taken up by the already existing

Health Sector M&E Subcommittee.

It is our opinion that the institutional structure of the project was well planned and actually led to a feeling among the ministry staff of ownership of the project. Although a good plan unfortunately it was not carried out in an optimal way.

It is evident that the project managers were more focused on technical aspects of the project, e.g. to develop national indicator set, to set-up database, to improve forms and features, to train in the use, than on management issues. Noticeable are the lack of decentralised management structures, i.e. HMIS committees, and the lack of monitoring of the performance of the HMIS.

An analysis of the actual expenses compared to the budget clearly demonstrates that the project management was not as effective as desired and thus the project did not achieve the expected results.

Although the result is not impressive it should be noted that financial control did take place and that the accountability of the expenditures has been transparent. However, the budget monitoring was not regular, consistent and timely and did not take into account the financial results achieved for effective re-allocation of available financial resources.

Results of the intervention on project management of HMIS

The ministry saw the support from the EU as seed funding that would help government and her sector development partners make a renewed commitment towards improved resource allocation for poverty reduction monitoring and evaluation in the health sector, e.g. by having access the more reliable routine HMIS data for management purposes.

It is argued in this report that the project actually achieved ownership and sustainability and improved availability of reliable HMIS data. Whether this has lead to improved resource allocation for poverty reduction monitoring and evaluation is a matter outside this assignment.

Although the project has had a general positive impact on monitoring, there is still room for improvement, e.g. by introducing real performance based financial monitoring, i.e. monitoring of budget on objectives, results and activities.

At lower levels no decentralised management is taking place: the envisaged committees have not been established and HMIS is not sufficiently action oriented, responsive and transparent.

Quality assurance of data is almost non-existing and planned interventions are therefore not based on facts and consequently not responsive to events.

The project has achieved considerable results on a lot of technical areas and features of the

HMIS but the project has had very limited real impact on the management of the HMIS, at central and at local level.

2. Evaluation Report

2.1 Background

Under the 9 th

European Development Fund (EDF) for Poverty Reduction Budget Support (PRBS), the European Union provided support to Zambia for poverty reduction programmes.

This support was meant to accelerate implementation of identified poverty reduction programmes for national development. The Poverty Reduction Strategy Paper of 2002-2004 and now the Fifth National Development Plan (FNDP) put emphasis on social equity and improvement in population health was identified as an important pro-poor contribution towards national development. In order to ensure effective monitoring of health interventions for poverty reduction and socio-economic development, key health indicators were selected both under the National Health Strategic Plan 2006-2010, and indeed the FNDP. The selected health indicators, reviewed regularly, are meant to provide a measure of health sector performance and support development policy dialogue. Hence the need for strengthening routine health services information systems in order to provide quality data for measuring these indicators.

Strengthening of the national Health Management Information System (HMIS) at all levels, is deemed critical to improving health sector policy implementation, and monitoring and evaluation.

In 2005, a detailed assessment of the Zambian HMIS was conducted by an independent team of consultants funded under the 9 th EDF Technical Cooperation Facility (TCF). The assessment found that the Zambian HMIS was basically functional at all levels of the health care delivery system and compares favourably with most African HMISs. The system had a defined indicator set and regularly collected data elements; data collection and reporting tools were in use and the flow of information was clearly set out using the “one channel” principle. There was regular quarterly analysis of routine data with reasonably good coverage for some MDG indicators, though many MDGs were at that time not being monitored. There was eminently poor integration of vertical programs and administrative information into the routine HMIS, quality of data was not checked and the system of vital registration was weak Most staff were found inadequately trained in HMIS procedures and there was not much faith in the results coming out of the HMIS. In light of this, five key strategic issues were identified to strengthen the HMIS:

i) Capacity development of all cadres of staff, including intensive skills development through in-service and pre-service training programs, upgrading of manuals and study of best practice sites.

ii) Return to the 1996 HMIS principles of Decentralisation, Action oriented, Responsive and Transparent health information system, and introduction of the information pyramid.

iii) Information and communication technology strengthening through making the database more flexible and strengthening of decentralised information centres that are linked by internet to a central data warehouse.

iv) Effective use of information through integration of vertical systems, with improved central coordination between stakeholders and sectors so that the information from HMIS can be used to assess output-oriented performance. Improved action research capacity is needed to improve feedback and dissemination and reduce overlap and duplication.

v) HMIS staff retention, particularly District Information Officers, is needed by improving skill and status and ensuring sustainability of systems, procedures and staff

Based on the five (5) strategic issues identified, a comprehensive Plan of Action was developed, outlining key deliverables for strengthening the then existing HMIS.

From the Plan of Action developed by an independent Team of consultants, the Ministry of

Health successively developed two Programme Estimates (PEs) whose primary objective was to address the identified weaknesses of the then existing system. Both PE I and PE II were developed and executed with funding from the 9 th EDF for PRBS II. Implementation of the extended PE II successfully ended on 21 st December 2008. Major deliverables, in line with the

PEI and PE II were achieved. To date, the Zambian health sector is operating a revised HMIS a primary health care level.

However the implementation of both PEs was not without technical and administrative challenges. In additional, although the Plan of Action outlined in detail interventions needed to have a comprehensive sector HMIS able to report on service development at all levels (primary and tertiary care levels), implementation fell short of resources to review the system for the tertiary level (Provincial and national level hospitals).

Development of a sector-wide (community, district, provincial and national) reporting system which incorporates all the levels, is deemed important for effective performance monitoring of the health sector and various poverty reduction programmes.

We wish to underline that in accordance with the Terms of Reference the purpose of this evaluation was to conduct a detailed evaluation of the PRBS/HMIS project performance against the set objectives and deliverables under both PE I and II and not the performance of the HMIS as such although the two issues are closely integrated.

The specific objectives of the consultancy were described as follows:

i) Evaluate whether the revised HMIS is integrated, flexible and responsive to the needs of all users;

ii) Determine the extent to which staff capacity has been built at all levels to ensure sustainable functioning of the revised HMIS;

iii) Evaluate strengthened ICT infrastructure for effective operations of the new HMIS;

iv) Determine improvements in the utilisation of data and information generated by the revised HMIS;

v) Evaluate the effectiveness of the adopted project implementation structure; and

vi) Provide recommendations and formulate a plan of action to address gaps for developing a sector wide HMIS

The present report documents addresses the issues no. i) to v) about and suggests recommendations for the way forward that eventually will lead to a plan of action for a suggested continuation of the project. The formulation of the requested plan of action will be elaborated in a separate document.

2.2 Methodology used during the evaluation

The evaluation team developed a questionnaire and visited a sample of 4 provinces (of total of

9) and facilities in 10 districts (of 72). The questions raised were based on the assessed needs and plan of action developed in September 2005, and the Programme OVIs of the Logical Framework Matrix, (of the Programme estimates 1 and 2 (PE 1&2). The questionnaire is developed by adapting formats of the Procedures Manual Quality Assurance of HMIS, delivered in April 2008, and not yet implemented in the HMIS. The questions assess the different levels at different stages of input, process, and output.

2.3 Revised HMIS that is integrated, flexible and responsive to the needs of all user

In the detailed assessment of the Zambian HMIS that was conducted in 2005 by an independent team of consultants five key strategic issues identified for an effective and successful HMIS revision and upgrading were:

1. Enhancing capacity development.

2. Building on 1996 DART principles: Decentralised, Action-oriented, Responsive and

Transparent information system.

3. Strengthening the information and communication technology.

4. Encouraging an effective use of the information at all levels.

5. Securing HMIS staff retention.

Based on the assessment report of July 2005

, a 3 year project has been developed to be based in the Ministry of Health and funded by the European Union. The project was based on five major outputs echoing and tackling the above key strategic issues.

As stated, the primary purpose of the present end-term review assignment is to evaluate the performance of the EU/PRBS/HMIS Project (9 ACP ZA 005) against the two executed programme estimates and draw recommendations that can be used for the secondary purpose, i.e. to formulate a plan of action for addressing gaps for development of a sector-wide health management information system (HMIS) for poverty reduction monitoring.

The two Programme Estimates (PE) were developed and submitted to the donor, the European

Development Fund (EDF), and approved for funding the PRBS/HMIS/Ref: 9 ACP ZA 005. The PE1 covered six months starting in 2006, the PE2 covered 18 months starting in 2007 and ending in December 2008.

The two PE closely replicated the 2005 project overall goal, that is “to contribute towards improved monitoring of poverty reduction in Zambia with emphasis on the Millennium

Development Goals”, the purpose, that is “to strengthen the capacity of the Ministry of Health

(MoH) to monitor health sector performance, particularly at district level”, and five outputs, that are expressed as five results, as follows:

1. Revised HMIS that is integrated, flexible and responsive to the needs of all users;

2. Capacity of staff developed to ensure sustainable functioning of the HMIS;

3. Strengthened ICT infrastructure of the HMIS;

4. Increased use of available information is facilitated;

5. HMIS project set up and managed effectively.

The core result no.1 is analysed here and the Table below lists the concurring activities as they were described in the Logical Frameworks annexed to the PE1 and 2.

Programme Estimate 1

Expected Results/Activities

Programme Estimate 2

Expected Results/Activities

1.1 Identify performance indicators through

discussion with all stakeholders:

- HMIS managers, district and provincial

- managers, priority vertical programme managers, policy makers and cooperating

3 Arthur Heywood, Erik Nielsen, Stanislaw Orzeszyna: Assessment of the Zambian Health Management

Information System, July-August 2005, Euro Health Group partners

1.2 Based on the indicators, develop the

following :

1.2.1 a minimum data set with clear definitions and a dictionary

1.2.2 relevant targets based on strategic and action plans, Consensus National Workshop

1.2.3 Implementation

strategy, key performance indicators and minimum data set

1.3 Review HMIS system design, data flow

and periodicity of reporting

1.4 Revise data collection, collation and

analysis tools and setting up quality assurance procedures

1.5 Assess needs and plan for unfinished

subsystems planned in 1996:

Hospital HMIS, Finance, drugs, logistics, assets, Human Resources system

Community based information system

1.1 Needs assessment and plan for

Hospital HMIS, Community HMIS and FAMS

1.6 Annual consultative review of HMIS

indicators and design

1.2 Create organisational set up to ensure

regular review of HMIS as part of the overall

M&E framework Table 2.2.1: Programme Estimates 1 and 2 – Activities and Expected Outputs.

Before starting the analysis, it will be useful to adopt a definition of the term “results”.

According to the United Nations Development Group (UNDG), which is in charge of monitoring the MDG progress and revised the OECD-DAC Key Term Glossary on RBM

(2002), the definition is as follows

Results - The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention.

It is also important to clarify that the term HMIS is used in Zambian health sector and in the current project in a broader meaning than the one usually referred to in specialised literature.

The term is generally synonymous of routine health information system (RHIS) and referred to any data collected regularly and frequently in the health facility and their community extension. However, the HMIS Unit is the only entity in the Zambia MoH formally devoted to organizing and managing the overall statistical data originated from all health-related sources, population-based and institutional/facility based. The electronic application DHIS1.4 is considered to be the database representing the core device on which to build the data warehouse serving the entire health sector. The staff in charge of the M&E Sub-Directorate produces

4 UNDG, Results-based Management (RBM) Terminology, Final, June 2003, web:

http://www.undg.org/index.cfm?P=224;

A Aqil. T Lippeveld and D Hozumi, PRISM framework: a paradigm shift for designing, strengthening and evaluating routine health information systems, Health Policy and Planning 2009; 1–12, March 2009,

p. 10.

http://www.undg.org/index.cfm?P=224 statistical information and M&E documents and statements, aggregating and integrating data from different MoH Directorates and multi sectoral sources. Therefore, the Zambian HMIS is going to be developed in the future comprehensive HIS, as defined in the WHO-Health

Metrics Network (HMN) and elsewhere, rather than restricting itself into a sub-system providing the information to the simple management of the health services. We are going to continue to use the term HMIS in the rest of the report, but the features and the performance results analysed will be spreading well beyond the borders of the standard routine health information system.

When assessing the achievements and the implementation of the activities related to the result

no. 1: “Revised HMIS that is integrated, flexible and responsive to the needs of all users”, three aspects will be discussed, as follows:

i. Which are the results achieved against the programme estimates?

ii. How well did the project perform against the programme estimates?

iii. What is the best way forward for further improvement of the Zambian HMIS?

2.2.1 Performance against Programme Estimate

One (simple) way of evaluating the performance would be to check to which extent the project has achieved the expected results. In order to do so the Objectively Verifiable Indicators (OVI) included in the log-frame would have to be consulted. PE1 and PE2 expected results are examined together because the delay in the actual implementation of the PE1, which was extended significantly from the original planned six months to in reality more than a year, reduces the meaning of conducting two separate analyses. The table below illustrates the result’s balance after the end of the project in December 2008. The activities are listed in one single column and have been slightly re-adjusted for making more evident the corresponding

OVI.

Programme Estimate 1 and 2

Result 1: Revised HMIS that is integrated, flexible and responsive to the needs of all users

Expected Results Indicators Achieved results

1.1 Identify performance indicators

through discussion with all stakeholders:

- HMIS managers, district and provincial managers, priority vertical programme managers,

- policy makers and cooperating

Partners

An agreed national set of indicators with numerators and denominators.

Achieved. HMIS team created trust, partnership and commitment among stakeholders. Marked delay in the implementation, instead of

October 2006, Copper Belt

Province’s pilot ended after mid 2007 and the paper tools’ roll-out occurred country-wide through a provincial phased approach from

October 2007 to May 2008.

1.2 A minimum data set with clear

definitions and a dictionary

- Consensus National Workshop

- Review HMIS system design, data flow and periodicity of reporting

A data set and dictionary that is based on the indicator set.

A revised HMIS system design and implementation strategy accepted by national consensus workshop.

Achieved. National Consensus Workshop held in

November 2007. However, multiple and iterative adjustments of single data elements and indicators have gone on until end of 2008.

6 Health Metrics Network, Framework and Standards for Country HIS, Second Edition, WHO, Geneva, 2008, p.8-9.

Final version (v.1.4) of the

Procedures’ Manual dated

December 2008.

1.3 - Relevant targets based on strategic and action plans,

- Implementation strategy, key performance indicators and minimum data set

National targets for all indicators, with provincial adaptations

Not yet achieved. Targets are currently used in the

Provincial and District Three-

Year Operational Plans, but not yet integrated in the new software application and/or paper tools.

1.4 Revise data collection, collation and analysis

tools and setting up quality assurance procedures

(Not provided) Revision of collection and collation tools achieved.

Some analysis tools introduced. Manual of Data

Quality Assurance elaborated, but staff not yet trained and procedures not yet implemented.

1.5 Assess needs and plan for

unfinished subsystems planned in

1996:

Community based HMIS

Hospitals

Administrative systems – finances, assets, and logistics

Human Resources (This sub-system was not considered in PE2 list)

Assessment reports and plan of action for four key subsystems written and piloted:

Community based

HMIS

Hospitals

Administrative systems

– finances, assets, and logistics

Human Resources

Assessment reports and plan of action performed at beginning 2007, covering:

Community

Hospitals

FAMS

Human resources system not yet done and dropped in the

PE2 list

1.6 Create organisational set up to

ensure regular review of HMIS as part of the overall M&E framework

1.6 Quarterly minutes of

meetings of National Steering

Committee.

Minutes available and NSC meetings held regularly.

Table 2.2.2: PE 1 & 2 – Achieved Results.

According to the interviews and after a short discussion, two interrelated factors emerged as important obstacles to a more efficient and timelier project implementation:

1. Substantial changes in the organisational environment and fragmentation of M&E responsibility within the MOH:

The 2005 National Health Services Act After repealed the previous 1995 Act and abolished the

CBoH. Their functions have been and are still integrated in the newly defined MoH. The organisational structure, functions and organogram have gone through an impressive process of re-definitions of new job descriptions and specifications for all posts. The Human Resource Management is still undertaking a massive restructuring of the entire MOH staff at all levels under the Public Service Reform Programme. This process introduces re-adjustments of managerial functions. The process is dragging on and the delay increase uncertainty and anxiety among the officers in charge, even the most senior who are afraid to see their posts put on public competition. The recent Mid-Term Review (MTR) of the National Health Sector Plan

(NHSP) 2006-2010, confirmed that this plan has been a crucial instrument to anchor and orient the personnel in this transitional phase, allowing them to score some significant achievements in the service delivery, e.g. mother, infant, child mortality improvements, and in the area of the support services too. For instance, the system extended the provision of drugs and medical stocks and the health spending per capita (up to $US23 in 2007). However, there are also the flaws: the process of decentralisation under the National Decentralisation Policy seems to proceed slowly and the abolition of the Health Boards “appears to have led …to a re-centralisation of decision-taking” ; and the SWAP instruments are all in place, but the harmonisation and alignment is not progressing well, “in 2007, only 20% of all financial resources to the health sector were ‘on budget’”, out of the total US$246.5 million.

The Ministry is subdivided in 5 Directorates. In the Directorate of Policy and Planning, the Sub-

Directorate of M&E is the main depositary of the M&E activities, supervises the HMIS and its development and the regular review of the National Set of Indicators. However, in the same Directorate, the Sub-Directorate of Planning oversees the annual planning at all levels, the Joint

Annual Review (JAR) and prepares the bi-annual meetings of the Sector Advisory Group

(SAG). Furthermore, the Directorate of the Technical Service Support being in charge of the coordination and networking with the Provinces, PHMT and DHMT, is responsible of the

Performance Assessment (PA) at all levels and keeps a separate and additional set of indicators.

Separate set of indicators are kept in force under the vertical programmes, Directorate of Public Health &Research and in the Directorate of Clinical Care for Drugs, procurement and logistic.

Graph 2.2.1: MOH Simplified Organogram 2009

3 Expected development capability of the MoH Unit not realistic

The project designers have not been realistic on the capability of the implementers to carry out the activities, when considering the project to be embedded in the ministry with permanent staff in charge of the implementation. This staff is usually assigned by their superiors to work on routine duties rather than on development activities for strengthening of the HMIS. The project designers did not pay enough attention to the social nature of the changes at stakes. The transition to a new information system implies by its nature changes of functions, institutional arrangements and power distribution and merely technical innovations. HMIS Technical Committee was not as efficient as expected, which had a negative impact on the implementation on the proposed review process at design stage. The M&E Sub-Committee has taken the full responsibility of running the project during the second PE and the head of the HMIS Unit is now at the level of Deputy Director. However, the problem of the leadership during the long

7 Zambia NHSP IV, 2006-2010, Medium Term Review, Final Report, 17/11/08, p. XXI.

Zambia MOH

Simplified organogram, 2009

Minister

Directorate

Policy &Planning

Directorate

Public Health

& Research

Directorate

Clinical Care

Permanent

Secretary

Directorate

Technical Support

Services

Directorate

Human Resources

Sub-Directorate

Budgeting

Sub-Directorate

Planning

Sub-Directorate

M & E

HMIS

(National Set

Indicators)

M&E

All Plans

Joint Annual Review

SAG (2x)

M&E

Performance

Assessment (PA)

HPMF

M&E

Vertical Programmes

(AIDS, TB, Malaria)

M&E

Drugs/Medical/

Logistics process of negotiations was aggravated by the absence of a permanent national advisor or external TA to be assigned to the project duties. Some delay occurred also during the contractual process of external experts.

2.2.2 How well has been conducted the revision of the HMIS?

The table above presents a summary of achieved results against expectations. However the results are the final matching of combined inputs, process and outputs planned by the EU/PRBS/HMIS Project and of the intended or unintended effects of each intervention. A deeper evaluation of the new HMIS performance is beyond the current end-project review mandate.

Examining the outputs 1.1, 1.2 and 1.3 we try to assess some features of the new HMIS, namely is the HMIS:

a. Responsive to users, providing the necessary national agreed set of indicators?

b. Flexible, in order to consistently updating the data elements and indicators requested by the programme and planning?

c. Simple and user-friendly in terms of paper and electronic tools?

Meanwhile that examining the outputs 1.5 we try to respond to analyse some other features, is the new HMIS:

d. Comprehensive and complete, covering all data sources?

e. Well integrating, processing and producing data and indicators, extracted from the main databases in existence?

The summary statement is as follow:

The new HMIS has interesting potentials: When completely developed the new HMIS will be an excellent tool to provide the managers with flexible, responsive and integrated routine data that contribute to the assessment of the performance of the health sector, to the planning cycle and monitoring and evaluation of the objectives, and quick action and response. However, both the paper tools in the health centres and hospitals and the electronic applications are undermined by several bottlenecks, are all, but not completed and fully functional.

1.1. Identify performance indicators

1.2 A minimum data set with clear definitions and a dictionary produced

1.3 Relevant targets based on strategic and action plans developed.

The 2005 evaluation report noted that “In 1996-1997, the old Zambian HMIS was considered functional at all levels of the health system and compares favourably with most African HMISs.

There was a defined indicator set and regularly collected data elements. Data collection and reporting tools are available at all facilities and district offices and the flow of information are clearly set out using the “single pipeline” principle. There was regular quarterly analysis of routine data with reasonably good coverage for service-based indicators of underweight children, measles immunization, supervised deliveries, malaria and TB incidence and TB DOTS coverage”. At that time 74 indicators were calculated regularly

The identification of new indicators was done. An impressive list of 220 indicators is available. The set has been agreed upon during the National Consensus Workshop, which represented quite an achievement. The consultative process has been conducted according to the

8 Establishment of an information culture: achievements and challenges in Zambia, First RHINO

Workshop, Bolger, Potomac, MD, USA, March 14-16 2001, p. 228;

9 HMIS Procedures Manual (Primary Health Care), version 1.4, Ministry of Health, Directorate of

Planning and Development/Monitoring and Evaluation, December 2008, p. 17-19 and Annexure 3.1;

staff involved and interviewed in a participatory way that created trust and collaboration from the different programme managers and aid partner representatives. The provincial training has been used for discussing and adjusting the list all along the period of the country roll-out. The additional indicators are calculated, analysed and reported on monthly basis. They cover important strategic areas such as:

MDG indicators, extending from the previous 6 to all 15 health-related;

Government PRSP key indicators for making available the results monitoring;

MOH NHSP 2006-2010 key performance indicators;

Mother and Child health additional indicators;

AIDS, TB and Malaria key additional indicators.

Gender and age further disaggregation possible.

The data elements populating the paper tools and the DHIS1.4 have been defined and described in the Procedures Manual, Annexure 3.1 and 3.2. This is an initial Data Dictionary, but it does not include until now the definition of other indicators and statistical terms in use in the health sector and other databases and the definition of the metadata. Coordination in this sense with the CSO is underway, but it has not yet produced an agreed and shared product such as the

National Metadata Dictionary.

The targets of the NHSP and the Provincial Operational Plans are not integrated in the paper and electronic DHIS1.4 application.