Attachment_9_-_PBG_Final_Performance_Evaluation_Report.pdf

August 8, 2014

This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Business and Technical Consultants, Inc.

A public notice board describing authorized taxes and their amounts in Bandundu

FINAL PERFORMANCE EVALUATION

Programme de Bonne Gouvernance (PBG) in the Democratic Republic of Congo (DRC)

PROGRAMME DE BONNE

GOUVERNANCE IN THE

DEMOCRATIC REPUBLIC

OF CONGO

FINAL PERFORMANCE EVALUATION

Final Report, August 8, 2014

IQC AID-RAN-I-00-09-00016, Task Order Number AID-660-TO-14-00001

DISCLAIMER

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

CONTENTS

Acronyms ................................................................................................................................................................................................... i

Executive Summary................................................................................................................................................................................ iii Evaluation Objectives and Questions .......................................................................................................................................... iii Methodology ...................................................................................................................................................................................... iii Main Findings and Conclusions ...................................................................................................................................................... iv

Recommendations ............................................................................................................................................................................. v Introduction Background

Background on Decentralization Profiles of Donor Support to Decentralization Profile of PBG Activities and Partnerships

Evaluation Purpose Evaluation Methods

Work Plan

Data Gathering Sites Key Informant Interviews Focus Group Data Collection Method Method for Inclusive Data Analysis

Comparison Site Limitations and Challenges

Findings

1. Institutional Capacity and Strength

1.1 To What Extent Are Project ETDs Better Resourced?

1.2 What are Effective PBG Revenue Collection Methods?

1.3 Public Private Partnerships in Katanga

1.4 Did Project Contribute to Transparency, Openness and Accountability?

1.5 Capacity of Parliaments

2. Citizen Demand for Accountable Governance

2.1 Increased Demand

2.2 Has Increased Demand Affected Government Behavior?

3. Service delivery of GDRC Institutions

3.1 Have Government Services in Technical Sectors Changed?

4. Implementation and Approach

4.1 Approaches to Harnessing Political Will

4.2 To What Extent are Results Sustainable?

4.2.1 Effective Implementation Strategies

4.3 Integrating Governance with Technical Sectors

5. Cross-Cutting Questions

5.1 Gender Programming

5.2 Impact on Urban and Rural Authorities Different?

6. Other Findings

6.1 Project Design

Conclusions Recommendations

Lessons learned

ANNEX I: PROFILE OF DONOR SUPPORT TO DECENTRALIZATION

ANNEX II: PBG PARTNER AND ACTIVITY PROFILES

ANNEX III: DATA ANALYSIS GRID BY EVALUATION QUESTION

ANNEX IV: RATING SCALE SCORES

ANNEX V: DATA COLLECTION INSTRUMENTS

ANNEX VI: SOURCES OF INFORMATION

ANNEX VII: DISCLOSURE OF ANY CONFLICTS OF INTEREST

ANNEX VIII: EVALUATION QUESTIONS AND SUB-QUESTIONS

ANNEX IX: EVALUATION STATEMENT OF WORK

ANNEX X: TABLE OF STATISTICAL ANALYSIS OF RATING SCORES

TABLE OF TABLES

TABLE 1: PROJECT EXPENDITURE CATEGORIES

TABLE 2: DATA GATHERING SITES

TABLE 3: PROJECT ETD REVENUE GROWTH 2008-2013

TABLE 4: % OF PEOPLE SURVEYED IN PROJECT ETDS WHO BELIEVE THEY HAVE

OPPORTUNITIES TO EXPRESS OPINIONS

TABLE 5: PBG PARTNER BENEFICIARIES PROFILE

TABLE 6: TRAINING WORKSHOP MODULES

TABLE 7: ANALYSIS GRID

TABLE 8: NUMBER OF LAWS/EDICTS DRAFTED AND APPROVED

TABLE 9: AVERAGE RATINGS BY RESPONDENT CATEGORY

ACRONYMS

AC Animateurs Communautaires AFEJUCO Association Des Femmes Juristes Congolaises (Sud Kivu) AFEM Femmes Des Medias Du Sud-Kivu (Sud Kivu) ANAPECO Association Nationale Des Parents D’élèves Du Congo (Katanga) APRODEF Association pour la promotion et défense des enfants et des femmes (Maniema) ASSOPAC Association Pour La Promotion De L’agriculture Au Congo (Bandundu) AVEMA Action des volontaires pour la promotion de l'environnement au Maniema BDD Bureau Diocésain De Développement (Katanga) CAS Country Assistance Strategy CBS Capacity Building Specialists CEDAC Centre D’étude De Documentation Et Animation Civique (Sud Kivu) CENADEP Centre National d’Appui au Développement à la Participation Populaire CJP Commission Justice Et Paix (Katanga) CLAC Comite Lutte Anti-Corruption CNONGD Conseil National des ONG de Développement CO Centre Olame (Sud Kivu).

COFERD Collectif Des Femmes Rurales Pour Le Développement (Bandundu) COJESKI Collectif des organisations des jeunes du Congo Kinshasa (Maniema) COREF Comité d’Orientation de la Réforme des Finances Publiques (Ministry of Finance) CRD Democracy Resource Center CRONGD Conseil Régional des ONG de Développement CSO Civil Society Organization CTAD Cellule Technique d’Appui à la Décentralisation DAI Development Alternatives Inc.

DPA Dialogues Publiques DRC Democratic Republic of Congo ECOFIN Economic and Finance Committee, National Assembly ETD Decentralized Territorial Entity (Entités territoriales décentralisées) FG Focus Group GDRC Government of the Democratic Republic of Congo HBM Haki Za Binadamu (Maniema) HQ Headquarters HR Human Resources IBTCI International Business and Technical Consultants Inc.

IP Implementing Partner IR Intermediate Result KII Key Informant Interview LMDP Le Miroir Du Peuple M&E Monitoring and Evaluation OLCAC Observatoire de Lutte Contre la Corruption en Afrique Centrale NA National Assembly of Democratic Republic of Congo PBG Programme de Bonne Gouvernance PAJ Administration and Justice Committee, National Assembly PI Performance Indicator PAP Plan d’Actions Prioritaires (Finance Sector Action Plan) PMP Performance Monitoring Plan PRCG Projet de Renforcement des Capacités en Gouvernance (World Bank) i

PROSADEF Promotion De Droit De Sante De L’enfant Et De La Femme (Bandundu) RBM Results Based Management RECIC Réseau De L’éducation Civique Au Congo (Bandundu) RIO Réseau D’innovation Organisationnelle (Sud Kivu) SI Social Impact SERACOB Service des Renforcement des Appuis aux Communautés de Base en Afrique Centrale SFCG Search for Common Ground SOCIKAT Société Civile Du Katanga SOW Statement of Work SWAP Sector-Wide Approach TEYESA SURVIE Teyesa Survie Telema Ya Sala-Survie (Bandundu) TOT Training of Trainers UMAMA Umoja Wa Mama Wa Maendeleo (Maniema) USAID United States. Agency for International Development USG United States Government ii

EXECUTIVE SUMMARY

The Programme de Bonne Gouvernance (PBG) is a five-year project (2009-2014) implemented by DAI. At the outset of the program, DAI had five sub-contractors, 50 partners in four provinces (Bandundu, Katanga, Maniema and Sud Kivu), and a budget of $36.5 million. PBG focused on three tracks – civil society, parliaments and local government – that correspond to the three Intermediate Results (IRs): citizens demand accountability (IR1); selected parliaments are more democratic and effective (IR2); and laws, policies and procedures for decentralization are established and implemented (IR3). By midcourse in the project, PBG concentrated on support to revenue management by 12 Decentralized Territorial Entities (ETDs) with citizen participation. The overall evaluation conclusion is that the project was very well-managed despite a highly challenging context and vast project scope. It was also concluded that working with local government on a pilot basis has tentatively confirmed the fundamental principle of citizen participation for effective decentralization.

Evaluation Objectives and Questions

As specified in the scope of work, the main evaluation objectives are to determine answers to the following two questions:

1) What is the performance of the program in strengthening institutions and increasing citizen demand?

2) To what extent is the project theory of change valid that strengthened institutions and increased demand result in better service delivery?

The PBG performance evaluation seeks to answer the following evaluation questions:

1) Do institutions have increased capacity and strength?

2) Has there been an increased citizen demand for accountable governance?

3) Were there improvements in delivery of public services in the target local governments?

4) What lessons have been learned regarding implementation strategies?

5) What are findings related to cross-cutting themes of gender and urban-rural impact? Some key evaluation questions address the effectiveness of revenue generation among the ETDs or Decentralized Territorial Entity (Entités territoriales décentralisées), and the most effective methods to achieve these results.

Methodology

A work plan was developed based on the proposed evaluation design of International Business and Technical Consultants Inc. (IBCTI), in coordination with representatives from the USAID Mission based in Kinshasa. The evaluation team comprised four Congolese experts and one international senior subject matter expert. Site selection to collect data included three intervention provinces (Sud Kivu, Katanga, and Bandundu); visits to all nine pilot local government sites in the three provinces visited; and one non-project local government site for comparison purposes. The fourth province of the program, Maniema, was excluded in the evaluation scope of work due to the difficulty of traveling to Maniema, given the limited time allocated for the fieldwork. The evaluation team utilized the following data collection methods:

Annual and quarterly reports, special project reports and PBG internal reports from the four PBG provincial offices were reviewed.

Key Informant Interviews (KIIs): Interviews were conducted with representatives from parliaments iii

(Members of Parliament, senior administrators, heads of committees, and research bureau staff) ETD officials (mayors, finance administrators and senior staff), and representatives from civil society organizations (CSOs). Approximately 137 interviews were conducted.

Focus Group Discussions (FGDs): In each ETD, the evaluation team organized three groups with 10 participants comprising women, youth and professionals. In total there were 30 FGDs with 300 focus group participants (more than 100 females).

Questionnaire: The evaluation team developed a questionnaire to elicit opinion ratings on topics pertaining to the evaluation questions, which was administered to 473 persons (202 of the 473 were female respondents). This questionnaire was developed to gather and triangulate information rather than to derive statistically significant results.

Ad hoc meetings: The evaluation team also met with individuals who may not have been originally selected for an interview but who the evaluation learned would be able to provide valuable information.

The data analysis process was conducted as follows:

The five-member evaluation team met in-person on five different occasions to compare and analyze notes from KIIs, FGDs and observations during and after field work.

The data collected from the interviews and FGDs were analyzed as per each of the 13 evaluation questions. This information is compiled in an evaluation grid (Annex III).

The responses to the opinion rating questionnaire were analyzed using excel and SPSS.

Main Findings and Conclusions

Despite facing difficult objectives, PBG was a successfully managed project. The PBG reports are comprehensive and all of the 4,000 activities delivered can be identified. Its quantitative performance indicators are numerous; these numerical indicators report and document the high activity and output levels of the project. They do not inform about outcomes. PBG made several creative and original attempts to measure outcomes by supplementing these (largely USG-designated) numerical indicators with qualitative (or opinion-based) indicators to measure baselines and changes in institutions and to assess public opinions. This was an important effort in performance measurement, since numerical indicators (for example, the number of persons trained) say little about attitudinal and behavioral changes needed for democratic governance outcomes.

The most successful activities were training and technical assistance in ETDs, both with civil society and ETD administrations. Through in-depth interviews and focus groups, the evaluation found evidence of behavior changes on the part of local government: greater transparency, more citizen participation, more outreach to citizens, a budget presented on an official city website, and formal receipts for taxes paid.

The most successful method was citizen and CSO inclusion through formal committee structures that allowed for interaction between stakeholders and consensus building around common issues and solutions.

The evaluation found convincing evidence that the theory linking citizen demand to better service delivery has merit. It may be more accurate to use “collaboration” as opposed to "demand," at least regarding the findings of this study. Demand as a stand-alone concept may not create change in this context. Change driven by citizen demand implies having elected representatives who depend on citizen confidence to be re-elected. ETD officials, however, are not elected; rather, they are appointed by superiors in their city, their province, or by political leaders in Kinshasa; their allegiance to those who appoint them takes precedence, a fact admitted to the team during interviews with officials. Nevertheless, and despite this known principle of democratic accountability, the study found evidence among ETD leaders that the opinions of citizens also matter, ever increasingly, once formal mechanisms for public dialogue and iv consultation (as tested in the project) become habitual. Further support for this theory of decentralization is the finding that some ETD leaders are making visible investments in city infrastructure using the increased tax revenues, and in some cases are reaching out to citizens for their feedback.

The project was handicapped at the outset by a vast scope. Specifically, the designated institutional targets and a rigid work planning structure caused these roadblocks. By project midcourse the thrust changed considerably, allowing greater flexibility to focus on ETD partnerships having greater potential for results.

A reader familiar only with project reports could miss this fact, considering the contract was never revised.

DAI continued to seek activities, deliver outputs and report based on the original scope. A number of activities were reduced or stopped altogether, for example, training activities with provincial governments, while efforts with parliaments continued – although there were clear reasons to discontinue (especially the National Assembly) and to re-allocate resources. This was an effective way to manage the project in the DRC context for both USAID and DAI. .

A significant conclusion arising from this evaluation is the importance of political will (motivation) as a necessary pre-condition to project investments, including the will to participate in activities (despite lower USG per diems); to allocate public resources that support the project goals; and to take decisions that implement reforms. The successful ETD tax revenue component had all three aspects. When partners, be they individuals or institutions, are not committed or interested, the donor's investment risks are much higher and success less likely. When partners want support and are committed to results, then technical support, both skills and material resources, are value added.

The evaluation also argues that the project scope, good governance, was far too broad. Had the project been “Support to Decentralization,” it might have focused activities sooner on relevant activities and partners and been able to fully test participatory budgeting models. Rebuilding classrooms in Sud Kivu, distributing agricultural kits for farmers in Katanga, or providing IT support to the National Assembly will have contributed to alleviating needs but did not advance decentralization. The ETD support activities were directly supportive of financial decentralization as it is evolving in practice; here, the project achieved results well beyond the interesting but inconclusive outputs in all the other project themes.

Recommendations

1. Replicate Successful Local Government Financial Decentralization Component.

USAID should continue to build on PBG's most successful component: support to ETD revenue generation with citizen participation. The component should be defined as support to revenue management (taxes and expenditures) for local government. The activity or project component should be replicated in additional ETDs and continued in the existing pilots.

This is important because according to evidence found in this study, PBG has been successful (albeit through a pilot activity in one percent of ETDs) in demonstrating that participatory local government can potentially work in the DRC. Results are preliminary, because the participatory planning-budgeting process has not even completed one full administrative cycle. Abandoning it now would leave unanswered questions about effective citizen participation in local government decentralization. The evaluation also found some evidence of imitation by other ETDs, which is a potentially good indicator of sustainability and possibly an indicator of sincere interest for change among other ETD leaders.

Partnerships with CSOs should be expanded to include larger NGOs eligible for cash grants.

CSOs played an important role in this experiment and should be further supported as a mechanism for v citizen participation. Cash grants, while difficult to implement with small CSOs, offer more opportunities for local ownership.

Support to Central Agencies. The evaluation recommends that present support to central agencies should be solely to advance and facilitate ETD decentralization (COREF, CTAD, Public Service, National Assembly, etc.).

2. Incorporate Technical Sectors into Project using Citizen Participation Strategy and

Existing Participatory Committees.

It may be possible to integrate the successful PBG process through participatory committees and pressure from the grass-roots to work both locally and vertically in the technical sectors. The same principle used by the project, to work where there is already a political will to practice decentralization, may be tested again by supporting a combination of province-capital city-ETD and commune in one common framework.

The purpose is to build will and consensus to reform from the bottom up, with a common focus by different actors on the same issues and objectives.

There are existing high-level and local dialogue committees which may be an entry point for programming.

Some examples of these committees are provided in this report.

Whether this approach will broaden the reach of a future project to the technical sectors is not known.

PBG was not successful providing training to sector ministries at the provincial level, due to weak political will and lack of interest.

3. Donor Collaboration. USAID should seek joint activities and agreements with donors having similar objectives.

This recommendation is relevant to both project level activities and dialogue at the central level with government on reforms. At the project level, PBG was actively engaged with UNDP; there may be means to increase that collaboration at technical and funding levels.

4. Capacity building without political will should be avoided. Political will should be a guiding factor used by USAID in the selection of partnerships and targets.

Technical inputs such as training were less effective with partners where political will was weak. In other words, technical assistance and other technical outputs and deliverables – such as roadmaps, etc. – have little impact on political will. The USAID evaluation Statement of Work (SOW) has defined an effective three-dimensioned indicator to assess political will.

The PBG project provided technical inputs to different institutions. Those that were successful, such as support to ETDs with citizen participation, were due to existing strong political will in ETD administrations, specifically related to tax revenues.

5. Flexible Design-Targeted Scope. Future projects in this sector should include, to the extent possible in the USAID programming context, a flexible design but tightly targeted scope. Results (outcomes) should be specifically defined, but partnerships and activities should be sufficiently open to allow an implementing partner to adjust its activities and partnerships depending on political will and potential for achieving results. The scope of this good governance support project was overly broad, a problem contributing to issues with partnerships and achievements in the first two years, so it lost the opportunity to fully test participatory local governance.

vi

INTRODUCTION

The Programme de Bonne Gouvernance (PBG) is a USAID five-year project (2009-2014) implemented by DAI with a budget of $36,252,000. The present study is the final performance evaluation of PBG. It is an independent evaluation, conducted by International Business and Technical Consultants, Inc. (IBTCI), and includes a scope of work (Annex VII) encompassing 13 evaluation questions and two objectives.

PBG had a vast scope of work that included 50 partners, three pillars (or intermediate results), four provinces (Bandundu, Katanga, Maniema and Sud Kivu), five parliaments, 12 local governments, 30 civil society partners and about 4000 training and technical assistance activities. Its primary thrust was support to administrative decentralization, but a number of other governance-related activities were supported.

To undertake this complex project, PBG’s work plans have 14 sub-objectives, more than 100 lower-level objectives, and originally had 108 performance indicators -- mainly output indicators (later reduced to about 30). Annex I gives a full profile of all partners and activities.

PBG began operations in September 2009, submitting a five-year detailed work plan as required within a few months of startup. The original and complex work plan structure remained largely intact throughout the life of the project, although, as will be discussed in this study, significant and beneficial revisions to institutional and other targets took place about mid-way through the project. PBG management requested a mid-term evaluation so as to revise the scope, based on lessons in the first two project years, but due to funding limitations the evaluation did not take place.

The work plan was developed around the three pillars or intermediate results (IRs). As presented in the evaluation Statement of Work, they were:

IR1: Citizens demand accountability: Increased demand for accountable governance among citizens and civic organizations.

IR2: Select legislative bodies are more democratic and effective: Improved capacity of provincial and national legislatures to conduct legislative actions and oversee executive activities.

IR3: Laws, policies, and procedures for decentralization are established and implemented:

Increased citizen support for decentralization and improved local government capacity to deliver services.

These three pillars were distinct, each with its own objectives, managers, partners, indicators, outputs and reports. PBG had a broad, if not vague, project purpose: "to improve management capacity and accountability of local governments and parliamentary assemblies in four provinces and in the National Assembly, and to increase citizen demand for accountable local governance."

In retrospect, the project's name, "Good Governance," was a misnomer. Support to good governance in the Democratic Republic of the Congo (DRC) is obviously an essential ingredient for evolution away from a failed and failing state. However, USAID may have had a more specific and focused objective in mind when it conceived the project early in 2008: support to administrative decentralization, as defined in the 2006 Constitution, adopted following a national referendum. The Constitution, with its clear rules and formula for decentralization, contained a message of hope for lasting reform that would solve conflict and stimulate development. "Good Governance," on the other hand, suggested a much wider scope of activities and partnerships, many of which were marginally relevant to decentralization.

Based on how the project began, the project story could have ended with many activities and outputs but few results. Fortunately, PBG changed its focus (with full USAID support), a fact not well reflected in project reports. From 2012, the prime thrust became assisting the 12 local government partners (entités territoriales décentralisées or ETD) to increase their tax revenues, with local civil society playing a pivotal and participatory role. Thus, two of the three pillars – IR1 (civil society) and IR3 (ETDs) – merged under this common, and as will be reported here, successful objective.

BACKGROUND

Background on Decentralization The 2006 Constitution presented a detailed framework for administering decentralization as follows:

Congo is divided in 11 provinces (to be expanded to 26 within three years); each province has exclusive competence in some fields of public policy (mainly education, health, agriculture and rural development); provincial governors are to be elected by provincial assemblies elected by universal suffrage; the central government must redistribute 40% of domestic national revenue to the provinces, 40% of which the latter must in turn redistribute to decentralized territorial entities (ETDs) such as towns, communes, sectors and chiefdoms; and provinces are allowed to legislate in the areas of their competence, including for the generation of revenue.1

Implementation requires a platform of laws, regulations, procedures, administrative reorganization and institutional capacity to implement reforms. For example, while the health sector is presently decentralized in principle, all its personnel are appointed and its resources controlled by central government Ministry of Health. During the evaluation, the Subject Matter Expert asked six respondents, all experts in the decentralization process in the DRC, for their opinions on the status of decentralization in 2014 (where full implementation equals 100%). Replies range from 10% to 40% complete. They agree that the most important next step is local elections to begin to formalize public accountability to citizens. In view of the fact that seven years have passed, this progress is not impressive. Respondents agree that the main issue is lack of political will at high levels rather than legal, technical or institutional issues, although legal and administrative issues in this vast country are also imposing problems.

Decentralization, when accompanied by reforms of democratization, promotes good governance, greater accountability, better representation, stability and development.2 Distribution of power means that decisions are taken closer to the beneficiaries who depend on those decisions, and that citizens can become partners of local government. Specific needs of citizens at the local level are more likely to be addressed and public services to better respond to their needs. For the DRC, these public services include health, education, agriculture and public infrastructure. At the same time, decentralization reduces political power at the center.

Within the Government of the Democratic Republic of Congo (GDRC), there are three departments vying for leadership of decentralization: Cellule Technique d’Appui à la Décentralisation (CTAD), the Ministry of Public Service, and the Ministry of Finance through COREF. After the Ministry of Decentralization became a department of the Ministry of the Interior, the influence of CTAD was reported to be significantly reduced. The Ministry of the Interior is responsible for a number of commanding issues, security and policing being the most significant along with decentralization and customary practices (Chiefdoms). The Ministry of Public Service is an important actor relating to human resources and administrative reforms required by decentralization. COREF (with the World Bank) has taken the lead in

1 Unpublished article "One Thing Led to Another…Donors, Decentralization and the Consequences of Partial Reform in the DR Congo" Pierre Englebert, Pomona College and Emmanuel Kasongo Mungongo, University of Kinshasa, February 2014.

2 Carothers, Thomas & Diane De Gramont. 2013. Development Aid Confronts Politics: The Almost Revolution. Washington, DC: Carnegie Endowment.

coordination of all public financial management reforms, including financial decentralization, also a vital aspect of reform.

Profiles of Donor Support to Decentralization The main donors supporting decentralization in the DRC are the World Bank, UNDP, DFID, the European Union, and the World Bank Institute (a participatory budget project in Sud Kivu was completed in 2012).

In addition to PBG, the USAID-funded BRIDGE project also supported decentralization. A profile of donor support to decentralization is found in Annex I.

DFID Governance Strategy DFID is one of four to five donors actively supporting local communities. Its £106 million (pound sterling) community development project, Tuungane, was an important pilot similar to PBG and was evaluated using social science methods through a US university at a cost of $1 million. It has also conducted a recent internal assessment of all its governance activities, all told providing significant lessons relating to USAID in the DRC. Subsequent to the assessment of its governance activities in parliament, civil society and public finance, DFID decided to stop all capacity building activities as currently configured, with some exceptions in public finance. It concluded that capacity building at the technical level was not achieving results, due to lack of high-level political will for reform. This is in effect the exact approach of PBG, capacity building at the technical level. DFID distinguishes accountability for results from accountability for outputs; since accountability for results is its primary mandate, it is reorienting towards a programming model combining political factors, creating new coalitions of stakeholders, and issue-based and opportunity-based flexible programming "from the bottom up." When an activity is not working it will be stopped. Citizen participation and oversight will now be integrated into other activities. This approach is similar to the USAID Fragile States Strategy of January 2005, having similar language and concepts.

In public finance, DFID is the main funder of the Multi-Donor Trust Fund, essentially a Sector-Wide Approach (SWAP), managed by the World Bank (Project pour les finances transparentes au Congo -DFID 21 M € WB 5 M. €, Belgium and Canada). Through COREF, a high level committee in the Ministry of Finance, an Action Plan for all finance reform has been approved. It has six pillars for investment: budget reform;

fiscal reforms (taxation/customs reform); expenditures management and accounting; oversight (including Auditor General); and financial decentralization.

IRs and Project Organization The project was organized and staffed under three distinct pillars (or IRs) – citizens demand accountability (IR1), selected parliaments are more democratic and effective (IR2) and laws, policies and procedures for decentralization are established and implemented (IR3). Increased citizen and civic group participation, essential to democratic governance, has been particularly fragile in the DRC. Citizen participation in local government planning and budgeting helps identify and address their needs, engage them in their roles as citizens, such as paying taxes, and gives them a means to advocate for gender budgeting. Citizen participation contributes to effective parliaments because parliamentary outreach to citizens and in-reach from citizens strengthens all three parliamentary functions of policy making, oversight and representation.

From 2012, PBG workshops generally combined CSO-ETD or CSO-parliament participants. This merging had another benefit, increased contact and interaction between the actor and more networking and relationships built, increasing trust. Table 1 presents an overview of PBG expenditure categories. Not surprisingly, the major category is labor – for project staff, trainers and technical assistance and for both international and local consultants. Much project staff time was consumed by the labor-intensive in-kind grant management process.

Table 1: Project Expenditure Categories Expenditure Category USD (million) %

Labor3 14.7 40.2 Workshops 2.8 7.6 In-Kind Grants and Assistance .790 2.1 Sub-Contractors 3.0 8.2 Travel 2.0 5.5 Other Direct Costs (office, vehicles, etc.,) 11.3 30.9 Allowances 2.0 5.5

TOTAL 36.5 100

Profile of PBG Activities and Partnerships The number of persons trained exceeds 12,000 (38% women). About 10,000 persons were involved in public dialogues and some 600 in participatory budget committees, while 6,000 received financial and management training related to decentralization and local government. National Assembly MPs and staff trained over 800 and provincial assembly MPs and staff some 4,800. Some 500 CSO personnel were trained in budget analysis. These large numbers tell of significant activity levels, but to a reader unfamiliar with the project, they do not paint a clear portrait of this complex and evolving project. The present section and the tables in Annex II give summary profiles of activities, beneficiaries and partners.

Most project activities can be summarily described as capacity building, i.e., training and technical assistance, however, unlike most capacity building projects, PBG gave only minimal equipment or materials support to institutions. About 60 in-kind grants (no money changed hands) were provided on a small scale (average under $10,000 per grant), primarily to CSOs, (about 15 were provided to local ETD governments) and generally with the goal of increased citizen participation in local government or policy advocacy. This in-kind support involved considerable project personnel involvement in logistics, procurement, organizing events, supporting grantees for reporting, etc. The project had a dozen standard training modules from which it selected the training events. Other activities included study tours and exchanges, creating local committees (planning and budget), roundtables, conferences, and short-term technical assistance, the latter often as follow up hands-on training to workshops. After 2012, training activities with provincial ministries and provincial sector ministries were stopped. Activities with the National Assembly were never frequent. From 2012, the focus of PBG was on supporting ETDs to increase tax revenues, with CSO participation. Involvement with CTAD during the project was mainly to support advancing the stalled decentralization process.

A full profile of partners and activities is found in Annex II.

EVALUATION PURPOSE

As per the scope of work, (Annex VII), the two main evaluation objectives are:

1) What is the performance of the program in strengthening institutions and increasing citizen demand?

2) To what extent is the project theory of change valid, that strengthened institutions and increased demand result in better service delivery? USAID intends to use evaluation findings to contribute to a new program.

The PBG performance evaluation seeks to answer the following evaluation questions:

3 Labor includes $933,000 of local consultant fees – 51 consultants/3959 days (May 2014)

1) Do institutions have increased capacity and strength?

2) Has there been an increased citizen demand for accountable governance?

3) Were there improvements in delivery of public services in the target local governments?

4) What lessons have been learned regarding implementation strategies?

5) What are findings related to cross-cutting themes of gender and urban-rural impact? Some key evaluation questions address effectiveness of revenue generation among the ETDs, and the most effective methods to achieve these results.

The evaluators were asked to assess: "The theory of change that strengthened institutions and increased demand result in better service delivery." This is a theory of decentralization. To decentralize means: to distribute the administrative powers or functions of (a central authority) over a less concentrated area: to decentralize the national government. 4

EVALUATION METHODS

Work Plan The IBTCI evaluation team drafted and submitted a work plan prior to beginning the fieldwork. The work plan was drafted based on the proposed evaluation design submitted by IBTCI and on an analysis of project materials provided by USAID. The work plan was discussed with USAID and representatives from CTAD and The Office of the Prime Minister during two meetings after arrival in Kinshasa, then revised prior to provincial field work. The work plan defined data collection in three provinces, Bandundu, Katanga and Sud Kivu, and in Kinshasa. Nine project sites and one comparison site that did not benefit from the program were selected for data gathering, including two rural sites. The evaluation questions presented in the work plan were discussed during meetings at USAID, and the rating scale questionnaire was revised.

All of the questions defined in the evaluation work plan were translated into French and the methodology was discussed by the two teams, along with briefings on the project and development of procedures for focus groups.

Data Gathering Sites A number of factors were considered in the selection process for the evaluation sites. Out of the four provinces that benefitted from the program, Maniema was not included in the scope of work due to the difficulty of traveling there within the limited timeframe available for the fieldwork. As Table 2 illustrates, all six intervention sites in Sud Kivu and Bandundu were included in the evaluation. In Katanga, all three ETD project sites were visited, along with one site where the intervention had not taken place (the commune of Kenya in Lubumbashi city). Kenya was chosen to be the one comparison site where no PBG project activities took place. Among these 10 sites, the evaluation team collected data in three cities, five communes (neighborhoods within a city) and two rural sites. For the latter, one site was a chefferie (the chief is a member of a ruling family), the other chief is appointed by the province, referred to as secteur.

4 http://dictionary.reference.com/browse/decentralization

Table 2: Data Gathering Sites Province Urban Site

(City) Urban Site

(Commune in City) Rural Site Control Site

(Comparison) Katanga Kolwezi, Likasi Katuba (Lubumbashi city) Kenya (Lubumbashi city) Bandundu Lukolela (Kikwit)

Mayoyo (Bandundu city) Bukangalonzo (Kenge)

Sud Kivu Bukavu

Kadutu (Bukavu city)

Chiefdom Ngweshe/Walungu

TOTAL 3 4 2 1

Key Informant Interviews Interviewees were selected from among ETD personnel responsible for finance and administration divisions who had also been beneficiaries. CSO interviews were with persons who had been recipients of PBG grants or other CSOs in the community. The evaluation team conducted approximately 137 semi-structured interviews, individually or in small groups, with persons belonging to the same organization.

Group interviews took place when individuals had limited time available due to work responsibilities (such as provincial administrative staff, ETD staff). Some interviews with CSOs also took place in groups.

Categories of interviewees included senior personnel of the National Assembly and provincial assemblies, donors, ETD personnel, provincial personnel, and CSOs. Some interviews were conducted by one of the team members while others were conducted by two team members interviewing together, the latter providing further analysis and comparisons of impressions and opinions. The list of persons interviewed is found in Annex VI.

Focus Group Data Collection Method Two members of the evaluation team are experts on organizing focus groups. In each data collection site, three focus groups comprising 10 persons each, including professionals, women and members of youth groups, were organized. Participants were selected by the evaluation team without support from PBG project offices. At each site, a focal point person was chosen to assist the team to identify and recruit 10 participants. In each site a note taker was employed to record the event, which was facilitated by the expert. In total, 30 focus group events were organized for approximately 300 persons. Ten of the focus groups comprised all women (all together there were more than 100 women since other focus groups also included women participants). Participants were asked to complete the 5-point rating scale (12 questions) at the conclusion of the focus group.

Rating Scale Questionnaire A 12-question 5-point rating scale was developed, revised with USAID and administered to a total of 473 persons during data collection. Of the 473 respondents, 202 were women. Most of the respondents had either been interviewed or had participated in a focus group. However, some questionnaire respondents were administrative staff of either parliaments or ETDs but had not been interviewed. The purpose of the ratings was to obtain numerical findings to correspond to opinions gathered. The scale was not intended to be a fully scientific survey, which would have required a stratified random sampling of respondents. A compilation of ratings by various criteria is presented in tables in the annexes.

Method for Inclusive Data Analysis During the data analysis stage, the five-member evaluation team met five times to analyze their notes and other findings. All 13 evaluation questions were systematically discussed. The data was then recorded in an analytical findings grid (later fully incorporated into the findings section).

The data used for the analysis included the many project reports, various PBG studies and background files from field offices that USAID provided, with a combination of notes from KIIs, focus groups discussion notes, observations made by the team members from their interviews or from observed project sites, and products and events observed. The questionnaire was analyzed using SPSS. At all times, the views of the four Congolese team members was key to extracting relevant material, based on their knowledge of context and personalities. Once compiled into draft findings and recommendations by the Subject Matter Expert (the only international team member), the team met again on other occasions to discuss the written material and make corrections. Some team members contributed written material in French, which was then translated into English.

Comparison Site The original evaluation plan called for control sites, ETDs where the project was not involved. The purpose was to ascertain whether differences could be found between project sites and non-project sites and what those differences were. During the completion of work planning in meetings with USAID/Kinshasa, discussions were held as to whether the control site option should be dropped. USAID recommended one additional site: Kenya commune in Lubumbashi city. This choice was made because it could be included in the field work with minimal time costs (available time for field work was already short) and because USAID was interested in this particular commune for other reasons. The team later realized that Kenya was too close and too similar to Katuba to serve as a control site.

Visits to Kenya commune took place at the conclusion of the field work in Katanga. The team had difficulty meeting ETD personnel, due to other events taking place in the commune. The PBG field office, which had helped schedule meetings in ETD communities did not have as many contacts as in their ETD project sites. Three focus group events were organized and conducted in Kenya. There were two other issues with this site. First, the Burgomeister of Kenya had served as the deputy in Katuba commune, and was familiar with PBG and its methods. Second, the same CSOs working in Katuba also worked in Kenya.

In fact this was one of several examples found where non-project ETDs were imitating project ETDs. In summary, Kenya yielded little information, except that PBG imitation is taking place, a sign that ETD methods were effective and potentially sustainable.

Limitations and Challenges The project had an extensive reach, including various sets of institutional partners and reporting systems.

There was limited time to examine all activities. The first work plan lists hundreds of outputs in some 30 or more categories. The evaluation could have conducted an extensive desk review of all outputs to compare to work plans over five years, had USAID wanted this approach. Full institutional and organizational assessments of the five parliaments alone could have absorbed the entire schedule. Travel was logistically complex and time consuming; for example, one sub-team needed three days to travel from Sud Kivu to south Bandundu. Data collection time exceeded the plan by three days, which reduced the available time for the five person team to work together to analyze the findings. Excluding Maniema from the evaluation may have impacted the findings. Specifically, the evaluation team found that the parliamentary capacity building activities had minimal results based on data collected at the sites visited.

However, it turns out that Maniema might have been an exception as Maniema was rated by a PBG parliamentary assessment in 2010 as having the most developed parliament.

These limitations did not in the end affect key findings; USAID had already indicated by the thrust of evaluation questions which areas were of greater interest, especially in relation to its interest in lessons for future programming: which of the activities and methods had the greatest potential for achieving results (as opposed to outputs). Ideally, had time permitted, it would have been useful to more deeply explore whether any PBG outcomes were achieved with provincial parliaments. Among the qualitative instruments developed by PBG to compliment the output indicators, the parliamentary assessment tool might have been helpful to compare 2010 and 2014 assessments, however PBG had not yet completed the 2014 parliamentary assessment. The CSO Index did measure changes in scores over the four year period.

However, neither this tool nor the other PBG assessment tools were evaluated as to their accuracy and realism in measuring changes.

FINDINGS

Findings are presented following the same structure as the evaluation questions from the SOW.

1. Institutional Capacity and Strength

1.1 To What Extent Are Project ETDs Better Resourced?

The evaluation found overall that the ETD project sites are better resourced at project conclusion compared to start up in 2009. Increased resources include tax revenues and strengthened human resources. The average increase of revenues for all 12 ETDs was 147.2% in four years. The evaluation found evidence that the project contributed, possibly very significantly, to increased revenues and strengthened skills of ETD personnel in transparent tax collection and finance management.

Increased Revenues: Table 3 presents tax revenues for the 12 project ETDs between 2008 and 2013.

Overall, there is a very significant trend of increased project-ETD own-sourced taxation revenues between 2009 and 2013. Eight of the 12 ETDs have increased their revenues from taxes – from 55% up to 514% in one ETD (Banghageli). Four ETDs had lower revenues or no change. The largest dollar increase, the city of Kolwezi, increased its revenues from $700,000 in 2009 to $2.5 million in 2013, a growth of 240%; the city reported to the consultants they expected nearly $5 million in revenues in 2014.

Retrocession. These tax revenues do not include taxes raised by the ETDs on behalf of provinces and central government, nor revenues obtained through retrocession (tax revenues from central government returned to provinces and distributed by provinces to ETDs). Regarding provincial levels, between 2010 and 2013, retrocession to provinces averaged between 5.6% and 6.3% of national revenues. When salaries of decentralized personnel and investments are included, retrocession was 28.8% in 2013.

Strengthened Human Resources: Human resources (HR) were strengthened due to project methods of repeated training combined with recurrent technical assistance (TA) support for ETD staff in areas of:

participatory five-year action planning, annual participatory budgeting, budget process and analysis, use of web sites, financial management and monitoring. Recurrent short-term on-the-job TA was provided to further strengthen the same targeted training and skills. Staff turnover in ETDs is relatively light, therefore these acquired skills are more sustainable. However, most human resources and leadership working for ETDs are, along with all provincial and technical employees (education, health, agriculture), employed by central government, about 330,000 persons in the provinces.

Materials/Equipment Support Minimal: With a few exceptions (such as Bukavu tax related activities), the project did not support material, equipment or systems for ETDs.

Evidence that Project Contribution to Increased Tax Resources: Interviews from different sources, both government and civil society, supported the conclusion that these different project-funded activities, targeting both…