WOPE.pdf

MONITORING AND EVALUATION SUPPORT ACTIVITY

(MEASURE-BiH)

THE WHOLE-OF-PROJECT EVALUATION OF

USAID/BiH PROJECT 2.2: IMPROVED

ECONOMIC ASPECTS OF GOVERNANCE

RELEVANT TO BUSINESS ACTIVITY

JUNE 2018

This document is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of authors and do not necessarily reflect the views of USAID or the United States Government.

MONITORING AND EVALUATION SUPPORT

ACTIVITY (MEASURE-BiH)

THE WHOLE-OF-PROJECT EVALUATION OF

USAID/BiH PROJECT 2.2: IMPROVED ECONOMIC

ASPECTS OF GOVERNANCE RELEVANT TO BUSINESS

ACTIVITY

June 2018

Prepared under the USAID’s Bosnia and Herzegovina Monitoring and Evaluation Support Activity

(MEASURE-BiH), Contract Number AID-168-C-14-00003

Submitted to:

USAID/Bosnia Herzegovina, January 2018

Contractor:

IMPAQ International, LLC

USAID.GOV MEASURE-BIH WOPE 2.2.

EVALUATION FINAL REPORT DRAFT 1

CONTENTS

CONTENTS

TABLE OF EXHIBITS

ACKNOWLEDGMENTS

ACRONYMS

EXECUTIVE SUMMARY

EVALUATION PURPOSE AND QUESTIONS

EVALUATION METHODS AND LIMITATIONS

FINDINGS AND CONCLUSIONS

MAIN RECOMMENDATIONS

I. EVALUATION PURPOSE, DESIGN AND LIMITATIONS

II. BACKGROUND INFORMATION ON PROJECT 2.2

III. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS

A. FINDINGS (RQ1)

B. CONCLUSIONS (RQ1)

C. RECOMMENDATIONS (RQ1)

D. FINDINGS (RQ2 – EIA)

E. CONCLUSIONS (RQ2 – EIA)

F. RECOMMENDATIONS (RQ2 – EIA)

G. FINDINGS (RQ2 - FSRA)

H. CONCLUSIONS (RQ2 – FSRA)

I. RECOMMENDATIONS (RQ2 – FSRA)

J. FINDINGS (RQ3)

K. CONCLUSIONS (RQ3)

L. RECOMMENDATIONS (RQ3)

ANNEX I: EVALUATION PURPOSE, DESIGN AND LIMITATIONS

I. EVALUATION PURPOSE

II. EVALUATION QUESTIONS

III. DATA COLLECTION AND ANALYSES

IV. EVALUATION LIMITATIONS

V. BACKGROUND INFORMATION ABOUT CONSTITUENT ACTIVITIES

ANNEX 1I: MAIN DATA COLLECTION INSTRUMENTS

ANNEX II.I: DOCUMENTS/DATABASES REVIEWED

ANNEX II.II: SEMI-STRUCTURED KEY INFORMANT INTERVIEW GUIDE FOR PROJECT-RELATED ISSUES

ANNEX II.III: MINI ONLINE SURVEY QUESTIONNAIRE FOR ENTERPRISES

ANNEX III: KEY INFORMANT INTERVIEWEES

ANNEX IV: THE EVALUATION TEAM RESPONSES TO COMMENTS FROM THE

IMPLEMENTING PARTNER

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 2

TABLE OF EXHIBITS

Exhibit 1. USAID/BiH 2012–2016 CDCS Results Framework, as of January 2012

Exhibit 2. Key Elements of PAD 2.2

Exhibit 3. Changes in the Project 2.2 Logical Framework at the Sub-Purpose Level

Exhibit 4. Assumptions Indirectly Outlined in the PAD

Exhibit 5. Doing Business Ranking of BiH

Exhibit 6. Score on the Regulatory Quality Dimension of the Worldwide Governance Indicators ... 16

Exhibit 7. Addressing Obstacles to Investments in the BiH Energy Sector

Exhibit 8. Indicator of Sub-Purpose of Fiscal Policy Reforms

Exhibit 9. In your opinion, how important is BiH's improvement in economic governance?

Exhibit 10. Sectors Important to Private Sector Companies

Exhibit 11. List of KIIs per Type of Key Informant

Exhibit 12. Evaluation Matrix

Exhibit 13. Map of EIA and FSRA Beneficiaries

Exhibit 14. Timing and Cost of the Project 2.2 Activities

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 3

ACKNOWLEDGMENTS

Guided by the U.S. Agency for International Development (USAID) Evaluation Policy, USAID/Bosnia and

Herzegovina (BiH) commissioned IMPAQ International within the USAID/BiH Monitoring and Evaluation

Support Activity (MEASURE-BiH) to conduct the Whole-of-Project Evaluation of USAID/BiH Project 2.2:

Improved Economic Aspects of Governance Relevant to Business Activity. Two activities contributing to this project— the Energy Investment Activity (EIA) and the Fiscal Sector Reform Activity (FSRA)—have been implemented by USAID/BiH within the current Country Development and Cooperation Strategy (CDCS)

2012–2016 period examined in this evaluation, beginning March 2014, when the 2.2 Project Appraisal

Document was finalized.

The evaluation team members include Davorin Pavelic (Team Leader and MEASURE-BiH Chief of Party), Edis Brkic (evaluation team member and Deputy Chief of Party); Lejla Lazovic-Pita (evaluation team member and MEASURE-BiH Research Fellow), Fuad Strik (evaluation team member), Anesa Hadzic

(evaluation team member and MEASURE-BiH Research Analyst), and Anela Kadic Abaz (evaluation team member and MEASURE-BiH Research Analyst).

The authors wish to thank those who contributed to the process of developing this evaluation report. At

USAID/BiH, the following people were instrumental in designing the evaluation: Elma Bukvic Jusic, MEASURE-BiH Contracting Officer’s Representative (COR); Andrew Boegel, Economic Development

Office (EDO) Director; Ankica Gavrilovic, Energy Investment Activity COR. Samir Dizdar, FSRA COR and Project 2.2 Manager helped us understand the design of Project 2.2 and its activities and management.

We also thank the EIA and FSRA activity management teams for their assistance with activity documentation and databases. Any errors in this report are the sole responsibility of the authors.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 4

ACRONYMS

ADS Automated Directives System BiH

BMIS

Bosnia and Herzegovina Budget Management Information System

CDCS Country Development and Coordination Strategy

COP

COR

Chief of Party Contracting Officer Representative

DO Development Objective EDO Economic Development Office

EIA

EnC

EU

Energy Investment Activity Energy Community European Union

FBiH Federation of Bosnia and Herzegovina FSRA Fiscal Sector Reform Activity

GIZ

German Agency for International Cooperation (Deutsche Gesellschaft für Internationale Zusammenarbeit)

IMF International Monetary Fund IR Intermediate Results KI Key Informant KII Key Informant Interview M&E Monitoring and Evaluation MEASURE-BiH USAID/BiH Monitoring and Evaluation Support Activity MoFTER Ministry of Foreign Trade and Economic Relations

NGO

NEAP

Non-Governmental Organization National Environment Action Plan

PAD Project Appraisal Document PARCO Public Administration Reform Coordinator’s Office PECI Projects of Energy Community Interest RS Republika Srpska TPP Thermal Power Plants TS Treasury System USAID US Agency for International Development USAID/BiH US Agency for International Development in Bosnia and Herzegovina USG US Government

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 5

EXECUTIVE SUMMARY

IMPAQ International has been commissioned by USAID/BiH, as part of the USAID/BiH Monitoring and

Evaluation Support Activity (MEASURE-BiH), to conduct the Whole-of-Project Evaluation of USAID/BiH

Project 2.2, Improved Economic Aspects of Governance Relevant to Business Activity. Two activities contributing to the purpose of this project— the Energy Investment Activity (EIA) and the Fiscal Sector

Reform Activity (FSRA)—have been implemented by USAID/BiH within the current Country

Development and Cooperation Strategy (CDCS) period examined under this evaluation, beginning March

2014, when the 2.2 Project Appraisal Document (PAD) was finalized.

The Results Framework of the USAID/BiH 2012–2016 CDCS includes two Development Objectives

(DOs)/Project Goals. The second DO states: “A competitive, market-oriented economy providing better economic opportunities for all its citizens governs the Mission’s economic development interventions.”

Within this DO, USAID/BiH has a two-pronged approach through two CDCS Intermediate Results

(IRs)/Project Purposes. The first IR aims to help enterprise-level competitiveness (notably in agriculture and small and medium-sized enterprises in key sub-sectors), while the second IR aims to support priority regulatory and policy reform to improve the business environment in BiH. The latter is addressed through

Project 2.2 and is the subject of this evaluation report.

EVALUATION PURPOSE AND QUESTIONS

The purpose of the whole-of-project performance evaluation (WOPE) is to examine the entire

USAID/BiH Project 2.2, including all of its constituent activities, and progress toward the achievement of the project purpose. In order to assess Project 2.2 as a whole, along with its constituent activities, this evaluation answers the following three research questions: (1) To what extent were the programmatic and contextual assumptions identified in the project theory of change sufficient to achieve the purpose of

Project 2.2? (2) To what extent have the activities implemented under this project achieved their expected results, and to what extent have they been able to address and collectively contribute to the purpose of

Project 2.2? (3) To what extent have the activities implemented under this project been coordinated and how have internal and external activity/project management practices supported progress toward the purpose of Project 2.2?

EVALUATION METHODS AND LIMITATIONS

The evaluation team used a mixed-method approach based on an extensive desk review of implementation documents and databases received from USAID/BiH as well as additional documents collected from the

EIA and FSRA activities, semi-structured interviews with 63 key informants (KIs), and a mini online survey of a sample of enterprises in BiH. The main limitations of this evaluation relate to possible selection bias, response bias, and recall bias.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 6

FINDINGS AND CONCLUSIONS

The number of implemented activities was not sufficient to achieve the Project 2.2 purpose. Due to its late start, changes in the Mission’s management, and lack of funding, Project 2.2 was only partially implemented and lacked a cohesive approach. The Project Appraisal Document for Intermediate Result

(PAD) 2.2. was designed and approved two years after adoption of the CDCS. Only three of eight planned activities were implemented: two activities (EIA and FSRA) are at the late stage of implementation, while the third one (FINRA) has only begun. The small number of implemented activities made project-level management almost unnecessary.

With regard to the project’s implemented activities, a majority of interviewed beneficiaries expressed their satisfaction with the technical assistance provided by EIA and FSRA and felt that the interventions implemented under these two activities are of highest priority and relevant for these two sectors.

With regard to the FSRA’s expected results, however, not much has been achieved in terms of improved coordination at the level of the Fiscal Council of BiH and consistent reporting to the European Union

(EU). Although analytical capacity of the entity’s ministry of finance in Republika Srpska (RS) is considered a success, the evaluation team could not confirm that the analytical capacity of two ministries of finance has improved. The Budget Management Information System (BMIS) is expected to achieve most results by the end of the activity. However, the implementation of the treasury system (TS) in local communities in the Federation of Bosnia and Herzegovina (FBiH) requires further actions through the end of the activity. Introduction of the TS to the RS Health sector was a premature intervention given the current state of the health sector in RS.

However, certain progress has been made. Strengthening of the Public Debt Management system in BiH was cancelled in October 2017, but some lessons were learned for the future. Although the expected quantitative results related to tax administration in BiH were confirmed in RS, those in FBiH are expected by October 2018. Implementation of the single registry of business accounts in FBiH was assessed as needed, and the implementation was conducted very quickly. Further actions are planned by the end of the activity.

Regarding achievement of the EIA’s expected results, an inadequate regulatory framework still renders the business environment in the electricity sector ill-suited to investments. Despite tremendous efforts of the EIA to identify all the impediments to investments, prepare comprehensive sets of recommendations for their removal and the requisite policy documents, and build institutional capacities, there has been no major breakthrough to improve the investment climate. Although the energy retail market in BiH is functional, the EIA still needs to achieve Result 2.1, Distribution System Operator (DSO) as an independent market facilitator, which is one of the preconditions for the development of other elements of an open electricity retail market. An analysis of coordination processes at the project level, and interaction among the constituent activities, showed that interaction between EIA and FSRA activity is weak.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 7

MAIN RECOMMENDATIONS

This report provides several recommendations for USAID/BiH’s consideration based on lessons learned from the evaluation:

• First, a project-level management function and a project-level coordination mechanism should be established, formalized, and institutionalized.

• The participatory approach to planning development interventions should be further strengthened in order to gather a more realistic overview of the context in which activities are implemented.

For policy-level interventions, USAID/BiH should bring the relevant stakeholders to the table early in the design process. USAID/BiH should ensure and confirm, through an active political process and a request for a guarantee and proof of local stakeholders’ commitment that programmatic and contextual assumptions are in place before beginning technical preparations for the implementation of activities. Where guarantees and strong commitments from local decision makers cannot be obtained, USAID/BiH should consider discontinuing further assistance.

• USAID should not expect the implementing partner to resolve problems resulting from a political arena that has not been well prepared for the technical implementation of activities. Contracts should be awarded to implementing partners with strong experience and expertise in both the technical and political segments of the implementation of development initiatives.

• Regarding energy sector reforms, USAID should be flexible and realistic in setting targets for its activities because energy sector reform is a long-term and complex process, especially given the constitutional complexity regarding the jurisdiction of BiH national and entity levels.

• USAID should assess support for the establishment of an electricity stock exchange in BiH for further development of the electrical energy market. Some experts consider such an institution to be an urgent requirement. With three existing electric power companies and a number of registered energy suppliers, BiH has the necessary preconditions for the development of a stock exchange for electric energy and its integration into an evolving regional stock exchange network.

• To strengthen the fiscal sector, further technical support related to macroeconomic analysis and macro-econometric modelling would be beneficial for building stronger analytical capacities in all ministries of finance, which should result in more reliable projections.

• Also, more training and education related to implementation of BMIS (program budgeting, budget beneficiaries, etc.) would be beneficial by the end of the activity. In the future, BMIS should be connected with the treasury system and potentially introduced to local communities.

• Interventions related to a single registry of business accounts in BiH should be continued. USAID should persist in interventions related to registries of parafiscal fees, especially in terms of elimination and reduction of these fees in both entities and Brcko District.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 8

I. EVALUATION PURPOSE, DESIGN AND LIMITATIONS

The goals of the whole-of-project performance evaluation of USAID/BiH Project 2.2, Improved Economic

Aspects of Governance Relevant to Business Activity, are (i) to examine the progress of the entire project toward its stated purpose, and the progress of its associated constituent activities toward project sub-purposes and outcomes/outputs; (ii) to describe project and constituent activities coordination and implementation; and (iii) to provide recommendations for possible future USAID/BiH interventions.

The evaluation questions are broadly based on the guidance from Automated Directives System (ADS)

201, Additional Help for Whole of Project Evaluation. The first question studied the strategic planning process of the Mission at the project level by examining strengths and weaknesses of the project-designed theory of change. The second question examined the project’s performance and the results that were achieved. The third examined the strategic coordination process of the Mission at the project level by looking at the interactions among constituent activities as they contribute to the project purpose.

The evaluation was based on a rigorous evaluation design and methodological approach to address the evaluation questions and to provide insights into the progress towards expected results at the project level, as well as at the level of the two constituent activities included in this project, by connecting the findings to conclusions, lessons learned, and recommendations.

The research design employed mixed methods so that the gathered information could be triangulated in addressing each of the evaluation questions using different data sources: documents, key informant interviews, and a mini online survey of a sample of enterprises in BiH.

The data from these sources were triangulated to address the same questions/sub-questions from multiple perspectives whenever possible. Comparing data helped the team to gain a more complete understanding of the issues and provided more confidence in the findings.

The main limitations of the evaluation include response bias, recall bias, and selection bias. These limitations were mitigated by drawing on multiple sources of information, by carefully designing and conducting data collection, and by making sure to include the main stakeholders and beneficiaries of the activities. More information on the evaluation purpose, design, and limitations is presented in Annex II.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 9

II. BACKGROUND INFORMATION ON PROJECT 2.2

The Results Framework of the USAID/BiH 2012–2016 Country Development and Coordination Strategy

(CDCS) is presented in Error! Reference source not found. 1.1 The framework includes two CDCS

Development Objectives (DOs)/Project Goals, one of which—DO 2: A competitive, market-oriented economy providing better economic opportunities for all its citizens—governs the Mission’s economic development interventions. Under this DO, USAID/BiH has two-pronged approach through two CDCS

Intermediate Results (IRs)/Project Purposes to promote enterprise-level competitiveness (notably in agriculture and small and medium-sized enterprises in key sub-sectors) and to support priority regulatory and policy reform to improve the business environment in BiH. The latter is addressed through Project

2.2, Improved Economic Aspects of Governance Relevant to Business Activity.

Exhibit 1. USAID/BiH 2012–2016 CDCS Results Framework, as of January 2012

At the time of developing the CDCS (January 2012), USAID/BiH identified financial sector reform and capacity building as priorities for assistance under this IR because of the importance of financial sector stability in attracting investment to BiH. Specifically, the CDCS aimed at supporting the growth of competitive financial systems while strengthening transparency and oversight to enhance the stability of

1 Exhibit 1 shows the Results Framework included in CDCS. The Mission subsequently modified the Results Framework in 2014, including replacing IR 2.2.1 with 2.2.1 Fiscal policies reformed; 2.2.2 Employment, social and disabilities policies reformed; 2.2.3 Financial sector in line with international standards; 2.2.4 Energy sector's sustainable contribution to economy increased; 2.2.5 Green development introduced; and 2.2.6 Improved public service delivery efficiency and enhanced broad stakeholder involvement in public service by high-quality e-Governance.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 10

financial markets, as well as possibly directing assistance also toward insurance and pension reform. Finally, the CDCS discussed planned support for energy sector reform in BiH, given the importance of the energy sector for exports, revenue generation for BiH, and attractiveness for foreign investment (i.e., power stability). To build on earlier USAID/BiH energy interventions, continued assistance was planned in developing transparent, functional and responsive energy regulatory bodies, to help lead the energy market in a more market-driven direction and promote the integration of the BiH energy sector into the larger

Southeast and European Energy Market and energy trade bodies.

As of September 2014, USAID/BiH began implementing two activities during this CDCS period: the Energy

Investment Activity (EIA) and the Fiscal Sector Reform Activity (FSRA). The third activity under this

Project—the USAID Financial Reform Agenda Activity (FINRA)—commenced in October 2017 and is thus not part of this evaluation.

� The Energy Investment Activity (EIA) is a $5.47 million activity for the period from

September 2014 to September 2019, that is being implemented by Advanced Engineering

Associates International. The purpose of the EIA, as specified in the award, is to increase investments in the energy sector in BiH and advance the BiH accession process related to energy requirements by working on the following tasks: improving coordination, management, and transparency at all levels of BiH’s regulatory framework, making a simplified energy investment environment, and targeting technical assistance to integrate BiH’s energy sector into regional and

EU markets.

� The Fiscal Sector Reform Activity (FSRA) is a $5.0 million activity for the period December

2014 to September 2019, that is being implemented by FINIT Consulting, a local contractor. As specified in the award, the FSRA supports efforts to facilitate gains in fiscal sector and public investment, which in turn should result in an increase in private domestic and foreign investments in BiH. The focus of the activity is fiscal discipline, which entails a comprehensive set of initiatives tackling coordination between the different levels of the BiH governments, and additional work with governmental institutions such as the tax administrations.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 11

III. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS

Evaluation Question 1: To what extent were the programmatic and contextual assumptions identified in the project theory of change sufficient to achieve the Project 2.2 purpose? What are the lessons learned for USAID/BiH future planning at the project level?

A. FINDINGS (RQ1)

Finding 1: The number of implemented activities and the number of sectors covered are not sufficient to support achievement of the Project 2.2 purpose. The Project Appraisal Document

(PAD) for Intermediate Result 2.2 was designed and approved in March 2014, two years after adoption of the CDCS. Both of the activities implemented under this PAD began in the last quarter of 2014 and will be completed 2019, one year after the expiration of an extension of the CDCS’s implementation. Of the eight activities planned for implementation, only two are in the late stages of implementation, while a third has just begun.

Finding 2: There are several issues with the PAD’s narrative:

� The PAD narrative describes a rather complex logical framework consisting of the goal, ultimate goal, purpose, and results (see Exhibit 2), and the LogFrame table does not reflect the PAD’s description.

� There is no clear hierarchy between “ultimate project’s goal” and the actual project goal.

� The definition of the project purpose “Improved economic aspects of governance relevant to business activity” is not clear and has resulted in many different interpretations.

� The project purpose outlined as an intermediate result (IR) 2.2 is different in the PAD and the

LogFrame.

� It is stated that IR 2.2. will have a specific focus on sub-IR 2.2.1 (Regulations and policies that foster economic space and private sector development and investments), which is a synthesis of the project’s goal and the project’s “ultimate goal.”

� Sub IR-2.2.1 is mentioned several times in the PAD, and each time it is defined differently.

Exhibit 2. Key Elements of PAD 2.2

PAD LogFrame Assumptions

Project’s ultimate goal

Create a single economic space and business friendly environment for economic growth.

N/A N/A

Project’s goal

Improve economic aspects of governance relevant to the business -enabling environment through improvement of the commercial legal framework, fiscal policies and delivery of goods and services provided by the government to citizens and businesses, N/A N/A

IR 2,2

Project’s purpose

Achieve more functional and accountable institutions and actors to meet citizens’ needs in BiH.

Improved economic aspects of governance relevant to business activity

N//A

IR 2.2.1.

Regulations and policies that foster a single economic space and private sector development and investments

Fiscal policies reformed N/A

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 12

Finding 3: None of the five studies and analyses2 planned to support and inform design of specific activities were completed. The absence of these analyses may have affected the design of activity-level theories of change.

Finding 4: The key informant interviews indicate that the changes at the activity level were extensive, influenced by lack of funding and changes in management, and had a negative effect on the initially envisioned theory of change. All planned activities were interconnected and sequenced so that full completion of one activity would create the necessary preconditions for the implementation of another. Not implementing certain planned activities undermines the results framework and the envisioned theory of change. ADS 201 states that PAD only provides an organizing framework, which should be adapted when new evidence emerges or circumstances change. The Project’s LogFrame has changed significantly over the last five years. This is especially true at the Project’s sub-purposes level, which changed in terms of numbers and sectoral focus (see Exhibit 3).

The following project sub-purposes were planned as part of Project 2.2’s purpose in PAD 2.2:

2.2.1. Fiscal sector strengthening activities (to result in improved coordination of the budget and treasury systems)

2.2.2. Adoption of employment, social, and disability policies related to economic growth (to result in introduction of a workman’s compensation system)

2.2.3. Financial sector providing stability and serving private sector needs (to result in financial sector reform – adoption of international standards)

2.2.4. Energy reform policies (to result in an increased sustainable contribution of the energy sector to the economy)

2.2.5. Anticorruption—Focus on improved control over public resources/funds (not included in the

LogFrame associated with PAD)

2.2.6. Green development (to result in the introduction of green development)

2.2.7. E-governance reform (to result in improved utilization of e-governance).

2 (1) Detailed feasibility studies for activities envisioned during the second phase of the project, such as pension and disability insurance fund reform, analyzing the overall environment for the reform, possible risks, cost-effectiveness, and expected results;

(2) detailed analysis of the disability insurance system; (3) detailed analysis focusing on labor market rigidities and possible reforms;

(4) detailed analysis focusing on pension system reform; (5) analysis focusing on green development reform.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 13

Exhibit 3. Changes in the Project 2.2 Logical Framework at the Sub-Purpose Level

USAID/BiH CDCS

January 2012

USAID/BiH Project

PAD 2.2

March 2014

USAID/BiH Project 2.2 LogFrame March 2014

USAID/BiH Project 2.2 LogFrame

December 2017

Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.1

Regulations and policies foster private sector development and investment

Fiscal sector strengthening activities

Improved coordination of budget and treasury systems

Fiscal policies reformed

Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.2

Adoption of employment, social, and disability policies related to economic growth

Financial sector reform— adoption of international standards

Employment, social, and disabilities policies reformed

Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.3

Financial sector providing stability and serving private sector needs

Workman's compensation system introduced

Financial sector in line with international standards

Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.4

/ Energy reform policies

Energy sector's sustainable contribution to economy increased

Energy sector's sustainable contribution to economy increased

Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.5

Anti-corruption—Focus on improved control over public resources/funds

Utilization of e-governance improved

Improved public service delivery efficiency and enhanced broad stakeholder involvement in public service by high-quality e-governance

Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.6

/ Green development Green development introduced Green development introduced

Sub-Intermediate Results 2.2 / Project Sub-Purpose 2.2.7

/ E-governance reform / /

Finding 5: Programmatic and contextual assumptions were not explicitly defined for PAD

2.2. However, certain programmatic assumptions can be identified through the PAD 2.2 narrative (see Exhibit 4). Four identified contextual assumptions are necessary for achievement of the

Project 2.2 purpose, but they are not sufficient for achievement of that purpose. It should be noted that none of these assumptions were examined or tested.

Exhibit 4. Assumptions Indirectly Outlined in the PAD

# Critical Assumption Satisfied (Yes/No) Comment

1 The project will benefit from coordination and cooperation with

USAID’s civil society organizations and independent media cross-cutting programs and the relevant departments of the US Embassy in Sarajevo (Economic and Political Sections) to facilitate rapid and smooth adoption of the regulations and policies targeted by the project.

Partially Activities implemented under PAD 2.2 established good cooperation with some of the cross-cutting programs of USAID’s civil society organizations.

However, adoption of the regulations and policies targeted by the project has not been rapid or smooth.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 14

# Critical Assumption Satisfied (Yes/No) Comment

2 Entity governments and state-level counterparts will readily accept USAID assistance in the identified areas (PAD 2.2)

Partially Not all areas of USAID’s assistance were accepted by entity-level governments. This was especially the case with RS, which refused to work on FSRA’s fiscal council component and EIA’s energy efficiency obligation schemes.

3 It is expected that all three levels of government will be USAID’s champions in the implementation phase.

No Very few, if any, of the partnering institutions could be considered champions of USAID’s supported reforms under PAD 2.2.

4 USAID will seek to partner with other principal donors in BiH to amplify its call for host country buy-in and to pursue opportunities for additional funding.

Partially USAID established partnerships with other principal donors in BiH, but no additional funds were raised.

Finding 6: The commitment of BiH authorities to champion necessary reforms, one of the critical assumptions outlined in the PAD, was not satisfied

(please see the European Commission

2018 report on Bosnia and

Herzegovina3). Lack of political will on the part of decision makers to pursue necessary reforms was also confirmed in interviews with the key informants. Almost one-third of all interviewees identified a lack of political will as the main obstacle to the successful implementation of EIA and FSRA activities.

Finding 7: Understaffed institutions and their low capacity to efficiently implement USAID’s recommendations is one of the important assumptions that was not taken into consideration during the design activities. Several issues continually surfaced during the key informant interviews:

inadequate human resources, both in terms of numbers and competencies, poorly equipped institutions, insufficient financial resources available for the operation of these institutions, and absence of communication and coordination of work (vertical and horizontal) between and within institutions.

Finding 8: The document review and the USAID/BiH staff interviews indicate that during the PAD concept development phase, USAID consulted host government officials, who recommended the majority of activities under PAD 2.2; however, the evaluation team found that some of EIA’s and FSRA’s stakeholders and beneficiaries were consulted and informed

3 European Commission. Key Findings of the 2018 Report on Bosnia and Herzegovina; httsp://ec.europa.eu/rapid/press-release_MEMO-18-3408_en.pdf

“(i) with the exception of a few reforms and the notable adoption of the excise legislation, delivery on a number of reforms was delayed by lack of agreement within the ruling coalition members.

Countrywide strategies in areas such as energy, employment or public financial management are yet to be adopted. (ii) A national programme for the country's legal approximation with the EU acquis has yet to be adopted; (iii) Bosnia and Herzegovina is at an early stage with the reform of its public administration and no progress has been achieved in the past year.”

—Excerpt from the European Commission 2018

Report on BiH

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 15

only after the contracts for these two activities had been awarded. During the development of

PAD 2.2, major donors active in sectors of interest to USAID (EU, UNDP, and GIZ) were consulted.

Finding 9: Out of thirty (30) FSRA key informants, only six (6) stated that they were consulted during the activity’s intervention design process. All six KIs that were consulted are relevant and hence were included in the creation of specific interventions under the FSRA, with the possibility of disseminating related information through appropriate internal mechanisms within their respective institutions. These KIs all responded that they were included in previous USAID/BiH fiscal sector activities, that they were informed in a timely manner about the FSRA, and that they were directly included in the creation or signing the Memorandum of Understanding between interested parties that was signed after the award of the contract in May 2015.

Finding 10: Eight (8) major stakeholders from the BiH energy sector, the state and entity-level ministries in charge of the energy sector, regulators in both entities, and three electric power authorities were informed about the EIA in September 2014 after the award of the contract. Eighteen (18) of thirty-seven (37) interviewed stakeholders confirmed that they were informed about the interventions USAID intended to support related to energy sector reforms within EIA. They also stated that they received a presentation by a project implementer before activities started or were contacted in advance and included in the period of activity implementation in 2015.

Finding 11: Although one USAID staff member is in charge of the management of Project

2.2, that position was never formalized within the Mission. According to ADS 201, the Project

Manager is responsible for project implementation oversight, monitoring and evaluation, and learning and adoption. However, none of these project management functions were ever applied to the management of Project 2.2. The entire management function was reduced to a coordination role. Initially, coordination started with regular monthly meetings, but even that practice ceased six months after the start of EIA and

FSRA.

Finding 12: The project’s goal-level indicators, the World Bank’s Doing Business ranking and Distance to Frontier, which are measurements of the quality of the business regulatory environment and set to track the progress of DO2 achievement, are equally applicable to the measurement of progress in achieving the Project 2.2 purpose. Changes in these two indicators (see Exhibit 5) show that little or no progress was made during the last four years. The World Bank indicator of regulatory framework quality and progress of reforms shows that BiH’s ranking has been continuously worsening

Exhibit 5. Doing Business Ranking of BiH

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 16

since 2015. The distance to frontier,4 as a measure of how BiH is performing in advancing its business regulatory environment, shows that there was no significant progress over the last four years.

Finding 13: The project-level indicator related to the regulatory quality of the

Worldwide Governance indicator shows that no progress was made over the last four years. Changes in this indicator show that the business environment and investment climate was worse in 2017 than it was in 2014 (see Exhibit 6).

Finding 14: The poor investment and business environment in BiH, specifically in the energy sector, was confirmed through the KIIs. Investing in the energy sector in BiH is a complex, time consuming, and expensive process with an unpredictable outcome. This is equally valid for foreign investors and for private and public sector investors from BiH. Also, the majority of private sector companies that responded to the mini-survey thought that during the last three years, no progress

(47%) or only small improvements (45%) were made in addressing obstacles to investments in the energy sector (see

Exhibit 7).

4 The distance to frontier measure shows the distance of each economy to the “frontier,” which represents the best performance observed on each of the indicators across all economies in the Doing Business sample since 2005. An economy’s distance to frontier is created on a scale from 0 to 100, where 0 represents the lowest performance and 100 represents the frontier. The ease of doing business ranking ranges from 1 to 190. The ranking of 190 economies is determined by sorting the aggregate distance to frontier scores, rounded to two decimal places.

51,7 51,0

47,6 48,6

2014 2015 2016 2017

47%

45%

7%

1%

No improvement Small improvement Better Significant improvement

Exhibit 6. Score on the Regulatory Quality Dimension of the Worldwide Governance Indicators

Exhibit 7. Addressing Obstacles to Investments in the BiH Energy Sector

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 17

Finding 15: There has been no significant progress at the sub-purpose level in fiscal policy reforms. Regarding the tax burden and size of the government indicator, there was almost no change over the last four years. There were some small improvements in the paying taxes indicator (see Exhibit 8)

Finding 16: A majority of private sector companies (83.87%) believe that improvements in economic governance are important for BiH’s further development (see Exhibit 9).

PAD 2.2 defines improvement of economic aspects of governance relevant to business activity as the purpose of the project, which further reduces the project’s focus on an improved regulatory and policy framework that will foster private sector development and investments. The significance of such a purpose is supported both by the key informants and the findings of the mini-survey conducted with private companies.

82,9 82,9 83,9 83,5

58,2 55,2 60,8 60,4

2014 2015 2016 2017

Score in the Tax Burden dimension of the Government Size pillar within the Economic Freedom Index Score in Paying Taxes dimension of Doing Business (measuring time and costs saved for businesses), distance to frontier

83,87%

13,15%

2,48%

0,25%

0,25%

Very important

Mostly important

Partially important

Not important nor not important

Not important

Exhibit 8. Indicator of Sub-Purpose of Fiscal Policy Reforms

Exhibit 9. In your opinion, how important is BiH's improvement in economic governance?

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 18

Finding 17: USAID/BiH’s selection of priority sub-purposes that were actually funded

(energy and fiscal sectors) is not in line with the priority sectors (sub-purposes) identified by private sector companies. Private sector companies gave the highest priority to sub-purposes related to anti-corruption, stability of the financial sector and e-governance (see Exhibit 10). The priority ranking given by private sector companies is logical and covers three important parts of enabling the business environment. Anti-corruption will give the same opportunities to all businesses in the market, a stable financial sector will provide businesses with better access to finance, and e-governance will save time and cost to businesses when they are dealing with public administration and bureaucracy.

Finding 18: A majority of key informants consider that the two activities, FSRA and EIA, directly contribute to the fulfilment of the Project 2.2 purpose. The design, objectives and expected results of the two USAID/BiH’s activities evaluated under WOPE are sound and could significantly contribute to the fulfilment of Project 2.2. However, full achievement of the project’s purpose requires implementation of additional interventions in other areas (anticorruption activity, less costly and more efficient and business-oriented public administration, etc.).

Finding 19: The initial LogFrames of both activities (EIA and FSRA) were changed or adjusted during the implementation period. The implementing partners indicated that the first LogFrames did not adequately capture the essence of the activity’s interventions.

“The new LogFrame resolved that issue. Now we have a system where lower activities move upwards and reflect exactly what we do.

The first LogFrame was not adequate in terms of describing what actually we do. “

Exhibit 10. Sectors Important to Private Sector Companies

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 19

B. CONCLUSIONS (RQ1)

Conclusion 1: The 2012–2016 CDCS has been extended through November 2018 and still provides a sound strategic framework for the programming and design of specific activities.

Conclusion 2: The main elements of PAD 2.2, such as the theory of change, programmatic and contextual assumptions, the project’s purpose, and the hierarchy of results are either only partially outlined and dispersed throughout PAD 2.2 or they are missing or not adequately defined.

Conclusion 3: Significant delays in the process of the PAD 2.2 design and the ratio of planned to implemented activities, make PAD 2.2 barely relevant for achievement of the CDCS’s goal and objectives.

Conclusion 4: Due to the small number implemented activities, project-level management was nonexistent and almost unnecessary.

Conclusion 5: Not much has been achieved regarding accomplishment of the DO2 and the

Project 2.2 purpose over the last four years. None of the project-level indicators show significant improvement.

Conclusion 6: At the sub-purpose level, programming was rather flexible, iterative, and responsive to contextual changes. However, the selection of sub-purposes was not based on the highest priority area, but rather was chosen because of long-term USAID/BiH involvement in the specific sectors.

Conclusion 7: The design, objectives, and expected results of the two USAID/BiH activities evaluated under WOPE are sound and could significantly contribute to the fulfilment of the

Project 2.2 purpose. However, EIA and FSRA by themselves are considered insufficient to achieve that purpose. Full achievement would require the implementation of additional interventions in other areas.

Conclusion 8: The combination of the long-term presence and assistance in fiscal and energy sectors, the absence of commitment and political will of local stakeholders to implement necessary reforms, and the statements by local stakeholders that USAID/BiH should continue with assistance in these two sectors reinforces the aid dependency trap.

C. RECOMMENDATIONS (RQ1)

Recommendation 1: USAID/BiH should invest in the training of its staff in project design in order to increase the quality of documents produced, such as the Project Appraisal

Document.

Recommendation 2: USAID/BiH should further strengthen a participatory approach to planning development interventions in order to gather a more realistic overview of the context in which activities are implemented. In addition, more comprehensive analyses of

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 20

programmatic and contextual assumptions and risks should be conducted. For policy-level interventions, USAID/BiH should bring the relevant stakeholders to the table early in the design process.

Moreover, USAID/BiH should ensure and confirm, through an active political process and a request for guarantees and proof of local stakeholders’ commitment, that programmatic and contextual assumptions are in place before beginning technical preparations for the implementation of activities. Declared support by local decision makers and the signing of memoranda of understanding should not be considered sufficient guarantees of local stakeholders’ commitment. Where guarantees and strong commitments from local decision makers cannot be obtained, USAID/BiH should consider ceasing further assistance.

Recommendation 3: USAID/BiH should carefully weigh trade-offs between political objectives and the costs of maintaining a presence in specific sectors through business-related activities that are expensive and difficult to implement. Long-term presence and involvement in certain sectors should not be a driving factor when selecting areas/sectors for interventions. Development needs and the solid commitment of all relevant stakeholders should be the two main criteria in the planning process.

Recommendation 4: USAID should not expect the implementing partner to resolve problems of a poorly prepared political arena for the technical implementation of activities.

Otherwise, contracts should be awarded to implementing partners with strong experience and expertise in both the technical and political aspects of implementing development initiatives.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 21

Evaluation Question 2: To what extent have the activities implemented under this project achieved their expected results and to what extent have they been able to address and collectively contribute to the Project 2.2 purpose? How have constituent activities been implemented and what were the main challenges and lessons learned from stakeholders’ perspectives for any potential future interventions?

D. FINDINGS (RQ2 – EIA)

Energy Investment Activity

Finding 21: The key informants (KIs) consider the interventions implemented by EIA highly relevant to energy sector reforms in BiH because investors are facing a range of important challenges. The twenty-eight (28) interviewed stakeholders all noted that there are a range of challenges in the implementation of the interventions aimed at energy sector reforms in BiH. One of the most often mentioned challenges is contradictions between laws on different institutional levels or different interpretations of the same law and unclear jurisdiction.

The absence of a development strategy for the energy sector on the state level (due to the political resistance against state-level strategies) is recognized as a long-term challenge for energy sector regulation on the state level. Lack of actual and detailed spatial planning documents on all levels relevant to electricity power sources and renewable energy is also considered a challenge.

All KIs viewed political unwillingness to adopt and implement solutions and improvements as a main challenge. For this reason they have concerns about the final implementation and practical adoption of the documents, rules, or recommendations developed under EIA.

KIs who were members of EIA working groups emphasized that they are powerless with regard to the full implementation of working group interventions. They also believe that EIA implementers cannot have significant influence on the adoption of the regulations or implementation of the agreements because higher-level political interests are the dominant influence.

Finding 22: The EIA faced difficulties in getting support and cooperation from certain RS institutions and ministries for implementation of the interventions. Support was lacking especially for certain components of the EIA, such as the introduction of obligation schemes.

Finding 23: Even the largest state-owned companies involved in electricity production could not successfully implement large investments in electricity production facilities. Jurisdiction for carrying out processes and steps within the authorization framework for the development of energy projects is divided among different government levels in Bosnia and Herzegovina (the state, the entity, and

Brcko District). Consequently, the permitting procedure in FBiH is conducted at the FBiH and/or cantonal and municipal level. In contrast, the permitting procedure in the RS is more centralized at the entity level, but may include local authorities, depending on the capacity of the planned power plant. As can be seen, the permitting, licensing, and other authorization procedures in Bosnia and Herzegovina are very complex and are not harmonized.

USAID.GOV MEASURE-BIH WOPE 2.2. EVALUATION FINAL REPORT DRAFT 22

Inefficiency, lack of transparency, and complex permitting procedures are slowing down the development of energy sector, as can be seen in the example of EP BiH and EP HZHB. Even government-owned companies like these cannot go through this process without obstacles and forced delays and stoppages.

For example, EP BiH has six major projects that have been in the design phase for more than eight years.

In its struggle to finish the…