Attachment_5-Advancing_Kosovo_Together_(AKT)_Performance_Evaluation_Final_Report_pdf.pdf

FINAL EVALUATION REPORT

USAID ADVANCING KOSOVO

TOGETHER

Final Performance Evaluation

DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION ii

FINAL EVALUATION REPORT

USAID Advancing Kosovo Together Final Performance Evaluation

USAID/Kosovo

AID-167-TO-17-00007; AID-167-I-17-00003

August 9, 2017

COVER PHOTO CREDIT: ADVANCING KOSOVO TOGETHER - LOCAL SOLUTION

This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by Mateusz Pucilowski, Jean-Camille Kollmorgen, Besnik Cecelia and Dijana Spalevic on behalf of Social Impact, Inc.

Contact:

Social Impact, Inc.

2300 Clarendon Blvd., Suite 1000 Arlington, VA 22201 Kerry Bruce, Executive Vice President kbruce@socialimpact.com Phone: +1 703.465.1884 Fax: +1 703.465.1888

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION iii

CONTENTS

Tables and Figures ....................................................................................................................... iv

Acronyms ....................................................................................................................................... v

Executive Summary ..................................................................................................................... vi

Project Background ..................................................................................................................................................... vi

Evaluation Purpose and Questions .......................................................................................................................... vi

Evaluation Methods and Limitations ....................................................................................................................... vii

Evaluation Question 1 – To what extent has the project been effective? ................................................... viii

Evaluation Question 2 - What are the key lessons learned from the project to date? ............................. xii

Recommendations ..................................................................................................................................................... xiii

Project Background

Program Background and Description

Evaluation Purpose and Evaluation Questions

Evaluation Purpose

Evaluation Questions

Evaluation Methods and Limitations

Methods

Limitations

Findings and Conclusions

Evaluation Question 1

Conclusions

Evaluation Question 2

Recommendations

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION iv

TABLES AND FIGURES

Table 1: Implementation Components within AKT

Table 2: AKT-LS Performance Under Objective 1

Table 3: AKT-US Performance Under Objective 1

Table 4: AKT-US Performance Under Objective 3

Table 5: AKT-LS Performance Under Objective 3

Table 6: Sex and Ethnic Disaggregation AKT-US

Table 7: AKT-US Performance under Objective 2

Table 8: AKT-LS Performance under Objective 2

Table 9: AKT-US Performance under Objective 4

Figure 1: AKT Results Framework

Figure 2: Data Collection Sites by Municipality

Figure 3: Respondent Groups

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION v

ACRONYMS

AKT Advancing Kosovo Together ASM Serb-Majority Municipalities B2B Business 2 Business CDCS Country Development Cooperation Strategy CDF Community Development Fund CF Community Fieldwork COR Contracting Officer Representative CSO Civil Society Organization DRG Democracy, Human Rights and Governance EQUI Evaluation Quality, Use, and Impact ET Evaluation Team EU European Union FGD Focus Group Discussion FTE Full-time Equivalent GI Group Interview IP Implementing Partner IR Intermediate Result JWG Joint Working Group KCC Kosovo Chamber of Commerce KII Key Informant Interview KRD Kosovo Relief Development LEAP Local Environmental Action Plan LOP Life of Project LS Local Solutions LUL Law on the Use of Languages M&E Monitoring and Evaluation NGO Non-Governmental Organization OCA Organizational Capacity Assessment OLC Office of the Language Commissioner PMP Performance Management Plan QA Quality Assurance RAE Roma, Ashkali and Egyptians RFA Request for Application SI Social Impact SIAP Service Improvement Action Plan MSME Minority-Owned Small and Medium Enterprise SOW Scope of Work TPM Team Planning Meeting USAID United States Agency for International Development USG United States Government

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION vi

EXECUTIVE SUMMARY

USAID/Kosovo contracted Social Impact Inc. to conduct a performance evaluation of the Advancing Kosovo Together (AKT) activity1 on January 17, 2017. This report presents findings, conclusions and recommendations structured around the two primary evaluation questions.

PROJECT BACKGROUND

Advancing Kosovo Together (AKT) is a USAID-funded activity designed to increase constructive inter-ethnic cooperation between Kosovo’s majority and minority populations. AKT is implemented through two complementary mechanisms. Chemonics International implemented the $7,587,654 AKT Prime mechanism (also known as AKT-US) between April 2014-April 2017. Community Development Fund (CDF) continues, at the time of writing, to implement the $6,999,706 AKT Local Solution mechanism (also known as AKT-LS). AKT-LS began in October 2014 and is scheduled to conclude in March 2018.

The two mechanisms were designed to work together in order to increase constructive inter-ethnic cooperation and interaction by achieving three intermediate Objectives:

Objective 1 – Improved autonomous, horizontal communication between (a) Kosovo Albanian and non-majority communities; and (b) the Government of Kosovo and non-majority communities

Objective 2 – Improved economic opportunities in target municipalities

Objective 3 – Increased efficiency and capacity of target municipal administrations to respond to the needs of all their citizens

In addition to these three shared objectives, AKT-US was designed to provide capacity building support to the AKT-LS implementer. Accordingly, AKT-US had an additional Objective 4. AKT implements activities across 16 Kosovo partner municipalities, including six Serb-majority municipalities in the South, six Albanian-majority municipalities, and four municipalities in northern Kosovo.

EVALUATION PURPOSE AND QUESTIONS

The purpose of the evaluation, as articulated by the USAID/Kosovo Mission was to:

1. Inform the 2018-2023 USAID/Kosovo Country Development Cooperation Strategy (CDCS) and future programing; and

2. Know whether the AKT engagement—focusing on minority community integration—has been working.

The primary target audience for the evaluation is USAID/Kosovo and AKT implementing partners (IPs).

Other key stakeholders include other US Government (USG) entities and international donors supporting ethnic minority integration in Kosovo; the Government of Kosovo (GoK), particularly local government agencies such as the Ministry of Local Government Administration, Ministry of Communities and Returns, and Ministry of Finance; and municipal government officials.

The evaluation had two main evaluation questions, each with a number of sub-questions:

1 When AKT was first designed, it was referred to as a “program” within USAID. The term now commonly used per the ADS to refer to a program such as AKT is “activity”. AKT is referred to as an activity throughout the remainder of the report. Similarly, smaller activities undertaken by AKT are referred to as “interventions” throughout the report.

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION vii

1. To what extent has the project been effective?

a. Has the program achieved its intended results, especially under Objectives 1 and 3?

Why or why not?

b. What are AKT achievements in North Kosovo to date? What worked and what did not?

c. How has the sustainability of USAID investments been addressed?

2. What are the key lessons learned from the project to date?

a. Should there be any programmatic shifts to the AKT program to achieve better results?

Are there new opportunities for minority integration that future programming could address?

b. Is there a potential for greater AKT program collaboration with other USAID programs and/or other donors?

EVALUATION METHODS AND LIMITATIONS

Primary data was collected through two rounds of data collection. In both rounds, the ET held in-briefs and out-briefs with USAID.

In the first round the ET purposively selected 10 out of the 16 AKT municipalities using three geographic and ethnic clusters. From these municipalities, the team selected individuals and organizations across five priority respondent strata: implementing partners, central government officials, municipal government officials, village/community councils, and local businesses. Sampling frames were provided by IPs in the form of detailed beneficiary lists, mapped against programmatic objectives and/or activities. The final sampling plan included a municipal official from each municipality, and two to five community and/or civil society beneficiaries to address the USAID focus on Objectives 1 and 3. The ET selected business representatives in four municipalities: two in the North, one Serb-majority municipality in the South, and one Albanian-majority municipality. The ET also selected national-level officials from the ministries with selection based on their involvement in or knowledge of AKT.

The second round was designed to deepen understanding of community-level implementation, dynamics and outcomes. The ET selected a sub-set of previously sampled municipalities to reflect: varying ethnic compositions, North-South geographic dispersion, and AKT implementation dimensions identified through first round interview notes. Within each municipality, the ET selected villages and attempted to speak with both Serb and Albanian community leaders in separate interviews. As detailed in Annex C, the ET completed the set of interviews in six villages. In addition to this core design element, second round data collection included four case studies in North Kosovo (three school-level quick impact projects and one Service Improvement Action Plan intervention at the municipal level), and interviewed one informal women’s group representative per municipality.

Across both rounds, the ET completed 61 interviews with 107 respondents (80 male, 27 female) through key informant interviews (KIIs) and group interviews (GIs). The detailed methodology and data collection tools are available in Annex B – Evaluation Design and Methodology.

The research design has four primary limitations. First, at the time of the interview, some respondents had not interacted with AKT for over a year. This elapsed time could lead to recall bias. Second, sampling frames were largely derived from IP beneficiary lists. While the ET used other sources to identify respondents, this primary sampling approach could have introduced some selection bias.

Specifically, interview selection could have been positively biased toward individuals with more interaction with the activity and overly represented male perspectives. The ET used a number of strategies to interview female beneficiaries, however but the gendered nature of community leadership and participation in Kosovo skewed participation, and consequently sampling frames, toward males.

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION viii

Third, the socio-political context of ethnic-relations in Kosovo is highly complex and charged. The team was expecting to encounter response bias, especially with regard to underreporting tensions, and tried a number of strategies to solicit truthful responses. Respondents were provided the option of meeting location, interviews were conducted in respondent mother-tongue whenever possible, and some interviews were conducted solely by local team members.

EVALUATION QUESTION 1 – TO WHAT EXTENT HAS THE PROJECT BEEN

EFFECTIVE?2

FINDINGS

Objective 1 Improving communication between Kosovo Albanians and non-majority communities

According to monitoring data, both AKT-US and AKT-LS met or exceed the majority of their performance targets. AKT-LS data indicate 3,360 direct beneficiaries and 99,471 indirect beneficiaries (targets = 2,500 and 90,000, respectively). AKT-US data register 24,276 community members as beneficiaries of small-scale infrastructure project and an additional 4,189 attendees at events designed to strengthening understanding and mitigating conflict between groups (targets = 21,266 and 1,240, respectively).

Respondents at the community and municipal levels were very satisfied with the professionalism, dedication, and efficacy of AKT staff. Working through local coordinators, AKT-US organized community meetings to identify and reach consensus on infrastructure needs. The model relied on broad-based community participation to foster inter-ethnic (and as a secondary consideration, cross-gender) communication and cooperation. The activity was successful in generating demand at the community level, with community forums generally well-attended. However, in many instances the lack of direct financial benefit to participants and the time required to participate in meetings resulted in significant attrition. The ET found one exception to equitable ethnic participation in a Serb-majority community where, according to the Serb community leader, AKT did not generate any Albanian participation.

In almost all interviews with community members, respondents described inter-ethnic relations in the community forums as cordial and equitable. The team found evidence of ethnic groups genuinely working together through AKT activities, though there were also a limited number of instances where ethnic tension may have been exacerbated. These included arguments over infrastructure priorities, some perceptions of inequitable benefits from infrastructure improvements, and an instance where an AKT-supported road was named in a way that offended a minority ethnic group. While these cases are worthy of attention, it is important to note that they are outliers in a process that solicited largely equitable participation from community members of different ethnic groups.

The activity clearly galvanized short-term communication and cooperation between ethnic groups in structured activities. There was some evidence that community-level impacts materialized through three different causal pathways. First, by working together toward a common goal in a structured environment, interpersonal bonds were created and strengthened between community members.

Second, multiple respondents brought up use of common spaces by different ethnic groups, particularly

2 The evaluation question formally prioritized AKT Objectives 1 & 3. We present results from these two objectives in the executive summary. Detailed findings from the other objectives can be found in the body of the report.

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION ix as they related to child and youth play. Third, some respondents reported low-level improvements in community relations. These sentiments were, however, outweighed by community responses indicating a lack of belief in lasting, significant impacts on inter-ethnic integration. These responses could be grouped in two categories. First, respondents claimed that inter-ethnic cooperation was positive before the activity came and, as such, was not affected any substantive way by AKT. Second, the intensity and duration of the intervention were not sufficient to affect such high-level and culturally-entrenched outcomes in a sustainable manner.

Improving communication between the Government of Kosovo and non-majority communities

AKT worked through municipal officials in all aspects of programming. In addition to municipal cost-sharing, municipal representatives participated in trainings, managed procurement for the infrastructure projects, participated in Joint Working Group (JWG) meetings, and led AKT-supported public budget hearings. These activities were intended to facilitate structured interaction between municipal officials and communities.

When asked about community-municipal relations before AKT, the prevailing sentiment across community respondents was that there was little-to-no interaction. The general sentiment of respondents was that municipalities had limited budgets and did not have much to offer to villages. Even in more affluent municipalities, however, respondents reported a generalized lack of engagement from local government. AKT galvanized increased interaction between the two stakeholder groups leading to some increased goodwill on the part of the community. A number of community respondents noted that AKT interventions were the only ones that got municipal officials to visit the communities, and the respondents appreciated the municipal officials’ presence. Other community representatives spoke about personal connections they made with municipal officials and how these connections have assisted them. While there were certainly cases of improved relations, the ET found evidence of some interactions leading to a worsening of perceptions toward local government. Some of this was driven by frustrations in the procurement process, where a number of community representatives reported feeling disempowered.

Achievements in North Kosovo

Due to concerns about potential corruption, the AKT model was modified for the North Kosovo context. In lieu of small-scale infrastructure investments, the activity completed ‘quick impact’ interventions. The intent was similar, though the prioritization process and procurement thresholds were more constrained. The ET interviewed stakeholders from three supported schools in Zubin Potok/Zubin Potok (two in Serb majority communities and one in an Albanian majority community). In the two Serb community schools supported by AKT-LS, the team found evidence of a robust consensus-building process between school management, teachers and parents association representatives. In contrast, the school in the Albanian community was more resource-constrained and received a less intensive engagement from AKT. According to a key informant, AKT only worked with one individual as opposed to a consensus-driven process with multiple community constituencies.

While these quick impact interventions deliver educational materials that will be used by many children, the fact that all schools in Zubin Potok/Zubin Potok are ethnically homogenous is a serious impediment to achieving the programmatic objective of increasing inter-ethnic cooperation. Not only are all benefits from these investments at the school level used exclusively by a single ethnic group, the fact that Serb schools received more financial support than the single Albanian majority school, the activity runs the risk of perpetuating perceptions of inequality.

Sustainability

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION x

Benefits stemming from physical investments funded by AKT, particularly small-scale infrastructure interventions, are likely to persist for years. These investments seem to have been well-constructed and well-utilized. Quick impact interventions in North Kosovo will also persist, though asset depreciation should be expected to be greater in school environments.

Community mobilization through formalized village-level meetings seem very unlikely to persist beyond the life of the activity. There was no evidence of any spontaneous, structured meetings not affiliated with AKT. Similarly, it is highly unlikely that the gains in community-municipality communication affected by AKT will persist. While it’s possible that some aspects of community outreach will continue, without external financing municipal governments will most likely revert to pre-intervention behavior.

Lastly, the prospects for sustaining advances made in the overall programmatic objective of bringing majority and minority ethnic groups together are not promising. It is highly likely that community members will continue using infrastructure financed by AKT and that this common use will facilitate some small-scale interactions (e.g. children playing together). It is also possible that interpersonal relationships formed through participation in the activity will persist. However, because the activity was not successful in substantively affecting community-level changes in ethnic cooperation, despite achieving and sometimes exceeding output targets, it is highly unlikely for these benefits to sustain.

Objective 3

Enhance capacity of municipal administrations for strategic planning and to deliver and oversee quality service

AKT supported municipal governments through a number of complementary activities. In the first year of the activity, AKT facilitated self-assessments of municipal capacity for each of the 16 targeted municipalities. On the basis of these documents, municipalities drafted Capacity Development Plans (CDPs) to strengthen perceived weaknesses. In conjunction with this process, municipalities developed Service Improvement Action Plans (SIAPs) to identify and co-finance investments that would improve municipal service delivery and received various trainings. These plans included a variety of interventions supporting energy efficiency, public lighting, waste management, primary health care, education (university and preschool), and cultural centers. Performance monitoring data indicate that an estimated 30,267 people are receiving improved service as a result of these interventions, with almost 80 percent of estimated beneficiaries Serb. Additionally, municipal officials interviewed by the ET reported the development of SIAPs as a positive step in improving service delivery and for planning economic development.

AKT provided training to municipal workers on a wide range of topics, including: public procurement, strategic planning, human resource management, budgeting, project management, environment protection, public transparency, democratization, gender and human rights. According to the most up-to-date performance monitoring data the ET could acquire, between the two mechanisms AKT trained a total of 2,311 people with the goal of strengthening local government. This figure represents a 38 percent overachievement relative to the performance target of 1,680. Field interviews with municipal representatives who attended these trainings suggested a significant divergence in satisfaction. Positive feedback included one respondent attributing improved service delivery to AKT capacity building activities, an official reporting that the municipality was operating without a human resources person until one was trained by AKT-US, and multiple respondents requesting additional training.

Feedback was not uniformly favorable, however. Criticisms of the AKT capacity building approach centered on four factors. A number of respondents spoke of heavy workloads that inhibited the ability of municipal workers to participate. Secondly, some respondents spoke of a generalized training fatigue among municipal officials. Third, a number of respondents expressed a perception that AKT trainings did not help them perform their jobs better. Finally, although few respondents mentioned translation issues

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION xi for different trainings across objectives, the ET registered one complaint about an Albanian AKT trainer being sent to North Kosovo to train municipal officials.

Achievements in North Kosovo

Northern municipalities face unique political dynamics and staffing shortages and participated less than other municipalities in AKT programming. Respondents from these municipalities identified more pronounced capacity-building needs, including difficulty filling job vacancies, training newly-hired and seasoned staff, learning and adjusting to Kosovo law, and building more effective relations with central institutions. Municipal officials noted that due to staffing gaps, the municipality cannot always participate in trainings that are offered.

The ET investigated one SIAP intervention in North Kosovo. The plan resulted in the replacement of a boiler to improve efficiency of heating the municipal building and an adjacent cultural center. While the process of consensus building, procurement preparation and management may have increased competencies of those involved, benefits of the investment seemed to accrue primarily to municipal workers. According to the director of the cultural center, the improved heating has improved the ability of the municipality to deliver cultural programming but only ethnic Serbs use the facility.

Sustainability

The physical investments made through SIAPs should be expected to persist for years and provide services to municipal residents. The plans themselves were designed to be actionable for four years and the engagement of community forum members to monitor activities could strengthen demand-side pressures. If municipal officials continue to follow through on them, the design feature should promote sustainability of capacitation efforts. However, many municipal respondents expressed skepticism that the momentum will continue absent AKT. Because municipalities are operating with limited funds, their ability to serve their constituencies is inhibited. In this context, capacity building can only do so much toward improving service provision.

CONCLUSIONS

Objective 1

• AKT was largely successful in fostering community participation from different ethnic groups.

Representatives of these groups collaborated to identify community needs in a broadly equitable manner, and infrastructure improvements were delivered to the satisfaction of communities.

• The transition from decision-making at the community forum level to procurement management at the municipal level was viewed by some community members as opaque, disenfranchising and, in some instances, disrespectful.

• There were indications of small-scale improvements in inter-ethnic relations, particularly from strengthened/created interpersonal relationships among participants, common usage of infrastructure funded by AKT (especially spaces for children and youth), and low-level trust-building through the process of working on structured interventions. However, the intervention was not sufficiently intensive, neither in duration or contact time, to galvanize substantive community-level impacts.

• The activity succeeded in fostering communication between community and municipal representatives. Community perceptions of these interactions and their opinion of municipal leadership varied widely. Some respondents spoke of improved relations, some of worsened relations, though most reported not changing their perception. Even in instances where cooperation improved, these changes were relatively small and are unlikely to affect inter-ethnic cooperation at a community-level.

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION xii

• Infrastructure investments will persist for many years after AKT closes out. However, improvements in community communication and collaboration should be expected to atrophy absent any future, structured engagements.

• AKT’s activities in North Kosovo will likely improve the quality of education for participating schools. However, due to the fact that all schools in Zubin Potok/Zubin Potok are ethnically homogenous, these investments will not have any substantive impact on inter-ethnic communication or collaboration. Furthermore, because Serb-majority schools received far more aid than the Albanian-majority school, the activity runs the risk of exacerbating inter-ethnic tensions.

Objective 3

• AKT was largely successful in meeting its performance targets under objective 3. All 16 targeted municipalities received CDPs and successfully drafted SIAPs.

• The successful engagement and cost-sharing with the municipalities in excess of $600 thousand should be viewed as a success. While it was beyond the scope of this evaluation to independently verify beneficiary estimates, AKT claims these investments will benefit over 30,000 people.

• AKT provided 2,353 people a wide variety of trainings. Some respondents reported deep satisfaction with these sessions, particularly when trainings directly affected job functions (e.g.

procurement certification). However, the ET found evidence of trainings being underattended due to heavy workloads, training fatigue, and, in some cases, perceptions that offerings were not useful.

• Overall, despite the improvements in municipal capacity, most municipal officials themselves— plus central-level officials, community and business respondents, and other knowledgeable experts interviewed—noted that municipalities will need to improve service delivery to address citizen needs.

EVALUATION QUESTION 2 - WHAT ARE THE KEY LESSONS LEARNED FROM THE

PROJECT TO DATE?

Theory of Change

While both mechanisms achieved impressive results from an output perspective, AKT did not seem to affect substantive changes in the programmatic goal of increasing inter-ethnic cooperation. Field data indicate that implementation intensity was insufficient to affect the ambitious programmatic objectives due to an ambitious program design and a relatively short 3-year timeframe.

Targeting

The selection of ethnically homogenous schools in North Kosovo or villages that are practically ethnically homogenous in South Kosovo is misaligned with the program theory of change. For activities working on improving inter-ethnic cooperation, there should be some benchmark of minimum participation to be considered for support.

Community Leaders

The efficacy of activities like AKT depends largely on the community leaders. Successful leaders are the product of a number of factors. They must have (1) time to engage with work on behalf of their community, the (2) capacity to represent their community, (3) be proactive in advocating for community interests, and (4) command the respect of their communities.

Elite Capture

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION xiii

Activities relying on disbursement of goods or services through participatory means will always have to cope with stakeholders attempting to divert investments in personally beneficial means. While the evaluation did not find evidence of corruption, the ET documented three cases of potential elite capture.

The first example revolved around the selection of schools for receipt of AKT support in Zubin Potok/Zubin Potok, the second example concerned municipal officials receiving the majority of programmatic benefits, and the third was about a mayor potentially subverting a democratic process to select an infrastructure priority.

Inter-ethnic Cooperation through Work

Interview data across all respondent types evidenced a consistent, shared belief that the best conduit toward true inter-ethnic cooperation was through shared economic interests. Taken in conjunction with the finding that gains in inter-ethnic cooperation are likely to atrophy without structured activities to bring people from different communities together, the notion that a long-term, shared engagement through work seems like an approach that would promote inter-ethnic cooperation and provide spillover benefits with respect to economic growth.

RECOMMENDATIONS

• The ET recommends significantly simplifying programming, particularly in this complex socio-political context. AKT implementers were engaged in too many disparate activities over too short a time to focus programming and affect substantive changes.

• Inter-ethnic, community-level programming in Kosovo has to be highly contextualized. To the extent possible, USAID should design implementation mechanisms with the flexibility necessary for meaningful adaptive management. This could take the form of a trial-and-error process where different approaches are piloted and, if successful, scaled. For this approach to work, a more robust monitoring and evaluation (M&E) system is necessary.

• If USAID is considering a co-implemented activity in the future, the Agency should conduct more thorough planning before procuring either mechanism. Ideally, both mechanisms should be designed in advance of procurement, with clearly delineated scopes (either geographically or sectorally). Additionally, both procurements should make reference to the other mechanism and explain in what ways the two contracts are expected to cooperate. These signals will result in more realistic budgeting during the procurement phase and a more efficient start-up process.

• If USAID plans on implementing similar programming in the future, eligibility criteria for communities should be strengthened to increase the likelihood of affecting community-level outcomes and increasing the cost-effectiveness of USG programming. The ET recommends two specific factors to consider in setting community minimum eligibility screening: communities should (1) have a minimum number of ethnic minority residents to participate, (2) register a minimum level of participation from targeted ethnic groups for programming to continue.

• Given the broad consensus from respondents at all stakeholder levels, USAID should continue to incorporate job creation activities in this type of programming.

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION xiv

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 1

PROJECT BACKGROUND

PROGRAM BACKGROUND AND DESCRIPTION

Attempting to reduce tension in the North, the Government of Kosovo (GoK) and Government of Serbia (GoS) entered into a European Union (EU)–facilitated Brussels Agreement of 2013.3 This agreement outlined key principles toward sustainable peace and integration, including how to address local governance issues such as police, judiciary, and local elections. Though municipalities still encounter challenges in delivering essential services and meeting the GoK’s target commitments related to these local governance issues, implementation of the agreement is in progress. The Northern Serb population is starting to show signs of increased cooperation with other parts of the country and is following Belgrade institutions less strictly. Paradoxically, this relaxation has created new challenges; any move toward political compromise between Kosovo Serbs and Kosovo Albanians is viewed as undesirable among both Albanian and Serb nationalists.

Due to past conflicts, relations between the Kosovo-Albanian majority and the Kosovo-Serb minority population remain fragile. Kosovo Serbs and other minorities report lingering difficulties, e.g. in accessing markets and value chains dominated by the majority community, and in receiving public services. Kosovo Serbs form a majority in ten municipalities. Four of these municipalities in the North used to operate under direct control of Belgrade after the war of 1998–99. The GoK has only recently started exercising control over the North as a result of the 2013 Agreement. The other six Serb-majority municipalities in the rest of the country are often regarded as enclaves, although they are considered more structurally integrated into the Kosovo system than the municipalities in the North.

Under the Brussels Agreement, Serbia agreed to dismantle its parallel institutions in Kosovo. These parallel institutions still employ thousands of Kosovo Serbs, even if many employees draw a salary without doing actual work. In turn, Kosovo agreed to create an Association/Community of Serb- Majority Municipalities (ASM). The integration of the parallel institutions is viewed as an opportunity for the development of Kosovo. ASM is seen to represent a central Kosovo Serb entity, which would not only channel the assistance/subsidies of Belgrade but also represent an umbrella organization that would serve as an intermediary between territorially and politically divided Serb municipalities. The health and education sectors are foreseen to be managed by the ASM, and further subsidized by Belgrade.

Though the GoK continues to make progress toward political and economic stability, it also faces a number of key challenges regarding the integration of non-majority populations into Kosovo’s institutions and society. Advancing Kosovo Together (AKT) is a USAID-funded activity aiming to increase constructive inter-ethnic cooperation and interaction between Kosovo’s majority and minority populations, which is vital to the security and stability of the region.4 AKT is implemented through two complementary mechanisms—AKT Prime (also known as AKT-US and implemented by Chemonics International) and AKT Local Solution (AKT-LS, implemented by Community Development Fund, or CDF). AKT-US began in April 2014 and concluded in April 2017 (Award Number: AID-167-C-14- 00003, Award Amount: $7,587,654), and AKT-LS began in October 2014 and will conclude in March 2018, following a no-cost extension that was granted at the time of this evaluation (Award Number:

AID-167-A-14-00008, Award Amount: $6,999,706). See Annex A – Statement of Work for more

3 The Brussels Agreement was made between the governments of Serbia and Kosovo in 2013, with the goal of normalizing relations between the two countries. The agreement process calls for the parties to sign particular sections of the agreement as those issues are agreed to.

4 https://www.usaid.gov/news-information/fact-sheets/advancing-kosovo-together-akt

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 2

information on the two mechanisms. Additionally, both AKT-US and AKT-LS have three subcontractors each. The subcontractors for AKT-US are: Business Support Center Kosovo, Development Professionals Inc. and the Berman Group. The subcontractors for AKT-LS are: Kosovo Relief Development (KRD), Centre for Peace and Tolerance, and AKTIV NGO.

AKT’s theory of change for both mechanisms posits that increased constructive inter-ethnic cooperation and interaction will be achieved as a result of:

1. Improved autonomous, horizontal communication between: a) Kosovo Albanian and non-majority communities; and b) the GoK and non-majority communities (Objective 1)

2. Improved economic opportunities in target municipalities (Objective 2)

3. Increased efficiency and capacity of target municipal administrations to respond to the needs of all their citizens (Objective 3)

Within USAID/Kosovo’s Country Development Cooperation Strategy, AKT is intended to contribute towards Intermediate Result (IR) 1.3, “Improved integration of ethnic minorities into Kosovo society,” which, in turn, supports Development Objective 1, “Improved rule of law that meets citizen needs.”

Figure 1: AKT Results Framework

In addition to the three shared objectives mentioned above, AKT-US was intended to provide capacity building to the AKT-LS implementer, and thus, includes an additional and separate Objective 4: “Build the capacity of host country organizations by leveraging local, cross-ethnic leadership to implement program objectives 1, 2, and 3.” This implementation structure also supports USAID Forward and efforts to promote sustainability at a local level.

AKT implements activities across 16 Kosovo partner municipalities, including six Serb-majority municipalities in the south, six Albanian-majority municipalities, and four municipalities in northern

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 3

Kosovo (See Annex F for list of AKT municipalities). AKT works across three distinct implementation components, namely: 1) municipal development; 2) business development; and 3) citizen engagement.5 Table 1 outlines the major implementation components within AKT.

Table 1: Implementation Components within AKT

TECHNICAL

AREA

ACTIVITIES

Municipal Development

Support GoK in implementation of laws that protect minority rights Improve government communications, responsiveness, and engagement with non-majority communities Build skills of minorities and women to obtain public-sector positions Provide constructive engagement through non-infrastructure and small-scale infrastructure projects Enhance capacity of municipal administrations for strategic planning and to deliver and oversee quality services Support municipalities to increase own-source revenue for municipal services and capital projects Build capacity of Municipal Offices of Communities and Returns Strengthen municipal capacity to manage government-to-government awards

Business Development

Assess and facilitate market linkages between majority and non-majority businesses Promote small and medium enterprise (SME) competitiveness and access to finance Support youth employability Support women-owned enterprises

Citizen Engagement

Strengthen mechanisms to promote transparency and community engagement Build the capacity of host-country organizations by leveraging local, cross-ethnic leadership to implement program objectives Facilitate community-led initiatives among majority and minority communities to address common issues Increase CSO capacity to preserve and advocate for minority rights

5 AKT-LS Year 1 Annual Report

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 4

EVALUATION PURPOSE AND EVALUATION QUESTIONS

EVALUATION PURPOSE

This report is a performance evaluation of the Advancing Kosovo Together (AKT) activity, inclusive of its two mechanisms, AKT-US and AKT-LS.

Considerable time elapsed between the drafting of the original Statement of Work (SOW) for the evaluation and the conduct of the evaluation itself. Importantly, AKT was nearing its conclusion when the evaluation team (ET) was mobilized. The purpose of the evaluation, as articulated by the USAID/Kosovo Mission during the in-brief in February 2017,6 was to:

1. Inform the 2018-2023 USAID/Kosovo Country Development Cooperation Strategy (CDCS) and future programing; and

2. Know whether the AKT engagement—focusing on minority community integration—has been working.

The above two points help to refine the purpose as outlined in the evaluation SOW:7

1. Evaluate the programmatic progress of the AKT program and its two separate implementing mechanisms—AKT-US Prime and AKT-LS;

2. Identify strengths and weaknesses, and describe the effectiveness of the project;

3. Provide key findings, conclusions, and recommendations for the two AKT activities; and

4. Provide recommendations for future activities.

The primary target audience for the evaluation is USAID/Kosovo, AKT IPs, and their partners. Other key stakeholders include other US Government (USG) entities and international donors supporting ethnic minority integration in Kosovo, the GoK, particularly local government agencies such as the Ministry of Local Government Administration, Ministry of Communities and Returns, and Ministry of Finance, and municipal government officials.

EVALUATION QUESTIONS

The evaluation had two main evaluation questions, each with sub-questions.

1. To what extent has the project been effective?

1.1 Has the program achieved its intended results, especially under Objectives 1 and 3? Why or why not?

1.1a What are AKT achievements in North Kosovo to date? What worked and what did not?

1.2 How has the sustainability of USAID investments been addressed?

2. What are the key lessons learned from the project to date?

2.1 Should there be any programmatic shifts to the AKT program to achieve better results? Are there new opportunities for minority integration that future programming could address?

2.2 Is there a potential for greater AKT program collaboration with other USAID programs and/or other donors?

6 See meeting notes, AKT Evaluation Team In-brief with USAID/Kosovo, February 21, 2017.

7 See Modification 1 to Evaluation SOW.

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 5

EVALUATION METHODS AND LIMITATIONS

METHODS

The evaluation was implemented between January 17 and July 14, 2017, in three phases:

PHASE 1: EVALUATION PREPARATION AND DESIGN

Upon contract award, the Evaluation Team (ET) conducted a team planning meeting, consultations with USAID/Kosovo and AKT implementing partners, a desk review of activity documentation and secondary data, sample design, instrument design, and field protocol development. The last three items were formalized in an Inception Report submitted to USAID. The evaluation design incorporated a utilization-focused and mixed-methods approach with consideration given to ethnicity and gender. SI integrated quality assurance (QA) through all phases of design and implementation through its proven EQUI® (Evaluation Quality, Use, and Impact) approach, while integrating gender and social analysis at each stage.

PHASE 2: DATA COLLECTION

Primary data was collected through two rounds of data collection. In both rounds, the ET held in-briefs and out-briefs with USAID. The first round was based on multi-stage sampling, where the ET purposively selected 10 out of the 16 AKT municipalities using three geographic and ethnic clusters:

Cluster I consisted of six Serb-majority municipalities in the South; Cluster II consisted of the six Albanian-majority municipalities; and Cluster III consisted of four Serb-majority municipalities in the North. From these municipalities, the team selected individuals and organizations across five priority respondent strata: IPs, central government officals, municipal government officials, village/community councils, and local businesses. Sampling frames were provided by IPs in the form of detailed beneficiary lists mapped against programmatic objectives and/or activities. In addition to IP-supplied lists, USAID/Kosovo recommended knowledgeable experts to be consulted. The final sampling plan included a municipal official from each municipality, and two to five community and/or civil society beneficiaries to address the USAID focus on Objectives 1 and 3. The ET selected business representatives in four municipalities: two in the North, one Serb-majority municipality in the South, and one Albanian-majority municipality. The ET also selected national-level officials from the ministries with selection based on their involvement in or knowledge of AKT.

The second round was designed to deepen understanding of community-level implementation, dynamics and outcomes. The ET selected a sub-set of previously sampled municipalities to reflect (1) varying ethnic compositions, (2) North-South geographic dispersion, and (3) AKT implementation dimensions identified through first round interview notes. Within each municipality, the ET selected villages and attempted to speak with both Serb and Albanian community leaders in separate interviews. As detailed in Annex C, the ET completed the set of interviews in six villages. In addition to this core design element, second round data collection included four case studies in North Kosovo (three school-level quick impact interventions and one Service Improvement Action Plan intervention at the municipal level), and interviewed one informal women’s group representative per municipality.

Across both rounds, the ET completed a total of 61 interviews with 107 respondents (80 male, 27 female) through key informant interviews (KIIs) and group interviews (GIs). The breakdown by respondent type is presented in Figure 3 below. The ET was careful to schedule data collection in locations where, and during times when, both women and men could participate. The detailed methodology and data collection tools are available in Annex B – Evaluation Design and Methodology.

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 6

Figure 2: Data Collection Sites by Municipality

Figure 3: Respondent Groups8

8 The graph shows 65 respondents in total; 4 of which were repeat interviews.

Zubin Potoku / Zubin Potok

Vushtrri / Vučitrn

Obiliq / Obilić

Prishtina / Priština

Gračanica / Graçanicë

Shtërpcë / Štrpce

Zveçan / Zvečan

Mitrovicë / Mitrovica

Leposaviq / Leposavic

Kllokot / Klokot

Gijlan / Gnjilane

USAID ADVANCING KOSOVO TOGETHER – FINAL PERFORMANCE EVALUATION 7

PHASE 3: ANALYSIS

Upon return from the field, the ET performed open coding on interview notes. Once a clear set of thematic codes emerged, the team closed-coded the 61 interview notes. In addition to overt focus on addressing evaluation questions, key analytical aspects included differential implementation and impacts across key respondent groups, geographic variance in implementation and efficacy, and differences between AKT-US and AKT-LS. This primary data was combined with activity documentation and secondary data to triangulate and deepen evaluation findings. The ET drafted a findings, conclusions and recommendations matrix, and integrated findings from the various data sources on an iterative basis.

This matrix became the empirical basis of the evaluation report.

LIMITATIONS

The research design has four primary limitations. First, at the time of the interview, some respondents had not interacted with AKT for over a year. This elapsed time could lead to recall bias. Second, sampling frames were largely derived from IP beneficiary lists. While the ET used other sources to identify respondents, this primary sampling approach could have introduced some selection bias.

Specifically, interview selection could have been positively biased toward individuals with more interaction with the activity, and overly represented male perspectives. The ET used a number of strategies to interview female beneficiaries; however, the gendered nature of community leadership and participation in Kosovo skewed participation, and the resultant sampling frames, toward males. Third, the socio-political context of ethnic relations in Kosovo is highly complex and charged. The team expected to encounter response bias, especially regarding underreporting tensions, and tried a number of strategies to solicit truthful responses. Respondents were provided options for meeting locations, and interviews were conducted in respondent mother-tongue whenever possible, with some interviews conducted solely by local team members.

USAID ADVANCING KOSOVO TOGETHER…