Attachment J.4 - EDGE Evaluation Report.docx

MID-TERM PERFORMANCE EVALUATION OF USAID/EUROPE EURASIA BUREAU ECONOMIC DEVELOPMENT, GOVERNANCE AND ENTERPRISE GROWTH IN EUROPE AND EURASIA (EDGE)

SEPTEMBER 2023

This publication is made possible by the support of the American People through the United States Agency for International Development (USAID) and was prepared by Integra Government Services International LLC for the Europe and Eurasia / Monitoring, Evaluation, Learning, and Decision Support (EE/MELDS) Activity.

MID-TERM EVALUATION OF USAID/EE ECONOMIC DEVELOPMENT, GOVERNANCE, AND ENTERPRISE GROWTH IN EUROPE AND EURASIA (EDGE)

Washington, DC 20005

This publication is made possible by the support of the American People through the United States Agency for International Development (USAID) and was prepared by Integra Government Services International LLC for the Europe and Eurasia / Monitoring, Evaluation, Learning, and Decision Support (EE/MELDS) Activity.

SYNOPSIS

Program purpose and description: To assess the efficiency and effectiveness of the Economic Development, Governance and Enterprise Growth (EDGE) in Europe and Eurasia project and present recommendations regarding future design(s) of regional economic growth activities in the region.

Key questions: Evaluation Question (EQ)1: To what extent has the project proved successful in assisting companies integrated into a) regional value chains and b) global value chains? Objectives and targets? EQ2: What lessons were learned from modifying the implementation of interventions during COVID-19 that can be used in future design considerations? EQ3: What are the strengths and weaknesses of the EDGE management structure? EQ4: To what extent did the leverage requirement contribute to the sustainability of activities? EQ5: To what extent are EDGE activities supporting countries in meeting and fulfilling their European Union (EU) accession criteria?

Methodology: The final evaluation applied a mixed methods design. Key methods used to answer five EQs included key informant interviews (KIIs), quantitative survey, performance monitoring data, and a desk review. Two validation workshops were held with USAID/EE and Missions.

Data Collected: The desk review included 36 documents. This was supplemented by 84 KIIs in Georgia (13) Kosovo (15), Moldova (19), North Macedonia (22) and Washington, DC (15). Fewer than 10 respondents participated in the quantitative survey despite repeated outreach efforts.

Challenges and mitigation strategies: The evaluation did not use statistically representative samples but applied a purposive sampling strategy to engage key stakeholders to draw high-level representative findings to answer the EQs, which was a limitation.

Key findings: EDGE has demonstrated variously effective strengths in its approaches to enhancing regional economic growth and competitiveness; About 83 percent of the nearly 3,000 participants reported to EDGE that they have applied the knowledge and skills acquired from EDGE training in their work, and 93 percent of participants stated in feedback questionnaires that the training was useful. EDGE has proven to be an efficient rapid response and bridging mechanism, but it duplicates some bilateral programs. As of May 2023, EDGE had fully executed 61 grants in its core and buy-in components for a total value of $3,115,476. EDGE technical assistance indirectly supports and links to each country’s EU accession requirements and key aspects of the accession criteria but cannot be measured in the short term.

Key recommendations: Continue and expand technical assistance to associations and organizations experienced in certifying firms to meet emerging the requirements of the EU and private brands for the energy efficiency, waste management, and environmental compliance of products and services; address logistics and internal transportation challenges; incorporate training taxonomy and digital data collection instruments within the Monitoring, Evaluation, and Learning system; use EDGE grants and training to fill gaps in bilateral programming and hew more closely to country and regional-level private sector engagement strategies; refine EDGE grant management processes and better address the needs of grantees; and adapt continuously to maximize the program’s impact for countries pursuing EU integration by meeting accession criteria.

TABLE OF CONTENTS

LIST OF TABLES

LIST OF FIGURES

ACRONYM LIST

ACKNOWLEDGEMENTS

The authors would like to acknowledge and thank the numerous stakeholders and beneficiaries of the Economic Development, Governance and Enterprise Growth in Europe and Eurasia (EDGE) project, particularly those who shared their time and experiences during remote data collection and during the preparation of this paper. The authors would like to thank USAID/Europe and Eurasia (E&E) Deputy Division Chief Sharon Valentine and David Tsiklauri (USAID/Georgia), Joshua Mike (USAID/Kosovo), Irina Nasalciuc (USAID/Moldova), Dawn Carmin and Rozalija Karchicka Vasilevska (USAID/North Macedonia), USAID/E&E EDGE Contracting Officer Representative (COR) Stephen Little, and MELDS COR Kraemer Lovelace, who contributed time and insights to the evaluation.

We thank EDGE Chief of Party Jasminka Varnalieva and other staff for providing information and support to this evaluation. The authors would like to acknowledge the entire EDGE team in Washington and the field for their time and efforts in providing project documents, extensive feedback, and support during this evaluation.

GS-10F-083CA / 720018M00013

Europe and Eurasia Monitoring, Evaluation, and Learning for Decision Support (EE/MELDS) Integra Government Services International LLC

Our team was composed of Team Leader Brenda L. Pearson, Senior Data Analyst and Georgia Subject Matter Expert Maia Giorbelidze, Kosovo Subject Matter Expert Arben Qirezi, Moldova Subject Matter Expert Silvia Morgoci, and North Macedonia Subject Matter Expert Vlatko Danilov. Also, we acknowledge the support of our home-office staff, Ganyapak (Pin) Thanesnant, EE/MELDS Project Officer, and Penelope Norton, Senior Associate.

EXECUTIVE SUMMARY

BACKGROUND AND METHODOLOGY

The United States Agency for International Development (USAID)/Europe and Eurasia Bureau (EE) requested that the Monitoring, Evaluation, Learning, and Decision Support (MELDS) team conduct a performance evaluation of the Economic Development, Governance and Enterprise Growth in Europe and Eurasia (EDGE) project implemented by International Development Group (IDG). The evaluation assesses the performance of ongoing activities in four countries: Georgia, Kosovo, Moldova, and North Macedonia. This evaluation sought to answer five evaluation questions (EQs). It provides key findings aligned to these questions and a set of actionable recommendations that can inform any adjustments that USAID/EE makes going forward.

METHODOLOGY

The purpose of the evaluation is to answer five EQs focused on EDGE implementation achievements related to the project’s effectiveness, efficiency, sustainability, and contributions to the Euro-Atlantic integration of the selected countries. The evaluation team used a mixed-methods approach that combined qualitative, in-depth remote, and in-person interviews with key stakeholders and local partners. All proposed semi-structured interviews and group discussions were organized around the EQs and supported with detailed instruments. The team developed each tool for a specific group of interviewees and mixed common questions and other questions unique to a particular group to obtain a full range of opinions regarding specific projects and to ensure that data was comparable across all respondent groups. In addition, the evaluation team carried out a desk review and analysis of performance monitoring data. Many of the baseline assumptions and market analyses that informed the design in 2018 are no longer valid due to the COVID-19 pandemic, war in Ukraine, and newly increased funding levels of bilateral Missions in the region. This evaluation focuses on the period of performance from July 29, 2019, through July 2023.

PROGRAM FINDINGS AND RECOMMENDATIONS

Overall, the evaluation team found that most EDGE capacity building and training activities were properly designed and relevant, delivering assistance aligned with local partners’ needs. The team also found EDGE coordination efforts to be effective, with demonstrated evidence of how the project worked with some bilateral USAID economic growth activities and other donors to address the policy and regulatory barriers that its priority sectors and value chains face. EDGE’s current modes of implementation may require revision in response to changed economic circumstances inherent in regional disruptions due to war and new security challenges. It may be necessary to revisit the selection criteria for future buy-ins for follow-on programming once USAID completes its new design, which will focus more on addressing systemic gaps or market failures than supporting individual firms. Similarly, grant solicitation themes may need to shift to a greater focus on automation, climate-smart agriculture, digitization, internal logistics, and more targeted investments in light manufacturing subsectors in anticipation of reconstruction and infrastructure needs in the region.

USAID investments across three value chains (fruits and vegetables, wood processing and textiles, and tourism) may benefit from more pilots that involve regional public sector organizations, private sector partners, and associations. In general, EDGE has been efficient in achieving its goals and creating a supportive ecosystem for established companies that seek to increase the volume of exports. To varying degrees, each of the 61 current grants has successfully contributed to reducing barriers to cross-border trade and investment, improved business sophistication and market integration, and expanded market linkages. More detailed findings are outlined below and discussed throughout this report.

KEY FINDINGS AND RECOMMENDATIONS

Based on the evaluation findings and consultations with USAID/EE, the team organized the key findings under four main themes: 1) effectiveness, 2) efficiency, 3) sustainability, and 4) Euro-Atlantic Integration.

TABLE 1: KEY FINDINGS AND RECOMMENDATIONS

EQ1: EFFECTIVENESS OF IMPLEMENTATION

EDGE has demonstrated various strengths in its approaches to enhancing regional economic growth and competitiveness.

EDGE has provided training and capacity building support to individuals and firms to advance their integration into regional and global value chains.

EDGE support was most useful in subsidizing participation in trade shows and other venues.

Business service providers (BSPs) said their participation in EDGE training events was good for networking and gaining insights into financing issues in the targeted value chains.

EDGE has been active in providing firm level support to small and medium-sized enterprises (SMEs) in the region to obtain export certifications and fostering best practices in quality control and management.

Ten SMEs received EDGE grants under contract (GUCs), which they utilized to improve financing, human resources, and automated production technology to engage in larger-scale production.

EDGE training and mentoring indirectly assisted SMEs in countering investors’ perceptions of high risks in the region and concerns about consistent quality of products.

Demand for organic products is increasing in regional and global markets, and the need for certification of organic products is increasing.

Support for regional tourism received the most mixed assessments from key stakeholders, who stated that national policies, investment by local government partners, and higher levels of customer services are necessary prerequisites for scaling any activities.

Continue and expand technical assistance to associations and organizations that are experienced in certifying firms to meet emerging the requirements of the EU and private brands for the energy efficiency, waste management, and environmental compliance of products and services.

Address logistics and internal transportation challenges by prioritizing the identification and resolution of logistics and internal transportation challenges that impede competitiveness.

Strengthen government partnerships, because they play critical roles in facilitating linkages to multilateral structures and addressing political challenges that hinder cross-border movement of goods and services.

EQ2: EFFECTIVENESS OF DELIVERY MODALITIES

About 83 percent of the nearly 3,000 participants reported to EDGE that they have been applying the knowledge and skills acquired from training in their work, and 93 percent of participants stated in feedback questionnaires that the training was useful.

Opinions favoring in-person or online training varied among stakeholders in different regions and value chains.

Women and non-urban populations, in particular, reported that their self-described low levels of digital literacy and technological proficiency made online learning events and presentations less effective for them.

Participants reported that many training instructors were unfamiliar with delivering online training and struggled to adapt their curricula and create interactive learning experiences.

Incorporate training taxonomy and digital data collection instruments within the EDGE monitoring, evaluation, and learning (MEL) system and develop a hierarchical training taxonomy that categorizes and classifies training programs based on their characteristics, objectives, and content.

Establish a structured legacy archive through a systematic approach to curating training and learning materials, recordings, and photo galleries of trade fairs and exhibitions. Select a suitable digital platform or system for storage and accessibility.

EQ3: EFFICIENCY OF THE EDGE MANAGEMENT STRUCTURE

EDGE has proven to be an efficient rapid response and bridging mechanism, according to USAID/Moldova.

EDGE activities should better complement ongoing USAID bilateral activities and include the leadership of flagship economic growth programs in work planning and learning events.

Two part-time EDGE subregional representatives liaise with Missions in the Caucuses, Ukraine and Moldova to advance cooperation and identify partnering opportunities.

The EDGE management structure is more effective in providing a range of technical support at the firm level rather than implementing a systems approach, according to USAID Mission staff and local partners. The number of active partners collaborating across border is difficult to assess at the output and outcome levels.

Donor partners characterized collaboration with EDGE as positive and useful in avoiding duplication of activities that support customs agencies and value chains.

The EDGE team does not reflect the diversity of the E&E region and focus countries.

Grantees raised concerns regarding the EDGE team’s competency in managing grants.

The EDGE Activity Monitoring, Evaluation, and Learning Plan (AMELP) could be better utilized to capture lessons and feedback from partners so best practices and innovations are scaled and replicated in GUCs and training.

EDGE should consult with other USAID economic growth programs to ensure that EDGE grants and training are complimentary and contribute to regional-level private sector engagement strategies.

EDGE’s grants procedures and AMELP output indicators and targets should be reviewed to ensure that important internal collaborating, learning, and adapting (CLA) opportunities are being identified to generate important lessons and promising practices for the region.

EDGE should be inclusive in its hiring to reflect the diverse needs, cultural sensitivities, and contextual issues in the region.

EQ4: INFLUENCE OF GRANTS’ LEVERAGE REQUIREMENT ON SUSTAINABILITY

Only 16 grants in the core component had been fully executed as of May 2023, although the EDGE work plan anticipated issuing about 30 grants during its period of performance. The buy-in component had executed 45 small grants in the total amount of $1.4 million.

Most grantees, local partners, and USAID staff viewed EDGE’s leverage requirements positively. The cost share requirement compelled grantees to think creatively about resources and seek collaborations that they might not have considered otherwise.

Stakeholder feedback indicated that EDGE grants facilitated the creation of connections that might not have occurred organically and are likely to generate sustainable partnerships.

EDGE should refine its grant management processes and better address the needs of grantees. The leverage requirement ratio of cost-sharing should be adjusted for different types of stakeholder groups.

EQ5: SUPPORTING INTRA-REGIONAL AND EURO-ATLANTIC INTEGRATION

In the short term, it is difficult to assess the relevance of technical support provided by EDGE activities in enabling Georgia, Kosovo, Moldova, and North Macedonia to meet their EU accession.

EDGE technical assistance indirectly supports and links to each countries’ EU accession requirements and key aspects of the accession criteria.

EDGE activities in support of relevant trade associations such as the Central Europe Free Trade Agreement (CEFTA), national customs agencies, and inputs to the revision of the Multi-Annual Action Plan on Regional Economic Area in the Western Balkans (MAP REA) are perceived as important policy-oriented investments.

Obtaining relevant export certificates such as Global Gap is an important achievement because it demonstrates the producers’ compliance with international standards for exports to EU countries.

EDGE’s facilitation of the introduction of the Authorized Economic Operator (AEO) program helped Georgia, Kosovo, Moldova, and North Macedonia progress toward meeting their EU accession criteria.

The effectiveness of EDGE’s activities has been tempered by significant political and regulatory challenges that include political disagreements, insufficient compliance with World Trade Organization (WTO) trade facilitation agreement (TFA) regulations, unsynchronized regulations among CEFTA countries, and mutual non-recognition of certificates.

EDGE has been instrumental in bolstering the capacity of business associations and individual companies in understanding and meeting EU regulations.

EDGE, with its vital contributions and rapid response potential, must continue to evolve in its strategies and interventions, adapt to changing dynamics, and overcome interconnected challenges. This is essential for EDGE to maximize its impact and assist the participating countries in their pursuit of EU integration by meeting accession criteria.

1. INTRODUCTION AND BACKGROUND

1.1. PROGRAM BACKGROUND

The EDGE project is designed to employ a flexible and adaptive learning approach that will make it possible to create and respond to economic growth opportunities as they arise. EDGE is implemented in the EE region. It works to create inclusive, sustainable economic growth and to support intra-regional and Euro-Atlantic integration. EDGE is managed by USAID’s EE Bureau in Washington, D.C. and works closely with USAID Missions and projects in the region. EDGE is designed to generate synergies among the USAID projects working in economic growth and to bring together stakeholders across countries to work on mutual problems, adding regional dimension to the EE portfolio. EDGE has three objectives:

1. Reduced barriers to cross-border trade and investment;

2. Improved business sophistication; and

3. Improved market integration and expanded market linkages.

Under EDGE, International Development Group LLC (IDG) is implementing regional economic development activities in the Balkans (Albania, Bosnia and Herzegovina, Serbia, Kosovo, Montenegro, and North Macedonia); Ukraine, Moldova, and Belarus; and the Caucasus (Armenia, Azerbaijan, and Georgia). Activities are rolled out through the Core Component and the Buy-in Component, a flexible mechanism that individual Missions, USAID Operating Units, and other donors can utilize with their own funds.

All core activities engage participants from at least two countries. Both components include a GUC mechanism to promote the capacity building and sustainability of local organizations, government entities, and private sector firms. Further, Core Component activities promote regionalization, reduce barriers to trade, and support the growth of SMEs in three key value chains: 1) fruits and vegetables; 2) eco-, agro-, and cultural tourism; and 3) light manufacturing in wood/furniture and textiles. Across the three value chains, EDGE promotes information communication and technology (ICT) opportunities for digitization. The buy-in component allows for quick mobilization of activities in any EDGE country and may be appropriate for bilateral or multi-country/regional activities. Buy-ins can fall under any of EDGE’s three objectives. The total budget ceiling for the core and buy-in components is $21 million.

1.2. PROGRAM CONTEXT

EDGE was launched in 2019 during a period of great turmoil and economic uncertainty that had negative effects on both regional and global economies. Events during this period were extremely unpredictable; the COVID-19 pandemic set in motion major disruptions in the supply chains, distribution problems, and difficult communication, resulting in the failure of many companies. In February 2022 Russia attacked Ukraine, initiating the greatest conflict in Europe since World War II. Energy and food prices in the region soared due to disrupted trade routes, sanctions against Russian energy and grain, and an influx of more than 5 million refugees into the region[footnoteRef:1] to flee from war or escape poor economic conditions. Domestic politics are influenced by Russian and nationalist parties’ disinformation campaigns, further exacerbating underlying ethnic, religious, and pro-Russian tensions in many countries.[footnoteRef:2] The massive influx of refugees has affected local economies and political stability in the region, especially in Armenia, Georgia, and Moldova. Concurrently, outmigration of youth in the region has increased labor shortages. The breakdown in regional cross-border trade reduced the availability of inputs for industry, agriculture, energy, and consumer goods. Households in the region are becoming more vulnerable because quickly rising costs of energy and food require them to spend a significantly higher percentage of their income on essential goods than before the pandemic and war in Ukraine.[footnoteRef:3] Evidence suggests that disruptions in supply chains in one country led to increased costs in neighboring countries. [1: World Bank, 2022 World Development Indicators; European Central Bank Access to Finance (2022); Organization for Economic Co-operation and Development Small and Medium Sized Enterprises (SME) Policy Index (2022).] [2: USAID Congressional Testimony of Assistant Administrator for EE Erin E. McKee before the Senate Foreign Relations Committee, “Countering Russian Aggression: Ukraine and Beyond.” January 26, 2023.] [3: According to a Regional Rural Development Standing Working Group (SWG) discussion on May 9, 2023, about 35 percent to 45 percent of household income in the region was spent on food from 2020-2022, whereas the average in Western countries is less than 15 percent according to the United Nations Development Programme.]

In response to the multiple crises, countries in the region are introducing measures to protect their economies, overcome liquidity problems, and subsidize energy costs. For example, central banks are raising interest rates to fight inflation—although doing so offsets the advantages of investing in the region because of traditionally low labor costs.[footnoteRef:4] Climate change impacts and sanctions against Russian energy imports affect the prices and availability of energy and other inputs needed for production. [4: World Bank, Europe and Central Asia Overview, April 6, 2023.]

1.3. PURPOSE OF THE EVALUATION

The purpose of this evaluation is to assess EDGE’s effectiveness and efficiency in addressing activity objectives and to develop recommendations to inform the EE Bureau’s thinking regarding the future design of regional economic growth activities. Of EDGE’s three objectives, this evaluation focuses primarily on Objective 1, reduced barriers to cross-border trade and investment, and Objective 3, improved market integration and expanded market linkages. The evaluation focuses on the period of performance from July 29, 2019, through July 2023. This report outlines actionable recommendations drawn from the key findings of each EQ; it suggests opportunities and possible adjustments that could enhance current program activities and help prioritize future regional activities in the USAID/EE portfolio.

1.4. EVALUATION QUESTIONS

This evaluation assesses specific programmatic approaches in achieving intended life-of-program results. To accomplish this task, the evaluation team developed an evaluation approach to address the five EQs listed below and outlined in the Evaluation Work Plan found in Annex A.

TABLE 2: EVALUATION QUESTIONS

1: To what extent has the project proved successful in assisting companies and organizations integrated into a) regional value chains and b) global value chains? Objectives and targets?

2: What lessons were learned from modifying the implementation of interventions during COVID-19 that can be used in future design considerations?

3. What are the strengths and weaknesses of the EDGE management structure?

4. To what extent did the leverage requirement contribute to sustainability of activities?

5. To what extent are EDGE activities supporting countries in meeting and fulfilling their EU accession criteria?

1.5. EVALUATION AUDIENCE

The primary audience of the evaluation is USAID/EE and the 12 USAID Missions in the region. USAID/EE may share the results of this evaluation with other stakeholders, such as government partners, industrial associations, nongovernmental organizations, and other USAID implementing partners (IPs) and donors working in this area.

1.6. HOW THE REPORT IS ORGANIZED

The evaluation report is organized as follows:

· Section 2, Methodology, outlines the methods used to inform this review, in addition to the sampling approach and data limitations.

· Section 3, Key Findings, Conclusions, and Recommendations addresses each learning question.

· Annex A provides the evaluation design for this review.

· Annex B provides a statement of differences, if any.

· Annex C presents the bibliography for the desk review.

· Annex D lists individuals who participated in key informant interviews (KIIs).

· Annex E presents the interview protocols for KIIs and the electronic survey related to MEL processes.

· Annex F provides disclosure of conflicts of interest forms signed by members of the evaluation team.

2. METHODOLOGY

2.1. EVALUATION METHODS

The MELDS evaluation team conducted this mid-term performance evaluation in Washington, D.C., and through fieldwork in Georgia, Kosovo, Moldova, and North Macedonia from April to July 2023. The team consisted of five core members: the team lead, senior data analyst and Georgian subject matter expert, Kosovar subject matter expert, Moldovan subject matter expert, North Macedonian subject matter expert, and a regional logistics coordinator. The evaluation team conducted KIIs with selected stakeholders from USAID; the IP, IDG; grantees; government partners; local partners;[footnoteRef:5] and other donors. The team used a mixed-methods approach that combined qualitative KIIs, small group discussions, and reviews of performance monitoring data. This approach reflects USAID’s similar combinations approach, which uses different methods to collect and analyze information that is then synthesized to answer evaluation questions.[footnoteRef:6] [5: Local partners include businesses, business service organizations, trade associations, BSPs, policy institutes, and professional associations.] [6: See USAID Technical Note: Conducting Mixed-Method Evaluations, Version I, June 2013.

The evaluation team was briefed on data collection protocols, as outlined in USAID’s Human Subject Protection Policy and USAID’s Evaluation Policy, regarding the rights and welfare of human subjects. The team leader trained the team in survey methodology, USAID’s survey regulations, relevant regulations, and the data collection plan. To start, the evaluation team held consultative meetings with the IP and USAID staff to gain inputs and solicit feedback, which it used to finalize the evaluation design.

2.1.1. SECONDARY DATA

The evaluation team conducted desk research before beginning fieldwork to identify and analyze secondary information that it could triangulate with data collected in the field. The team conducted an extensive desk review of key program and external documents to become familiar with key activities and build understanding of private sector engagement to situate the evaluation. The team worked with USAID/EE to retrieve relevant program documents including secondary data and background documents (relevant academic periodical publications, other donor reports, project surveys, monitoring and evaluation plans, work plans, and quarterly and annual reports). In total, the evaluation team reviewed 36 documents to obtain a comprehensive picture of the development context, challenges and priorities, economic policies, laws, and regulations, as well as insights into the business enabling environment, and competitiveness in the selected countries. These are listed in Annex C.

2.1.2. QUALITATIVE DATA COLLECTION

To obtain complementary and supplemental qualitative data focused on processes and results, the evaluation team conducted fieldwork from May 1–26, 2023, with team members based in Kosovo, Moldova, and North Macedonia. KIIs were conducted remotely in Georgia via online meeting platforms. During the three-week fieldwork, the evaluation team held KIIs and small group meetings with 71 stakeholders (48 percent women, 52 percent men). Figures 1 and 2, respectively, show the disaggregation of KIIs by sex and country. Key informants included IP leadership, MEL team members, and team leaders; subcontracting partners, including MEL and communications support; and representatives from USAID and other donors (see Figure 3). Evaluators captured responses using semi-structured and open-ended questions and used targeted probing questions and techniques to deepen the discussion and capture rich, high-quality data. The evaluation team sent brief discussion guides to the interviewees in advance of conducting KIIs.

FIGURE 1: KEY INFORMANTS DISAGGREGATED BY SEX

FIGURE 2: KEY INFORMANTS DISAGGREGATED BY COUNTRY

FIGURE 3: KEY INFORMANTS DISAGGREGATED BY STAKEHOLDER GROUPS

2.2. ANALYSIS

At the macro level, data analysis and synthesis were guided by an abductive reasoning approach advocated by Schwartz-Shea and Yanow.[footnoteRef:7] Unlike inductive and/or deductive modes of inquiry, which typically follow a more linear logic, abductive reasoning entails a more circular or spiraling process and represents a simultaneous puzzling-out of insights from data gained through the desk review and KIIs. The team’s approach to analyzing and triangulating findings used four distinct protocols to analyze data: [7: Peregrine Schwartz-Shea and Dvora Yanow. Interpretive Research Design: Concepts and Processes (New York: Routledge, 2012).]

· Content and comparative analysis of document review;

· Descriptive statistical analysis of quantitative MEL and survey data (using Survey Monkey);

· Coding and content/thematic analysis of KIIs (manually, through notes); and

· Data triangulation.

2.3. LIMITATIONS

Selection bias: Nearly all the key informants for this evaluation were based in four countries, although EDGE covers 12 countries in the EE region. Not all were familiar with the overall portfolio of activities. The evaluation team mitigated this limitation by conducting multiple interviews with IP and USAID staff who had high-level knowledge and understanding of EDGE.

Limited responses to the quantitative survey: The evaluation team designed an online quantitative survey (see Annex E) that it disseminated to more than 100 individuals who participated in EDGE training activities. However, with fewer than 10 responses, the response rate did not meet a 30 percent threshold for statistical significance. Therefore, the evaluation team was unable to use additional data to validate qualitative findings. The team considered lessons learned during recent evaluations regarding extensive use of qualitative collection methods.

Limitation of data comparability in two collection methods: The study was planned for data collection through two methods, in-person interviews and remote (online) surveys. The evaluation team understood that potential differences in the quality and depth of responses collected through these methods might influence the generalization of findings across the entire population of interest. The team acknowledges this limitation and considers ways to minimize its impact, such as by comparing the findings obtained through different methods and using appropriate statistical techniques to adjust for potential biases.

Difficulty assessing progress to sustainability and long-term objectives: The evaluation was conducted while implementation was ongoing. However, EDGE’s work to influence sustainability, such as leveraging funds and contributions to intra-regional and Euro-Atlantic integration, may take years to deliver the intended results. The period of performance for the activities ranges from less than two years to four years, and implementation is at various stages. The team assessed progress to date and highlighted potential gaps to be addressed in the remaining period of performance and in future design considerations.

2.4. GENDER AND SOCIAL INCLUSION CONSIDERATIONS

The evaluation findings captured the situations and experiences of males and females who participated in and/or benefited from EDGE activities. The evaluation team engaged with women and men regarding the effects of this engagement, including any unintended consequences for women—whether positive or negative. The evaluation design and methodology ensured that data collection did not disproportionately reach men or women participants by assessing the universe of participants and sampling from this group proportionally by sex. Data collection instruments and protocols were gender-sensitive and reflected an understanding of gender roles and constraints in local contexts, including an acknowledgement that 50 percent of EDGE participants are women.

3. EVALUATION FINDINGS, CONCLUSIONS AND RECOMMENDATIONS

The five subsections below present findings, conclusions, and recommendations for each EQ. Each begins by presenting the evaluation question, followed by the findings, then the conclusions, and ends with the recommendations.

EFFECTIVENESS OF IMPLEMENTATION

EQ1: TO WHAT EXTENT HAS THE PROJECT PROVED SUCCESSFUL IN ASSISTING COMPANIES AND ORGANIZATIONS INTEGRATING INTO A) REGIONAL VALUE CHAINS AND B) GLOBAL VALUE CHAINS?

FINDINGS

F.1.1. EDGE has demonstrated various strengths in its approaches to enhancing regional economic growth and competitiveness. From helping companies improve their management capacities and export capabilities to organizing high-impact regional forums and facilitating important legislative actions, EDGE’s presence has been meaningful and multifaceted. The project’s proficiency in working closely with other donor partners in the three targeted value chains (fruits and vegetables, wood products and textiles, and tourism) is evident and contributes to the effective and efficient use of resources. Moreover, its activities are improving trade integration and transparency in the region by facilitating consultations among relevant authorities according to participants in the Authorized Economic Operator (AEO) program. EDGE is demonstrating that systemic changes have the potential to increase and strengthen cross-border trade as a result of the study visit to two joint border crossings between Croatia and Bosnia Herzegovina. The EDGE team has shown considerable flexibility and responsiveness to mitigate the impact of economic disruptions affecting SMEs due to COVID-19 and the war in Ukraine.

F.1.2. EDGE has provided training and capacity building support to individuals and firms to advance their integration into regional and global value chains, as indicated by several key performance metrics. The project has reached 1,606 individuals and 520 organizations who participated in trade facilitation events capacity building activities.[footnoteRef:8] Seventy-five firms obtained certifications from international quality control institutions. The certifications, which signify adherence to global product standards, are essential for integration into global value chains. EDGE provided technical support to 231 organizations to improve SME competitiveness and facilitated new buyer linkages for 287 SMEs. Anecdotally, EDGE stakeholders producing textiles, furniture, and organic food products said their exports to EU markets increased, although networking opportunities did not lead to increased exports to regional markets. EDGE performance indicators do not track increases in sales, volume, revenue, or expansion of products, which limits the evaluation team’s ability to assess effectiveness outcomes. [8: These reported figures may include instances of double counting, potentially overrepresenting the number of unique entities. This reflects a limitation of the EDGE AMELP and MEL system.]

F.1.3. EDGE support was most useful in subsidizing participation in trade shows and other events, according to several respondents in all three value chains. One producer shared, “Promotion and marketing support [was useful], because no one knows what Kosovo produces. Companies need to be present in fairs and [business-to-business events] to be closer to buyers.” Apart from supporting participation of textile manufacturers in business-to-business (B2B) events, respondents working in the apparel value chains reported that the most consequential activity was a textile productivity simulation model rolled out in May 2022. In partnership with Axiom-Tech, which specializes in software for optimizing processes, EDGE identified four companies to implement computer modeling, simulations, and optimization of one production line. The goal is to improve productivity efficiency by 30 percent. Using a productivity simulation model should enable companies to identify and eliminate bottlenecks and improve productivity and quality. However, at the time of the evaluation it was not clear if more than four textile and apparel firms would benefit from this activity.

F.1.4. BSPs participated in two training courses that focused on improving SMEs’ access to finance and strengthening their competitiveness. EDGE joined other partners in organizing the regional online Conference on Organic Agriculture to exchange lessons, discuss the latest market trends and requirements related to the EU Action Plan for Organic Production, and explore opportunities to establish closer regional cooperation. BSPs participated in a regional conference, the Digitization in Agriculture Webinar, which introduced digitization in agriculture to regional stakeholders, including agricultural producers and digitization service providers. BSP respondents said their participation in training events was good for networking and gaining insights into financing issues in the targeted value chains. To date, no BSPs have reported gaining new clients as a result of these events, so it is difficult to assess the effectiveness of the two trainings. According to EDGE staff, BSPs and business support organizations (BSOs) were “involved in all EDGE activities” but the extent of their involvement cannot be assessed without reported outputs.

F.1.5. EDGE took a three-pronged approach to supporting companies to implement standards and certificates. One was direct support for certification using the grant scheme, through which companies from different value chains were certified. A second approach was direct engagement of certification bodies to certify companies. The third approach was training auditors and BSPs via learning events that informed companies of the procedures for certification and the importance of certificates for their work, especially in exporting to EU countries. Respondents noted that EDGE provided timely and quality support in these activities.

According to respondents, the CEFTA framework does not provide clear and transparent rules for trade. Regulatory disparities among CEFTA countries result in non-recognition of mutual standards, which reduces the competitiveness of goods and services produced in the region. For example, it is essential for food products to reach destinations on time, but CEFTA countries do not recognize the phytosanitary certificates and certificates of origin issued by neighboring countries.

Several respondents suggested that it would be preferable (and more efficient) to adhere to EU regulations and ignore the CEFTA framework. One respondent said, “Certification and training are a regional need because of the EU requirements. We can lose access to EU markets due to lack of certification. There is a space to invest in regional projects that support the certification process, because donors don’t do much here.” According to World Bank 2021 cross-border trade data, goods worth about $5.4 million were exported from one country in the Western Balkans subregion to other CEFTA countries, whereas goods worth more than $20 million were exported from the Western Balkans to EU countries. Many respondents, including staff from USAID and other donors, suggested the political stalemate between Kosovo and Serbia and the limited technical expertise of the CEFTA Secretariat make the organization ineffectual. Several respondents said further investments in the CEFTA process are not warranted.

F.1.6. EDGE training and mentoring indirectly assisted SMEs in countering investors’ perceptions of increased risks in the region due to political instability caused by the war in Ukraine, past inter-ethnic conflicts, insufficient application of the rule of law, extensive corruption, and slow and ineffective public administration. Many firms across the region suffer reputational damage due to uneven productivity and the poor quality of products in the agricultural and wood processing value chains, as well as slow delivery and corruption.[footnoteRef:9] Several respondents commented on the difficulty of finding trade partners due to negative perceptions of the region. Respondents from companies in all three value chains stated that their survival is tied to integration into larger regional and international markets, especially EU countries. [9: United Nations Office on Drugs and Crime, Business, Corruption and Crime in the Western Balkans, 2021; Freedom House Index for Government Accountability and Transparency. 2022.]

F.1.7. Under the core component, EDGE executed 10 GUCs in Year 2 and six GUCs in Year 4 that were used to improve financing, human resources, and automated production technology to engage in larger-scale production. Grantees and other respondents noted that the region is slow to digitize the entire production process that maps value chain participants and makes the process transparent, traceable, and efficient. One respondent said that digital farming techniques are cost-effective and efficient for improving farm management, which integrates financial and field-level records (data on soil, weather, and crop growth) but that only a few local producers implement them. EDGE delivered online regional webinars on ICT innovations and applications in smart agriculture that introduced digital tools and logical digital bundles for precise irrigation, satellite and drone crop monitoring, and pest monitoring with digital traps. About 20 individuals from 10 farmers’ associations participated, but no data is available regarding the application of this learning. According to EDGE staff, the project is actively collaborating with USAID Kosovo Compete Activity and CATALYZE Engines of Growth Activity to establish a secondary market for equipment, but the concept has not become operationalized. According to agricultural stakeholders, there is high demand for this type of technical assistance to improve decision-making and adapt new technologies.

F.1.8. Agriculture-focused activities have started to invest more in organic production of products with high export potential, especially in response to consumer demands in EU markets. Many donor projects support the agriculture sector in this area, but a need remains for large investments in organic production to achieve mass scale. To do so, producers must obtain annual certifications in organic standards, food safety, and quality ingredients. This is a high hurdle, because the process is comparatively expensive and there is a lack of accredited companies to provide certification in the region. EDGE issued a grant to a company in North Macedonia that provides certifications to organic food producers.

F.1.9. USAID Mission staff and other donors noted that legal frameworks are outdated and poorly suited to supporting the development of new niche areas of tourism. The Swiss Development Corporation respondents in Kosovo said that, only after 10 years, could it see a return on the agency’s tourism investment in capacity building and adequate local government support in one municipality for critical investments in infrastructure and workforce development. At the same time, USAID bilateral programming is investing strategically in niche tourism in Georgia, Moldova, Armenia, and other countries in the region. Large economic growth flagship programs like the Economic Security Program in Georgia have robust budgets and are building the capacity of local public and private sector partners to engage in historical, religious, cultural, gastronomic, and adventure sports tourism. EDGE partnered with USAID/Georgia and its bilateral programs to collaborate with Destination Management Organizations (DMOs) in Armenia. USAID/Moldova had an EDGE buy-in in the first months of Russia’s invasion of Ukraine to support local tour operators in providing shelter to Ukrainian refugees. Those tour operators, in turn, upgraded their hotels to make them more energy efficient, thereby extending their tourist season.

CONCLUSIONS

C.1.1. Several aspects of the EDGE project produced tangible results in regional value chain integration. The project has driven capacity building initiatives that show clear positive results for a majority of involved organizations. The project’s training sessions have also been largely successful; learnings were applied in 83 percent of cases and deemed useful by 93 percent of participants, suggesting that the initiatives have met relevant needs and delivered significant benefits.