SEDA OH evaluation final report.pdf

EVALUATION REPORT

FINAL PERFORMANCE EVALUATION OF

THE SOCIO-ECONOMIC DEVELOPMENT

ACTIVITY (SEDA) FEATURING

OUTCOMES HARVESTING

April 2020

This publication was produced at the request of the United States Agency for International Development.

It was prepared independently by ME&A, Inc.

FINAL PERFORMANCE

EVALUATION OF THE SOCIO-

ECONOMIC DEVELOPMENT

ACTIVITY (SEDA) FEATURING

OUTCOMES HARVESTING

This publication was produced at the request of the United States Agency for International Development under Contract/Task Order Number: AID-OAA-I-15-00024/72011219F00001

Prepared by:

Zehra Kacapor-Dzihic, Team Leader Kathy Selvaggio, Deputy Team Leader Naila Hashimova, Local Technical Expert Chinara Abbasova, Evaluation Assistant

Contractor:

ME&A, Inc.

4350 East-West Highway, Suite 210 Bethesda, MD 20814 Tel: 301-652-4334 http://www.meandahq.com/

DISCLAIMER

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

http://www.meandahq.com/

TABLE OF CONTENTS

EXECUTIVE SUMMARY ................................................................................................................................................... i

1.0 INTRODUCTION

1.1 Evaluation Purpose

1.2 Evaluation Questions

2.0 PROJECT BACKGROUND

2.1 Development Context

3.0 EVALUATION METHODS

3.1 Overall Approach

3.2 Evaluation Limitations

4.0 EVALUATION FINDINGS

4.1 EQ 1: What outcomes from the SEDA Activity are a) most evident and b) sustainable? . 10

4.2 EQ 2: What aspects of SEDA’s methodology, if any, are proven to be effective or not effective and how?

4.3 EQ 3: What distinguishes sustainable CDCs from the unsustainable ones?

5.0 CONCLUSIONS

6.0 LESSONS LEARNED

7.0 RECOMMENDATIONS

7.1 EQ 4: Based on the OH and evaluation of SEDA’s performance, what are the team’s observations or recommendations for areas of opportunity for future programming?

ANNEXES………

Annex 1: Statement of Work Annex 2: Data Collection and Analysis Tools Annex 3: List of Consulted Documents Annex 4: List of Interviewed Persons Annex 5: List of Visited Communities Annex 6: Sample of Communities Annex 7: List of Infrastructure Projects Annex 8: Evaluation Team Members Annex 9: Disclosure of Conflict of Interest Form Annex 10: SEDA Activity Intervention Logic Annex 11: Evaluation Schedule Annex 12: Evaluation Matrix Annex 13: Results of Questionnaires for CDCs, Local Executives and Municipalities, and

Beneficiaries Annex 14: Assessment of Outcomes Harvested With the SEDA Team Annex 15: Observations on Outcome Harvesting Process Annex 16: 2016 Public Perception Survey Data Annex 17: Statement of Difference

LIST OF FIGURES

Figure 1: Map of SEDA Regions and Communities Figure 2: KIs and Participants in GDs (n=208) Figure 3: Survey Responses to the Question “In your opinion, what proportion of community members has the SEDA infrastructure project benefited?”

Figure 4: Outcome Substantiation Through Field Inquiry “Community members have built greater trust and confidence in their ability to affect their own development through work with municipal authorities and ExComs”

Figure 5: Overview of Substantiation of Outcomes Relating to CDCs’ Ability to Engage in and Influence Public Policy Decisions Through Field Inquiry in Communities

Figure 6: Outcomes Relating to Women’s Empowerment Figure 7: Field Inquiry Map Figure 8: SEDA Activity Intervention Logic Diagram Figure 9: Results for Question “In your opinion, to what extent do members of the community who were very active before, for example, women and youth, participate more actively in community discussions and in setting community priorities through the CDC?” (n=118)

Figure 10: Results for Question “To what extent has the CDC helped to create a strong, trusting relationship between community members and local municipal authorities and/or ExComs?”

(n=119)

Figure 11: Results for Question “Compared with the time before the SEDA project, to what extent do ExComs and municipal authorities share information critical to government decisions more openly with communities?” (n=53)

Figure 12: Results for Question “Compared with the time before the SEDA project, do you believe that local socio-economic projects better reflect the needs and priorities of community members?”

(n=53)

Figure 13: Results for Question “What proportion of community members has the infrastructure project benefited?” (n=23) file://Users/michelleghiselli/Desktop/FINAL%20Evaluation%20report_SEDA%20Activity.docx#_Toc38886212 file://Users/michelleghiselli/Desktop/FINAL%20Evaluation%20report_SEDA%20Activity.docx#_Toc38886213 file://Users/michelleghiselli/Desktop/FINAL%20Evaluation%20report_SEDA%20Activity.docx#_Toc38886215 file://Users/michelleghiselli/Desktop/FINAL%20Evaluation%20report_SEDA%20Activity.docx#_Toc38886215 file://Users/michelleghiselli/Desktop/FINAL%20Evaluation%20report_SEDA%20Activity.docx#_Toc38886215

LIST OF TABLES

Table 1: Number and Type of GDs Conducted Table 2: Number and Type of Infrastructure Projects in the Conducted SOs Table 3: Respondents to Surveys Per Type of Survey Table 4: Evaluation Limitations and Mitigation Measures Table 5: Evaluation Recommendations Table 6: USAID-Provided Documents Table 7: Other Relevant Documents Table 8: Interviewed Persons Table 9: List of Communities According to Each Rayon and Region Table 10: Sampled Communities in Guba-Khachmaz Region Table 11: Sampled Communities in Aran Region Table 12: Sampled Communities in Ganja-Gazakh Region Table 13: Sampled Communities in Lankaran Region Table 14: Sampled Communities in Shaki-Zagatala Region Table 15: Overview of Infrastructure Projects Supported by SEDA Table 16: Evaluation Calendar Table 17: Evaluation Matrix Table 18: Results of Field Inquiry Towards Substantiation of Outcomes Relating to Democratic Practices and Sustainability Per Community Table 19: Results of Field Inquiry Towards Substantiation of Other Outcomes (per KII/GD) Table 20: Participation in Activities Concerning Socio-Economic Problems/Region

ACRONYMS

Acronym Description AZN Azerbaijani Manat AzRIP World Bank-supported Azerbaijan Rural Investment Project BTI Bertelsmann Stiftung’s Transformation Index CDC Community Development Council COE Council of Europe CSEP Civil Society Empowerment Program CSO Civil Society Organization CSOSI Civil Society Organization Sustainability Index CSSN Council on State Support to Non-Governmental Organizations under the

President of the Republic of Azerbaijan DRG Democracy, Human Rights, and Governance EQ Evaluation Question ET Evaluation Team EU European Union EWMI East-West Management Institute ExCom Executive Committee GD Group Discussion GDP Gross Domestic Product GOAJ Government of Azerbaijan ICNL International Center for Not-for-Profit Law IDP Internally Displaced Person IL Intervention Logic IP Implementing Partner IREX International Research & Exchanges Board KI Key Informant KII Key Informant Interview LLC Limited Liability Company M&E Monitoring and Evaluation MOE Ministry of Economy (former MOED) NGO Non-Governmental Organization OH Outcome Harvesting OR Operational Recommendation PEC Project Evaluation Committee PMP Performance Management Plan PPP Public-Private Partnership PPS Public Perception Survey PR Public Relations R-NRTC Regional NGO Resource and Training Center SEDA Socio-Economic Development Activity SO Site Observation SOW Statement of Work SR Strategic Recommendation TA Technical Assistance TOC Theory of Change U.S. United States

Acronym Description UMID Support to Social Development Public Union UNDP United Nations Development Programme UNFPA United Nations Population Fund USAID United States Agency for International Development WB World Bank i

EXECUTIVE SUMMARY

EVALUATION PURPOSE AND QUESTIONS

The purpose of the Final Performance Evaluation of the Socio-Economic Development Activity (SEDA) Activity, conducted from October-December 2019, was “to establish, using a mixed-methods evaluation to include desk review, key informant interviews (KIIs), and Outcomes Harvesting (OH), the most significant outcomes for increasing civic participation in governance to address social and economic needs in the regions of Azerbaijan, achieved by the SEDA activity.”1 The evaluation’s main scope was to determine the community mobilization and civic participation outcomes related to SEDA activities by answering the following four evaluation questions (EQs):

EQ 1: What outcomes from the SEDA activity are: a) most evident, and b) sustainable? Distinguish between outcomes, including: the empowerment of local communities, Community Development Councils (CDCs), and local government officials to engage in democratic processes; CDCs’ ability to engage in and influence public policy decisions; and factors affecting women and youth participation.

EQ 2: What aspects of SEDA’s methodology, if any, are proven to be effective or not effective and how?

EQ 3: What distinguishes sustainable CDCs from the unsustainable ones?

EQ 4: Based on the outcomes harvest and evaluation of SEDA’s performance, what are the evaluation team’s (ET) observations or recommendations for areas of opportunity for future programming?

At the request of the United States Agency for International Development (USAID), the ET also included a general assessment of outcomes within the Components II and III of the SEDA project.

PROJECT BACKGROUND

SEDA is a 10-year (2011-2021), $20 million-dollar USAID-funded program, co-financed by the Government of Azerbaijan (GOAJ), designed to increase citizen participation in governance and address the socio-economic needs of rural communities across Azerbaijan. Implemented by the East-West Management Institute (EWMI), and working in partnership with the Ministry of Economy (MOE) and local government officials, the project initially focused on advancing “citizen-driven development” through three main objectives: 1) establishing CDCs to increase civic participation in socio-economic development at the local and regional levels; 2) strengthening the capacity of civil society organizations (CSOs) to engage with citizens and government officials on local, regional, and national social and economic priorities; and 3) strengthening the capacity of the GOAJ to engage with stakeholders on key socio-economic development issues.

During project implementation, there was a shift in focus and scope of project activities away from direct capacity-building of CSOs and government institutions (Components II and III) toward supporting CDCs and local executive and municipal authorities to work together to identify, design, and implement small-scale infrastructure projects to address priority socio-economic needs through citizen mobilization (Component I). SEDA therefore revised its overarching objectives to:

1. Improve democratic practices at the local level; and

2. Contribute to solutions to socio-economic problems at the local level.

EVALUATION METHODS

The evaluation featured the OH method to help identify outcome-level results. The OH methodology -defined as changes in behavior, relationships, actions, or policies across SEDA interventions and across the time period of their implementation - allowed the ET to identify and substantiate outcomes across

1 Statement of Work (SOW), p. 3.

ii sectors, geographic locations, levels of engagement, and diversity of stakeholders in such a complex program like SEDA. In line with the methodology, the ET performed OH as a series of steps to map outcomes from primary activity documents and through a workshop with the SEDA project team. The workshop and follow-up work with the SEDA team resulted in a total of 36 harvested outcomes.

In addition, the ET employed traditional qualitative and quantitative methods to establish and validate outcomes, including: a thorough review of SEDA program documents; a total of 25 KIIs with 30 stakeholders at the national, regional, and rayon levels; group discussions (GDs) with CDC members and, in some cases, project beneficiaries of 54 infrastructure projects; site observations (SOs) of community infrastructure projects; and three separate surveys directed at CDC members, government authorities (municipal authorities, local/rayon Executive Committees, [ExComs]), and other beneficiaries, respectively. For the GDs and SOs, the ET conducted site visits to 37 communities (or 25 percent of the current 146 project communities) across five regions of the country. The ET used a heterogeneous purposive, non-probability sampling methodology to select the sample of communities.

The ET encountered several limitations, including: very limited reflection on outcome-level results in SEDA project documents; the need for the Ministry of Foreign Affairs and the President’s Administration to grant the ET permission for visits to site communities; difficulty in identifying and meeting final beneficiaries of SEDA-supported infrastructure projects; some degree of social desirability bias;2 and limited data and sources of information on SEDA’s Components II and III, which ended in 2015. The ET took steps to mitigate these limitations, where possible.

FINDINGS

EQ 1 Findings

1. The rapidly changing political environment over the course of project implementation has not been conducive for achievement of SEDA’s outcomes related to the enhancement of democratic practices.

2. SEDA’s investment in infrastructure produced the most evident and sustainable outcomes across SEDA project communities. The ET found ample evidence that local infrastructure delivers the improved services and public utilities demanded by citizens. Variations exist, but SEDA’s contribution to improvement of living conditions and livelihoods has been positive and has added value to local community development.

3. New and/or renovated roads improve accessibility, bringing improvement of investment and potential for income generation, household savings for community members (on transport costs), and improved access to services (medical, educational, other).

4. Providing or upgrading household water and electricity supply has improved living conditions through reliable electricity supply for lights and other appliances and better water and sanitary conditions, resulting in enhanced productivity and quality of life for household members.

5. Enhanced access to medical services by constructing medical centers in local communities has brought significant positive results, not only improving health but also indirectly contributing to sustainability because they are linked with the state health system.

6. Improved conditions in schools and kindergartens have increased attendance and enhanced the safety and health of students. However, the ET found less utility of sports fields.

7. Other types of projects, e.g., roof replacement and solar lighting, were also supported. While benefits from roof replacement were evident, the solar lighting project did not bring long-term results.

8. SEDA contributed to increased income generation. The direct contribution to generation of jobs was important yet limited to small numbers of people (a total of 93, mostly seasonal, jobs). Economic

2 The tendency of key informants and survey respondents to answer questions in a manner viewed favorably by others.

iii infrastructure such as feed grinding and egg incubation facilities as well as milk collection points directly contributed to improved livelihoods of local populations.

9. A total of 48 of SEDA-supported income-generation Civil Society Empowerment Programs (CSEPs) have been registered as businesses by the respective municipalities. Three are public-private partnerships (PPPs) and the rest are owned by the respective municipalities. According to SEDA, municipalities have the legal authority to run such businesses, most of which are small enough to operate under simplified tax laws. However, feedback from the field notes different levels of understanding of ownership and type of registration of such small businesses.

10. SEDA’s work with local communities promoted participatory processes and community mobilization to plan and implement infrastructure projects, which were valued by stakeholders. SEDA applies standard methodology for community mobilization through assistance to establish CDCs and encouraging all communities to go through a process of identifying the top priority projects through either a door-to-door survey or a community meeting in which people present their priorities and then jointly select the top three. However, across the project, stakeholders view SEDA first and foremost as a local infrastructure development project, which relies upon community participation mechanisms (e.g., local election of CDCs, community participation in the prioritization process, social mobilization, etc.) as tools to achieve the end goal of improved services through infrastructure, and not necessarily as a project that promotes empowerment of local communities, CDCs, and local government officials to engage in democratic processes.

11. Community members participating in CDCs have built greater confidence in their ability to work collectively to identify and address community problems. However, there is limited evidence that this empowerment leads to sustained democracy and civic mobilization in targeted communities.

12. There is ample evidence of improved relationships between CDC members and municipal authorities and local ExComs, but less evidence of the extent to which this resulted in a shift in government decision-making, especially at the rayon level and above.

13. Women community members take part in SEDA-supported participatory processes and women CDC members are vocal in expressing their interests and needs, although the extent of their direct influence on decision-making was not clear. Yet, although some women were strong and respected leaders in CDCs, and others are serving as the municipal authority or local ExCom in some communities, it was not possible to attribute this outcome to SEDA’s interventions.

14. There is little evidence of SEDA’s contribution to youth engagement. Youth was not mainstreamed by the project and youth engagement, in the few places where it happened, was not attributable to

SEDA.

15. SEDA’s activities under Component II led to results at the output-level, in terms of building CSOs’ capacity and conducting successful advocacy campaigns.

16. Discontinuation of activities under Components II and III had a negative impact on Component I by ending complementary efforts to promote more open, participatory government decision-making.

The discontinuation, however, was beyond SEDA’s control.

17. Under Component III, SEDA played a significant role in shaping, supporting, and promoting the 2014 Law on Public Participation, which established Public Councils as a central mechanism for citizen participation among other provisions, a notable outcome of the program. However, so far, the implementation of the law has been weak, with only a limited number of Public Councils established and their performance has not been particularly strong.

18. Other SEDA initiatives under Component III to expand opportunities for citizen participation in decision-making at the local and central government level produced no tangible outcomes, mainly because many activities under Component III were never fully operationalized.

iv

EQ 2 Findings

19. SEDA has a standard methodology for selection of community priorities that is applied across the project target areas. While the methodology enables the project to be efficient and allows consistent approach across communities, it presents deficiencies in terms of ensuring full engagement of local constituencies and relevance to local needs. The decision to expand the pool of targeted communities (breadth), rather than concentrating assistance in a smaller number of communities over a longer period of time (depth), spread SEDA’s resources thin, preventing the project from achieving deeper and more sustainable results through Component 1. In addition, the discontinuation of Components II and III prevented a more holistic approach to the empowerment of communities and enhancement of democratic practices. These factors might have affected SEDA’s focus of deeper and long-term promotion of local democracy practices toward infrastructure.

20. SEDA’s approach to CDC trainings, mentoring, and exchanges between communities is viewed as highly effective by CDC members and local government authorities. However, other standard SEDA approaches (e.g., the annual conference for CDC members, long-term development plans) were cited much less often as significant by stakeholders in communities.

21. SEDA’s approach to building a practice of local democracy through Component 1 activities was effective for the purposes of planning and implementing the infrastructure project, but not as effective in transforming the dynamics of citizen participation in local decision-making overall.

22. SEDA’s monitoring and reporting approaches are rather weak. Review of SEDA’s reports and other publications shows that SEDA succeeds in collecting and presenting good account of outputs and outcomes of its support to infrastructure in its reports, but achievements of results in terms of its contribution to outcomes relating to its first objective, “Improving democratic practices at the local level,” ranges from limited to none.

23. SEDA’s organizational structure secures a strong regional presence, adding value to engagement with targeted local communities and regions. However, periodic contact by SEDA’s regional representatives after the completion of an infrastructure project leaves communities anticipating future investments from SEDA, which often do not materialize.

EQ 3 Findings

24. The sustainability prospects for CDCs are often fragile. The CDCs are still informal bodies, which mainly engage with full membership in preparation for a new project by SEDA. The main driver of CDC sustainability appears to be high-quality leadership in the local community; the pre-existing level of organization in a community was also a large factor in sustainability.

CONCLUSIONS

1. SEDA has effectively contributed to solutions to socio-economic problems at the local level, fulfilling its second overarching objective. SEDA helped increase the capacity of elected CDC members and local government actors to apply civic participation practices in identifying and addressing local socio-economic development priorities. Such an approach enhanced delivery of services and public utilities demanded by citizens, reaching approximately 304,000 people across six targeted regions of Azerbaijan thus far, helping to improve living conditions and livelihoods.

2. SEDA’s contributions to the improvement of democratic governance practices (its first overarching objective) is limited at the time of this evaluation. SEDA assisted in building links between local community members, authorities, and ExComs. However, actual change towards more participatory government prioritization and decision-making processes has not materialized, particularly at the rayon level or higher, despite SEDA’s efforts. Thus, SEDA’s contributions are generally limited to bringing participatory, democratic processes centered around specific infrastructure projects.

3. The sustainability of results is fragile except for those derived from supporting specific infrastructure projects.

v

4. SEDA has not invested sufficient efforts in cultivating women’s and youth participation in CDCs, empowering women and youth leaders, and ensuring that CDC processes adequately considered women’s and youth’s specific interests and barriers. This has been a missed opportunity of the project.

LESSONS LEARNED

1. Projects that seek to address key bottlenecks/barriers to good governance and participatory democracy require integrated cross-sectoral/cross-government programming and careful examination of externalities and change processes to sustain optimized results.

2. Projects supporting long-term change in democratic practices need to consolidate rather than broaden their support efforts by focusing on key themes and priority support areas founded on precise project theory.

3. Implementing organizations can maximize their comparative advantage towards effective project implementation through an operational structure that stimulates and enables strategic partnering, and long-term, hands-on support.

4. Projects need to put participatory democratic practices at the center of their interventions to ensure that project results are not jeopardised by entrenched social roles or driven by closed group of highly motivated, competent, and trusted leaders in the community.

5. A participatory process of decision-making around a single community development project does not necessarily translate into broader democratic practices. Long-term intentional and targeted efforts to empower citizens and local change agents are needed to help make such processes a habit.

6. For such multidimensional projects such as SEDA, in order to ensure a smooth process, the monitoring and reporting on project interventions needs to be strong and ensure reporting at output and outcome levels.

RECOMMENDATIONS

Strategic Recommendations

1. SEDA should reset its approach to ensure that the social mobilization and participatory democratic governance practices are at the heart of the intervention, rather than a means toward infrastructure as the goal. It should also incorporate local leadership development as well as capacity-building approaches to support CDC advocacy with higher-level authorities at the rayon, regional, and state levels, perhaps through collaboration among communities within the rayon on common goals.

2. SEDA should focus on the depth, rather than the breadth, of support, by focusing on a limited number of communities over a longer period of time. Such an approach will deepen the commitment to and practice of participatory democracy by allowing CDCs to cultivate community participation and mobilization in a variety of settings.

3. SEDA should also invest efforts in building upstream partnerships and links with main service providers and decision-makers at the rayon level and above, to ensure full buy-in not only for the achievement of infrastructure projects, but also for the participatory, citizen-driven process.

4. SEDA should develop a gender and youth strategy, outlining opportunities and approaches in these areas. The strategy should cover both organizational and programmatic aspects, i.e., by strengthening its team with gender expertise, undertaking a thorough analysis of barriers and opportunities for greater participation of women and youth in CDCs, and reorienting itself to advocate and lead by example.

Operational Recommendations

1. SEDA should strengthen its outcome-level monitoring and reporting practices, more closely tracking the results of supported interventions over time, as well as what change they generate at the local level and factors contributing to their longer-term sustainability.

2. SEDA should collect and promote best practices on participatory socio-economic development and democracy to stimulate exchange of ideas and debate and set standards for CDCs’ operations.

1.0 INTRODUCTION

1.1 EVALUATION PURPOSE

The purpose of the Final Performance Evaluation of the Socio-Economic Development Activity (SEDA) Activity was “to establish, using a mixed-methods evaluation to include desk review, key informant interviews (KIIs), and Outcomes Harvesting, the most significant outcomes for increasing civic participation in governance to address social and economic needs in the regions of Azerbaijan, achieved by the SEDA activity.”3 The evaluation was conducted from October-December 2019 and focused on: 1) assessing the strategies applied in SEDA to identify changes that occurred at the community and local government level(s) in community mobilization, local government participatory processes, and citizen participation as a result of SEDA program activities and the extent of the contribution made by SEDA’s activities to these changes in behavior; and 2) exploring unanticipated/unplanned outcomes as well as those expected by the United States Agency for International Development (USAID).

The evaluation findings will be used to: a) inform future programming by better understanding the process of change or the causal linkages between community mobilization/civil society programming, and improvements in democracy and governance factors (like citizen participation and the engagement of government in communities); and b) test the efficacy of outcome harvesting (OH) as a methodology that could be used to evaluate other complex USAID/Azerbaijan programs.

1.2 EVALUATION QUESTIONS

The Statement of Work (SOW) for this evaluation (see Annex 1) included four Evaluation Questions (EQs):

EQ 1: What outcomes from the SEDA activity are: a) most evident, and b) sustainable? Distinguish between intentional and unintentional outcomes including but not limited to these areas: the following stakeholders’ empowerment to engage in democratic processes—local communities, Community Development Councils (CDCs), and local government officials; CDCs’ ability to engage in and influence public policy decisions; and factors affecting women and youth participation.

EQ 2: What aspects of SEDA’s methodology, if any, are proven to be effective or not effective and how?

EQ 3: What distinguishes sustainable4 CDCs from the unsustainable ones?

EQ 4: Based on the OH and evaluation of SEDA’s performance, what are the evaluation team’s (ET) observations or recommendations for areas of opportunity for future programming?

Per USAID’s request, the ET also included assessment of outcomes within Components II and III of SEDA to ensure the full scope of the project was considered.

2.0 PROJECT BACKGROUND

SEDA is a 10-year (2011-2021), $20 million-dollar USAID-funded program, co-financed by the Government of Azerbaijan (GOAJ), designed to increase citizen participation in governance and address the socio-economic needs of rural communities across Azerbaijan. Implemented by the East-West

3 SOW, p. 3.

4 For the purpose of assessing sustainability within the framework of this evaluation, the ET proposes to apply USAID’s definition of sustainability: “Sustainability refers to the ability of a local system to produce desired outcomes over time. Discrete projects contribute to sustainability when they strengthen the system’s ability to produce valued results and its ability to be both resilient and adaptive in the face of changing circumstances.” USAID (2014); Local Systems: A Framework for Supporting Sustained Development, p. 5, https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf.

Management Institute (EWMI), and working in partnership with the Ministry of Economy (MOE) and local government officials, the project initially focused on advancing “citizen-driven development” through three main objectives:

1. Revitalizing or establishing CDCs to increase civic participation in socio-economic development at the local and regional levels;

2. Strengthening the capacity of civil society organizations (CSOs) to engage with citizens and government officials on local, regional, and national social and economic priorities; and

3. Strengthening the capacity of the GOAJ to engage with stakeholders on key socio-economic development issues.

During the course of the project implementation, the focus and scope of project activities shifted away from direct capacity-building of government institutions toward mobilizing and supporting CDCs and local executive and municipal authorities to work together to identify, design, and mobilize resources from citizens and local governments, and implement small-scale infrastructure projects to address priority socio-economic needs. SEDA therefore revised its overarching objectives to:

1. Improving democratic practices at the local level; and

2. Contributing to solutions to socio-economic problems at the local level.

To date, SEDA has implemented small-scale infrastructure projects benefiting approximately 304,000 people from 146 communities across six economic regions of Azerbaijan—each with several implementation sites, with some communities benefiting from more than one project (see Figure 1).

While the scope and scale of the project changed over the course of its implementation, SEDA remained consistent in its belief that citizen participation in the governance and development of their communities helps both meet local and regional socio-economic needs and promotes transparent, accountable and sustainable service delivery, to help build trust between the government and its citizens. This evaluation offers insights across the entirety of SEDA activity, including components that were discontinued at an earlier stage of implementation.

5 https://data.worldbank.org/indicator/NY.GDP.PCAP.CD?locations=AZ.

Figure 1: Map of SEDA Regions and Communities

2.1 DEVELOPMENT CONTEXT

Azerbaijan’s economy has witnessed strong growth over the last decade, while making substantial progress in poverty reduction; it is now an upper middle-income country with a gross domestic product (GDP) per capita of $4,721 in 2018.5 Azerbaijan’s rapid economic development is mainly attributed to the exploitation of hydrocarbon resources. However, the reliance on petroleum revenues places the country at risk of volatility and raises concerns about long-term sustainability and macroeconomic stability. Azerbaijan’s key economic challenge is to develop a diversified economy that has new and sustainable sources of growth and becomes more competitive in global and regional markets. The GOAJ is resolved to address critical economic challenges and has adopted a number of state programs to facilitate economic development, with specific focus on developing rural areas.

Despite the achievements of the past decade, important disparities remain between the oil-rich capital agglomeration and the rest of the country and, more generally, between its urban centers and the rural areas. The Absheron region takes up only 3.9 percent of Azerbaijan’s territory but produces 80 percent of overall industrial output and accounts for about 90 percent of all oil and non-oil investments in the country. GDP per capita in the non-oil sector is still three times higher in Baku than the national average and disparities are also evident with regard to other socio-economic indicators. Yet, the agriculture sector is a key contributor to job creation, food security, and poverty reduction. While accounting only for about 5 percent of the GDP, it provides income and employment for about 40 percent of the employed population and remains the second largest export sector after oil and gas, averaging 50 percent of exports in the non-oil sector.

Despite the substantial increase in government investment in basic infrastructure over the past few years, gaps in terms of access and quality of public services for the rural population remain. A key constraint is the infrastructure deficit, the lack of rural roads, irrigation, drainage, and household water systems. Over the past decade, foreign investments have been made by USAID, the European Union (EU), the World Bank (WB), British Petroleum, the Asian Development Bank, and other international organizations to improve access to basic infrastructure in the regions. These investments led to the creation or reinforcement of representational structures capable of and responsible for community mobilization and maintenance of the installed infrastructure. However, legal persecutions launched in 2013 against non-governmental organizations (NGOs), which culminated in legislative changes in 2014, forced the majority of foreign organizations implementing socio-economic development projects in the regions (e.g., OXFAM, Save the Children, the International Research & Exchanges Board [IREX], etc.) to withdraw from Azerbaijan, leaving a funding vacuum for the implementation of small-scale community projects.

Historically, there are some traditional forms of civil society within Azerbaijani culture, such as urban and rural communities (mahalla) and elders’ councils.6 However, even though Azerbaijan’s legislation provides for the basic prerequisites of public participation, most of the population continues to view the state as the main defender of public interests. Laws and regulations are not properly followed on all social levels;

and most citizens act like passive observers to socio-political processes having not yet taken full advantage of the newly installed democratic system by exercising their right to participate in elections or the other freedoms guaranteed by law.

Data from the 2016 Public Perception Survey conducted for EWMI/SEDA by a team of three experts show that the respondents’ participation in voluntary groups and organizations in communities included in the research was extremely low.

The government’s restrictions on civil society and introduction of the 2014 legislation on NGOs has also had an impact on the extent of public influence on public policy decisions, and on civic activism in general.

National independent and international NGOs working in Azerbaijan faced both administrative and judicial problems, which eventually changed the whole civil society landscape. Administrative problems faced included administrative fines, unclear tax claims, and challenges using bank accounts. Judicial problems related to criminal investigations and arrests of NGO leaders on charges of treason, tax evasion, illegal entrepreneurship, and abuse of power.

Additionally, women’s participation in politics and in leadership positions remains a topic of contention in Azerbaijan. Despite laws upholding gender equality, women, at times, suffer from gender stereotypes, 6 Mahalla and aqssaqqals’ unions are informal structures of mutual support in which people get together to undertake common social tasks and create a system of informal social service provision.

which pervade public perception and generate situations where women are marginalized and overlooked for promotion to decision-making positions.

3.0 EVALUATION METHODS

3.1 OVERALL APPROACH

The evaluation methodology adhered to USAID Evaluation Policy7 and applied a mixed-methods approach,8 using the OH method to help identify outcome-level anticipated and unanticipated results.9 The SOW stipulates that the evaluation is to be used to “test the efficacy of Outcome Harvesting as a methodology that could be used to evaluate other complex USAID/Azerbaijan programs.”

The ET designed the methodology to feature OH, at the same time also integrating other methods. The OH methodology allowed the ET to identify and substantiate outcomes across sectors, geographic locations, levels of engagement, and diversity of stakeholders in such a complex program like SEDA. The methodology applied in this particular evaluation followed the standard steps, as elaborated below, and also integrated other qualitative and quantitative methodological elements that helped respond to EQs.

As basis of the OH process, the ET reconstructed a SEDA Intervention Logic (IL) during the inception phase (see Annex 10). The reconstructed IL helped to understand the envisioned/anticipated outcomes of SEDA’s assistance as presented in the main program documents (e.g., Logframe, Performance Management Plans [PMPs] and Work Plans). The IL was of critical importance in understanding and categorizing not only the anticipated and unanticipated outcomes but also linkages between these and their joint contributions to change across target groups and communities.

As shown in the IL diagram in Annex 10, SEDA’s anticipated outcomes relate to strengthening participatory and democratic practices in local government decision-making, as well as strengthening the role of civil society and citizens in this process. In order to ensure a systematic approach to categorization and analysis of outcomes relating to local governance and democratization, OH adhered to the good governance objectives as recognized by USAID’s Democracy, Human Rights, and Governance (DRG) Strategy10 (i.e., promotion of participatory, representative, and inclusive governance processes, and fostering greater accountability of institutions and leaders to citizens). The Civil Society Organization Sustainability Index (CSOSI) elements (i.e., organizational capacity, advocacy, service provision) were also used to guide more structured outcome classification and analysis of findings relating to civil society support to inform the evaluation conclusions and recommendations.

During the Inception Phase, the ET created an OH matrix as a tool to capture, categorize, and elaborate outcomes harvested through the document review, and updated and validated during the workshop with the SEDA team, the field, and the synthesis phase (see Annex 14).

The ET performed OH as a series of steps to map outcomes from primary activity documents validated through a workshop with the SEDA project team and interviews with key activity stakeholders, which were then winnowed down, substantiated, and verified by outside experts (substantiators) and secondary studies. The OH approach allowed for an elaboration of a full set of outcome definitions, i.e., identifying key outcome statements, explaining their significance, assessing SEDA’s contribution (partial/whole, direct/indirect), and examining any consequences (primary/secondary, intended/untended, positive/negative). Detailed steps are presented in Section 3.1.1 below. Also, Annex

7 https://www.usaid.gov/sites/default/files/documents/1870/USAIDEvaluationPolicy.pdf 8 See Stern et al. (2012), “Broadening the range of designs and methods for impact evaluations,” Department for International Development (DFID), Working Paper 38.

9 Defined as changes in behavior, relationships, actions, or policies across SEDA interventions and across the time period of their implementation.

10 https://www.usaid.gov/sites/default/files/documents/1866/USAID-DRG_fina-_6-24-31.pdf.

https://www.usaid.gov/sites/default/files/documents/1866/USAID-DRG_fina-_6-24-31.pdf

15 offers an analysis of the utility of OH methodology and main issues to be taken into account in planning evaluations that feature this methodology.

The ET conducted 37 site visits across the country: 10 communities in the Guba-Khachmaz region, 16 communities in the Aran region, 2 communities in the Lankaran region, 3 communities in the Shaki- Zagatala region, and 6 communities in the Ganja-Gazakh region. These site visits were critical for substantiating and validating the most significant stories of outcomes in the framework of the final evaluation. The ET used this technique while looking to examine a diverse range of communities and gain as much insight from different angles.

See Annex 6 for the full list of sampled communities included in the field inquiry and a map of visited communities.

3.1.1 Qualitative Research and Analysis

The qualitative evaluation was conducted primarily through:

Document and Literature Review

The ET reviewed documents related to the SEDA project, including the project work plans, monitoring and evaluation (M&E) plan, PMP, annual reports, quarterly reports, data sets, and other relevant documents (see Annex 3 for the list of reviewed documents). A total of 91 outcomes were harvested in the literature review, mostly elaborating outcomes of individual infrastructure projects. This initial list of outcomes was used as basis for the OH workshop with the SEDA team. The quality of the outcomes harvested during this phase was affected by the reporting quality and methodological issues (see Section

3.3. for limitations).

Outcome Harvesting Workshop with the SEDA Team

In line with the methodology, an OH workshop was held in Baku on October 30-31, 2019, with the SEDA team to: harvest the outcomes; share the OH method principles and approaches; and validate and elaborate outcomes harvested during the document review, ensuring that the definitions and elaboration are clear and based upon evidence. Due to the limited number and quality of outcomes harvested within the document review phase, the ET worked intensively with the SEDA team to harvest and elaborate further outcomes. Following the workshop, the ET continued working with the SEDA team to finalize the outcomes harvested list towards their validation and substantiation during key informant interviews (KIIs), group discussions (GDs), and site observations (SOs) across 37 sampled communities (one more than the requested 36 communities stipulated in the SOW) and in Baku. The workshop and follow-on work of the SEDA team resulted in a total of 36 harvested outcomes.

Key Informant Interviews

The ET collected data from key informants (KIs), comprised of the SEDA team, USAID, one development partner (the United Nations Development Programme, or UNDP), executive committee (ExCom) officials at the village and rayon levels, municipal authorities’ representatives, CDC members, and representatives of local CSOs. Information was collected through in-depth, semi-structured KIIs based on outcomes harvested with the SEDA team and the EQs for this evaluation. In total, 30 stakeholders were interviewed through 25 KIIs. ET members took detailed notes from all KIIs to assist substantiation and validation of outcomes.

Group Discussions

The ET met with CDC members and, in some cases, project beneficiaries of 52 infrastructure projects to learn about their perception of SEDA’s performance and achievements. The ET used the outcome matrix as the basis for GDs to assist substantiation and validation of outcomes but also explore views and feedback on other relevant EQs. In total, 41 GDs (13 GDs in the Guba-Khachmaz region, 7 GDs in the Ganja-Gazakh region, 16 GDs in the Aran region, 3 GDs in the Shaki-Zagatala region, and 2 GDs in the

Lankaran region) with 185 project stakeholders were conducted (see Table 1 below). The ET conducted a very small number of interviews and discussions with project beneficiaries who were not CDC members, which is a limitation of the evaluation (see Figure 2), due to the time limitations for setting the meetings and visits to a large number of communities but, importantly, also due to ET’s lack of independent access to citizens in communities (see Evaluation Limitations below). The ET relied upon a contact list from SEDA as well as coordination through rayon ExComs provided by the government to arrange community meetings, and there was no opportunity to meet with community members or even CDC members separately.

Figure 2: KIs and Participants in GDs (n=208) The ET met with CDCs (but did not succeed in interviewing final beneficiaries) in a number of communities in the Guba-Khacmaz region (10), Lankaran region (2), Shaki- Zagatala region (3), Ganja-Gazakh region (6), and Aran region (16). However, the team was able to meet final beneficiaries in the following regions: Guba-Khachmaz region (4), Aran region (1), and Ganja- Gazakh region (1), generally because they participated in the ET meetings with the CDCs, although they were not members of the CDC.

Table 1: Number and Type of GDs Conducted

Types of GD Guba- Khachmaz

Ganja- Gazakh Aran Shaki-

Zagatala Lankaran Total

GD with CDC members 9 6 15 3 2 35 GD with beneficiaries 4 1 1 0 0 6 Total 13 7 16 3 2 41

Site Observations (SOs)

The ET conducted SOs of (re)constructed infrastructure in nearly all 37 sampled communities in five regions. These SOs were useful to observe utility of (re)constructed infrastructure in real-time and the extent to which these have been beneficial to local population.

Table 2: Number and Type of Infrastructure Projects in the Conducted SOs

Type of Infrastructure Aran Guba- Khachmaz

Ganja- Gazakh

Shaki- Zagatala Lankaran Total

Milk collection point 0 1 0 0 0 1 Medical center 1 5 1 0 0 7 Feed-grinding 2 4 2 0 1 9 Road rehabilitation 4 3 0 1 0 8 Kindergarten repair 1 0 0 0 1 2 Sport facility 2 0 1 0 0 3 Electrical supply 1 2 0 0 0 3 Carpet weaving 1 0 0 0 0 1 Water supply 1 1 3 0 1 6 School heating 1 1 1 0 0 3 Solar street lighting 1 0 0 0 0 1 Irrigation water 1 0 0 0 0 1

Type of Infrastructure Aran Guba- Khachmaz

Ganja- Gazakh

Shaki- Zagatala Lankaran Total

School fence 0 1 0 0 0 1 Roof replacement 0 1 0 0 0 1 Mobile tent 0 1 0 0 0 1 Residential roof 0 1 0 0 0 1 Egg incubation 0 0 1 1 0 2 Kindergarten heating 0 0 1 0 0 1 Total 16 21 10 2 3 52

3.1.2 Quantitative Research and Analysis

Surveys

To complement qualitative data collection methods, the ET designed three separate surveys directed at CDC members, government authorities (municipal authorities and local/rayon ExComs), and other beneficiaries, respectively. In total, 119 CDC members completed a survey for CDCs; 54 ExComs and municipal authorities responded to the survey for government authorities; and 23 questionnaires were collected from beneficiaries. These surveys contained closed-ended questions. The surveys were anonymous and were distributed to participants in Azerbaijani language at the end of each KII or GD. In total, 196 KIs or GD participants took part in surveys (see Table 3).

Table 3: Respondents to Surveys Per Type of Survey

Type of Survey Guba- Khachmaz

Ganja- Gazakh Aran Shaki-

Zagatala Lankaran Total

Survey with CDC members 32 10 62 6 9 119 Survey with ExComs and municipal authorities 13 13 20 2 6 54

Survey with beneficiaries 18 2 3 0 0 23 Total 196

Analytical reports for each survey are presented in Annex 13.

Sampling Methodology

SEDA is active in six out of 10 economic regions in Azerbaijan: Guba-Khachmaz, Aran, Ganja-Gazakh, Lankaran, Shaki-Zagatala, and Upper-Garabagh. In the course of its 10-year lifespan, SEDA has benefited 146 communities in these regions.

The ET used a maximum variation or heterogeneous purposive, non-random sampling methodology to select a sample of 37 communities for the evaluation, or 25 percent of the 146 communities in which SEDA has supported projects.