ATTACHMENT_J_5_CITIES_Impact_Evaluation_Design_Report.pdf

EVALUATION DESIGN REPORT

CITIES IMPLEMENTING TRANSPARENT, INNOVATIVE,

AND EFFECTIVE SOLUTIONS (CITIES)

December 2017

This publication was produced at the request of the United States Agency for International Development. It was prepared by Dan Killian, Jacob Patterson-Stein, and Dima Toukan of Management Systems International, a Tetra Tech company.

CITIES EVALUATION DESIGN REPORT

Submitted September 2017 Revised December 2017 USAID/Jordan Monitoring and Evaluation Support Project (MESP) Contracted Under AID-278-C-13-00009

Cover Photo: Municipal officials express their hopes and concerns regarding implementation of the Kingdom’s new decentralization agenda.

Photo Credit: Jacob Patterson-Stein

Management Systems International Corporate Offices 200 12th Street, South Arlington, VA 22202 USA

DISCLAIMER

The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

CONTENTS

Acronyms

Executive Summary

Introduction

Background

Cities Implementation

CITIES Development Hypotheses and Theory of Change

Other Local Governance Efforts

Evaluation Purpose and Scope

Evaluation Methodology

Evaluation Questions

Evaluation Theory of Change and Hypotheses

Evaluation designs for causal inference

Evaluation indicators

Learning Agenda

Relationship between the Learning Agenda and Impact Evaluation

Gender and Social inclusion in Evaluation

Data collection

Implementation Fidelity

Analysis

Matching municipalities

Sample sizes and statistical power

Qualitative Data Analysis

Strengths and Limitations

Data Management Plan

Conclusion

Annexes

Annex 1: Selected bibliography

Annex 2: Select Activities Aligned with CITIES Goals

Annex 3: Evaluation Hypotheses Mapped to Illustrative Survey Questions

Annex 4: Analyzing Institutional Capacity

Annex 5: Listing of Selected Treatment and Comparison Municipalities

TABLES

Table 1 CITIES evaluation questions Table 2 CITIES Evaluation hypotheses for measuring effectiveness Table 3 Evaluation indicators Table 4 Primary learning agenda questions Table 5 Data collection methods and indicators Table 6 Initial balance across treatment and comparison, Class B-D municipalities Table 7 Standardized differences across municipality characteristics Table 8 Indicators from CEP survey Table 9 CITIES evaluation team Table 10: MICA Items Under Municipal Control

FIGURES

Figure 1 Local government structure Figure 2 Jordan's Governance Structure and CITIES activities Figure 3: Simplified CITIES Theory of Change Diagram Figure 4: Evaluation Theory of Change Figure 5 Difference-in-differences (d-i-d) evaluation design Figure 6 Difference-in-differences with spillover group Figure 7: Distribution of propensity scores across treatment and comparison groups Figure 8 Standardized differences across selected treatment and comparison municipalities Figure 9 General sampling design for the population survey Figure 10 Distribution of CITIES sample sizes from population survey, from 10,000 simulated draws Figure 11 Simulated power curves based on CEP data Figure 12 Population survey implementation schedule

ACRONYMS

ADS Automated Directives System CEP Community Engagement Project CITIES Cities Implementing Transparent, Innovative, and Effective Solutions DEC Development Experience Clearinghouse d-i-d Difference in differences FY Fiscal Year GoJ Government of Jordan IE Impact evaluation MEL Monitoring, Evaluation, Learning MoI Ministry of Interior MoMA Ministry of Municipal Affairs MoPIC Ministry of Planning and International Cooperation PSM Propensity Score Matching SOW Statement of Work USAID U.S. Agency for International Development

EXECUTIVE SUMMARY

USAID/Jordan has requested technical support from its institutional contractor, the Monitoring and Evaluation Support Project (MESP), in designing a prospective evaluation and learning agenda of its municipal governance initiative, Cities Implementing Transparent, Innovative, and Effective Solutions (CITIES). By strengthening municipal institutional capacity, improving public service delivery, and strengthening community resilience and cohesion, it is expected that citizens will ultimately judge their local municipality to be more responsive to their needs and more effective in executing their core functions.

This Design Report establishes the rationale, design, and operational activities for such an evaluation.

The evaluation uses a mixed methods approach to compare institutional-level measures between 15 treatment and 15 comparison municipalities, small-scale surveys of direct beneficiaries of municipal service improvements1, and qualitative inquiry to understand the context-specific factors behind municipal institutional capacity development and successful implementation of a decentralization agenda.

The evaluation also includes a learning agenda to help understand the environment in which findings are situated, identify heterogeneous treatment effects, and learn which factors strengthen or impede activity success. To the extent feasible, all data will be disaggregated by youth, gender, and membership in a marginalized group, and these disaggregations will also be examined as a cross-cutting theme across the primary fields of inquiry around institutional development and improved service delivery.

For much of its history, USAID has relied upon local governance and community development programming as a primary tool to foster democratic norms and institutions.2 Complementary to this effort has been longstanding support of and advocacy at the central level for political and fiscal decentralization. While community development and political and administrative decentralization has a long history amidst a general consensus that such programming is effective at delivering resources to the local level while building democratic capacity through participatory planning and budgeting processes, only recently has a body of evidence started to coalesce around the specific outcomes of such programming and their magnitudes.

This evaluation will contribute to the emerging body of evidence in order to help USAID better tailor subsequent democracy and governance investments in Jordan and elsewhere. The findings from this evaluation and the learning agenda will help USAID, the Government of Jordan, and the broader donor and governance community better understand the impact of decentralization efforts at the municipal institution level and for the citizens who ultimately benefit from efforts to make government more effective, equitable, and responsive.

1 The evaluation design currently does not offer specific detail on small-scale beneficiary surveys. Detailed planning for such surveys will develop alongside the specific disbursement of CITIES grant funding or other service delivery initiatives induced by CITIES activities.

2 See Immerwahr (2015) for an engaging history of American interest in and practice of local community development.

INTRODUCTION

This document provides a framework for an impact evaluation (IE) design and identifies illustrative questions for an exploratory learning agenda. First, it describes the Jordanian governance context and the main CITIES activity components, which have been designed to address key decentralization and local governance efforts specific to Jordan. The document then lays out the evaluation questions, theory of change, analytical approach, and key indicators for the learning agenda. The data collection approach and implementation fidelity monitoring are somewhat novel in this design as they combine household survey, implementing partner data, and qualitative measures to develop an overall estimate of the activity’s impact. Subsequent sections focus on the analytical approach for matching pre-chosen CITIES municipalities with a group of comparison municipalities and the sampling plan, followed by a discussion of the data analysis methods for qualitative data. The final sections cover the data management plan, the proposed evaluation team composition, and the evaluation’s baseline timeline.

Since the proposed IE design requires that endline data be collected after the current end date for the MESP contract (November 2018), the design and estimated timeline for data collection, analysis, and reporting activities are presented for baseline only. Additional coordination between MESP and USAID will be needed to determine how to proceed with evaluation activities that extend beyond the current MESP mechanism.3

BACKGROUND

Strengthened democratic accountability is critical to Jordan’s stability and future prosperity. Over the past 10 years, Jordan has pursued several structural reforms to enhance the conditions for liberalization and more inclusive growth. Decentralization is an integral part of the reform agenda, undertaken to encourage a greater proportion of citizens to engage politically and to mitigate the stressors that are affecting them on a daily basis. Adverse regional developments, in particular the Syria and Iraq crises, continue to affect Jordan in the form of increased socio-economic pressures due to the vast inflow of refugees.

There is a long tradition of official municipal organization in Jordan, yet developing relevant, responsive, and representative local institutions remains an ongoing challenge.4 Municipalities as a modern governance institution were developed in 1955 and have been subject to several reform efforts (Ababsa 2003). In 2002, the Ministry of Municipal Affairs (MoMA) consolidated the number of municipalities across the Kingdom to address financial insolvency issues, with the total number of municipalities falling from 328 to 99.5 There are now 100 municipalities, which receive funding from the national government based on their population, and within which there is a municipal council voted into office through local elections.6 Municipalities collectively cover an estimated 83 percent of Jordan’s population. This local

3 MESP will prepare a handover memo with accompanying documents after delivery of the baseline report, expected to be in July 2018.

4 For an excellent backgrounder on the historical context of decentralization in Jordan, see Jordan - Third Tourism Development Project: Annex B.

5 See Bergh, 2016, page 81. Also, MOPIC (Ministry of Planning and International Cooperation), 2006, Municipal Joint Service Councils Assessment Study, ERM, 154 p. as quoted in Ababsa 2013.

6 See Bergh 2016. Also, Jordanian Embassy, About Local Government.

administrative structure overlaps with other institutions that exist at the sub-national-level, such as governorates, and Ministry offices and initiatives, such as Local Development Units, which are responsible for socioeconomic development in municipalities and governorates.7

In 2015, Jordan passed both a Decentralization Law and a Municipalities Law as part of an effort to integrate municipalities into a general governance framework and increase accountability among local democratic institutions. Amman has been the locus of control and remains the primary service provider across Jordan. Recent local elections that took place in August 2017 under the new legal framework provide a window of opportunity to strengthen local governance and engage communities in their own development. The elections brought elected local councils, mayors, municipal councils and governorate councils under a new system of governance that is expected to strengthen accountability and transparency, improve service delivery and increase community engagement. It remains to be seen if these recent legislative measures, and the increased enfranchisement at the municipal level lead to improved local governance and increased citizen engagement.8

Figure 1 Local government structure

CITIES IMPLEMENTATION

USAID commissioned Cities Implementing Transparent, Innovative, and Effective Solutions (CITIES), a five-year activity that Chemonics International9 started implementing in October 2016, to further assist

7 See OECD, 2017.

8 Ibid 9 Chemonics leads a consortium of implementing partners consisting of VNG International, Planet Partnerships, ConsultUs, and Interdisciplinary Research Consultants.

the Government of Jordan (GoJ) in its efforts to improve local service delivery and governance, citizen engagement, and accountability at the municipality level. CITIES will be implemented in 33 municipalities across all 12 governorates in Jordan.

CITIES implementation is based around four components that cut across its overarching decentralization facilitation goal:

1. Improved Service Delivery: Illustrative activities include the Development of Service Delivery Improvement Plans (SDIP); assistance with solid waste management (SWM); conducting municipal institutional capacity assessments (MICA); and additional activities that align with MOMA priorities.

2. Increased Effectiveness of Local Government Operations: Illustrative activities include analysis on legal and regulatory frameworks; review of human resources, financial, organizational, and budgetary management in municipalities.

3. Increased Responsiveness to Community Priorities: Illustrative activities include support for the Ministry of Political and Parliamentary Affairs; and assessment of public outreach tools.

4. Enhanced Capacity to Promote Community Cohesion and Resilience: Illustrative activities include youth engagement, grant support and local development capacity building.

By strengthening municipal institutional capacity, improving public service delivery, and strengthening community resilience and cohesion, it is expected that citizens will ultimately judge their local municipality to be more responsive to their needs and more effective in executing their core functions.

As shown in the figure below, most of CITIES activities are focused at the municipality level of government, but implementation for component 2 activities does include interventions at higher-levels, as well. The results of CITIES are expected to contribute to USAID’s development objective of stronger democratic accountability and a more stable, prosperous Jordan.

Figure 2 Jordan's Governance Structure and CITIES activities

CITIES DEVELOPMENT HYPOTHESES AND THEORY OF CHANGE

According to the CITIES Monitoring, Evaluation, and Learning (MEL) plan, if municipal government effectiveness and service delivery is increased, and if municipalities are better able to address the priorities of their community members while promoting community resilience, then democratic governance will be enhanced; trust and confidence in elected officials will be improved; and stability will be strengthened. A simplified theory of change (with arrows denoting the ‘how’ rather than ‘so what’ question) is as follows:

Figure 3: Simplified CITIES Theory of Change Diagram

OTHER LOCAL GOVERNANCE EFFORTS

There are several USAID and other donor-funded activities that support aspects of local institutional strengthening in Jordan. Annex 2 provides a summary of each of these activities. For example, the World Bank is coordinating with USAID to strengthen municipal capacity in response to the influx of Syrian refugees, particularly in host communities. The Federation of Canadian Municipalities is implementing a five-year program to strengthen municipality resilience in the Central and Southern parts of Jordan. The German development agency, Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ), also has three water, wastewater, and treatment activities that are similar to CITIES municipal service delivery component. In addition, USAID has programs across Jordan that support essential public services that affect life in the municipalities, such as water, education, and health interventions. USAID also supports engagement between civil society organizations (CSOs) and government institutions;

promote social cohesion and build local institutional capacity in areas with large refugee populations; and enhance local governance, elected official capacity, and conduct public opinion research.

Each of these activities has their own geographic implementation areas and scopes, but support the general goal of strengthening local institutions and supporting decentralization broadly. CITIES will likely implement in areas where at least some of the residents have had the opportunity to benefit from another donor-funded activity, as well as any locally funded development efforts.

EVALUATION PURPOSE AND SCOPE

USAID/Jordan is commissioning this evaluation to assess CITIES progress towards achieving its objectives, and examine how the implementation of CITIES affects municipal operations, the quality of public services, public perceptions of local municipality effectiveness, relevance, and responsiveness, community cohesion, and general quality of life. This evaluation is tasked with focusing on outcomes at the municipal level as measured through the MICA score, but will also collect household-level information to assess how individuals are responding to changes in their communities. The evaluation will also consider the implications of any unanticipated positive or negative outcomes. The data collected for this evaluation will complement the data collection and reporting that Chemonics will perform on their target municipalities.

Audience and Use

USAID/Jordan is the primary audience for this evaluation. The implementing partner and GoJ are also key recipients of the findings of this evaluation. Given the broad presence of donors working on decentralization and limited rigorous evidence on decentralization, the results of this IE will be relevant for general learning on how development activities affect governance.

USAID will use this evaluation to inform the design and implementation of future democracy, human rights, and governance (DRG) activities. As one of the first rigorous evaluations of local governance in Jordan, the evaluation will serve as a tool for USAID, the implementation community, other donors, and GoJ to learn about how active engagement on civic outreach, capacity building, and municipal responsiveness affect people in their daily lives. Given that there is no certainty about what the results will show, it will be important to use the findings of this IE as a platform for dissemination and discussion on decentralization and local governance in Jordan, as well as the impact of DRG activities more broadly.

Evaluation Scope

As described further under the data collection section, this evaluation will cover 15 of the 33ITIES municipalities as well as a comparison group of municipalities to draw conclusions about the effect of CITIES. The primary focus of evaluation measures is at the institutional level, in order to test the intervention’s theory of change that capacity development will engender improved citizen engagement, municipal planning, and service delivery at its most critical juncture. Small-scale surveys will assess the extent of additional or improved services and solicit citizen perceptions proximate to improved services.

Additionally, a learning agenda will ask ancillary questions to understand the implementation environment or explore specific pathways through which outcomes are realized. Finally, the evaluation will draw from a general population survey and learning agenda that MESP is undertaking separately for USAID. This activity will provide household-level data on both CITIES and non-CITIES households that will be used to test CITIES hypotheses on citizen engagement and government responsiveness and effectiveness.10

EVALUATION METHODOLOGY

This section reviews the general evaluation questions, specific hypotheses testing the intervention’s theory of change, the inferential design strategy to answer whether the intervention is realizing outcomes, a set of learning agenda questions to ask secondary questions about CITIES outcomes or implementation dynamics, and modes of data collection that will map to 10 indicators measuring specific hypotheses or learning agenda questions.

10 In some cases, there may be CITIES municipalities with population survey data but not institutional measures.

These cases will be examined as a type of secondary analysis of the CITIES evaluation.

EVALUATION QUESTIONS

The evaluation team will organize its work according to broad performance evaluation questions that examine activity effectiveness, sustainability, synergies, and learning. Impact measurement will test specific hypotheses under activity effectiveness. Qualitative / mixed methods inquiry will address broad themes of sustainability and synergy. Finally, a set of learning agenda questions will formalize what can be learned analytically through triangulation of data sources and secondary data analysis. Additional lines of inquiry may be added as the evaluation team learns more about implementation, and CITIES adapts to local municipality needs. The evaluation questions are summarized below.

Table 1 CITIES evaluation questions Theme Question(s) Approach

Effectiveness

1. What is the overall effectiveness of CITIES for achieving its goals of enhancing democratic governance, improving trust and confidence in elected officials, and strengthening stability?

1a. Did the activity achieve its intended outcomes?

1b. Are there certain areas/activities and components that have been more effective than other areas/activities and components?

Effectiveness is measured by 10 indicators across six hypotheses. See the indicator list for additional detail.

Sustainability

2. Which interventions under CITIES are most likely to be sustained over time? Why and how? To what extent are CITIES capacity building efforts being sustained by municipalities and other targeted bodies of decentralization?

Qualitative inquiry, triangulation of data sources, expert observation and judgment

Synergies

3. How well has CITIES developed synergies (coordination and collaboration) with other USAID activities that focus on similar objectives, for example, decentralization and working with Municipalities? Are donors operating in the decentralization and local governance space achieving synergies through coordination or knowledge sharing?

Qualitative inquiry, triangulation of data sources, expert observation and judgment

Learning

4. Based on the performance of CITIES, both in terms of effectiveness and sustainability, what are some key lessons learned, by component and sub-activity?

Summary reflection across all evaluation activities and data.

Learning agenda questions for specific inquiries about the mechanics of implementation and outcomes.

EVALUATION THEORY OF CHANGE AND HYPOTHESES

As first stated in the CITIES evaluability assessment, it can be helpful to explore the causal pathways of a development hypothesis in more depth than is traditionally captured in USAID activity MEL plans. The evaluation team has examined CITIES implementation plans and met with key CITIES staff to understand how the various activities relate to the activity’s overarching goal. Any evaluation must have a strong understanding of the various mechanisms that contribute to the outcome of interest. The figure below follows the various programmatic streams under CITIES and relates these through seven hypotheses to key outcomes and overall goal.

Figure 4: Evaluation Theory of Change

In this extended logic model, boxes in yellow represent programmatic inputs, light green boxes represent intermediate outcomes, light blue boxes represent activity outcomes, and purple boxes represent mission-level outcomes. All boxes feed into the overall USAID mission goal of a stable and prosperous Jordan. Note that the highest level impact indicator in the CITIES MEL plan is municipal government effectiveness, while the CITIES evaluation includes the higher level impact measures of confidence and stability. For accountability purposes, CITIES performance targeting applies only to its sub-purpose outcomes of service delivery, institutional capacity, responsiveness, and cohesion, and its activity level impact measure of municipal government effectiveness.

For measurement and learning purposes, the overall CITIES development hypothesis and theory of change is broken into seven constituent hypotheses based on how programmatic inputs and activity outcomes may interact to produce desired results. This also enables the evaluation team to better assess the different aspects of activity implementation and their respective contributions to observed outcomes. However, while the validity and directionality of each hypothesis will be explicitly tested, and the evaluation learning agenda will examine how the failure of one hypothesis may affect downstream hypotheses, the figure above cannot pretend to fully capture the realities of activity implementation.

Rather, this extended logic model is designed to identify hypotheses that contribute to mission learning about what works and what doesn’t, but also recognizes that actual interactions from implementation will likely remain too complex to be fully diagrammed. These constituent hypotheses referenced in the figure above are presented in the table below.

Table 2 CITIES Evaluation hypotheses for measuring effectiveness Hypothesis 1 If CITIES provides institutional capacity building, then municipal operations will improve.

Hypothesis 2 If CITIES provides institutional capacity building, combined with incentive grants and improved public service delivery, then this will lead to an improvement in the quality of public services.

Hypothesis 3 If CITIES provides institutional capacity building, combined with proactive outreach by municipalities, then this will increase public perception that municipalities are responsive to citizen needs.

Hypothesis 4 If CITIES supports increased attention to marginalized groups, such as youth, gender, nationality, and disability, then this will strengthen a sense of community cohesion and resilience.

Hypothesis 5 If CITIES provides institutional capacity building, combined with proactive outreach and increased / improved public service delivery, then this will increase public perception that municipalities are effective.

Hypothesis 6 More effective public service delivery, responsiveness to citizen needs, and strengthened community cohesion and resilience will increase citizen confidence in their municipality.

Hypothesis 7 More effective public service delivery, confidence in local leadership and performance, and strengthened community cohesion and resilience will lead to a more stable and prosperous Jordan.

See below for the proposed evaluation indicators to measure each hypothesis.

EVALUATION DESIGNS FOR CAUSAL INFERENCE

This section presents the primary inferential design features of the evaluation. The base model will employ difference-in-differences across treatment and comparison municipalities, and difference-in-differences with the addition of a spillover group within intervention municipalities. These models assume that the measured units are repeated cross-sections of a larger population; in the event that the same units are tracked over time, the evaluation team will employ more efficient models that exploit the advantages of tracking the same units over time. Learning agenda questions will be addressed by multivariate analysis and/or qualitative inquiry, but without a specific inferential design.

Difference-in-differences

The basic measurement to estimate outcomes will follow a difference-in-differences (d-i-d) approach, in which a sample of both treated and untreated units (in this case, municipalities) are examined before (baseline) and after (endline) treatment. Examining the first differences in outcomes across both groups (trends) rather than at single points in time (levels) has the important effect of removing any unobserved sources of bias in the data that are common to both groups and remain constant over time. The difference between the trend lines of each group (the second difference) then generates an estimate of the overall treatment effect. Diagrammatically, the design is as follows:

Figure 5 Difference-in-differences (d-i-d) evaluation design

Analytically, the d-i-d model may be expressed as follows:

In this model, is the outcome in the comparison group at baseline, is the difference between the treatment and comparison groups at baseline, and is the change across baseline and endline in both treatment and comparison groups that is unrelated to CITIES. The overall treatment effect of the intervention is , or the interaction of treatment and time. One may also include a set of control β β δ δ υ= + + + ⋅ + Β Χ +0 1 0 1 i iy Treatment Endline Treatment Endline β0 β1 δ0 δ1 variables, , that help statistically adjust for any correlation between observable characteristics and outcomes.11 The final term is a theoretical construct representing all dynamics or factors that are unobserved by the analytical model. As this term is unobserved, it must be theorized by the evaluator to assess how the causal effect estimates may break down in a real world setting. If there are any dynamics or factors correlated with outcomes but not captured by first differences or the set of control variables, then this model may produce misleading or unreliable estimates. Additionally, if the treatment and comparison groups would follow different trend lines over time in the absence of treatment, then the d-i-d estimator fails to produce unbiased measures.12

Difference-in-differences with spillover

In the previous model, municipalities are the basic unit of treatment and municipal level measures are compared against measures from other, untreated municipalities. This design becomes problematic when examining activity outcomes measured amongst citizens, as there would be a significant proportion of a municipal population that remain untouched by CITIES activities. The evaluator must then limit the study population to direct beneficiaries within municipalities.

However, it is of programmatic interest to identify the boundaries of CITIES effects. That is, how much service delivery or other improvement is needed before a local governance initiative begins to affect entire populations within a service delivery area? Is there a ‘contagion’ effect of improved municipal operations affecting some beneficiaries which begins to positively affect other residents within the municipality? In order to address such dynamics, the beneficiary population of a municipality is partitioned into direct beneficiaries exposed to CITIES activities, and indirect beneficiaries who live within the municipality but were not directly exposed. The population of indirect beneficiaries can be further partitioned into those who have witnessed the implementation of activities and/or heard about them through media or social networks.

Questions of indirect exposure will be addressed at the data analysis stage, but for purposes of illustration indirect beneficiaries within a treated municipality may be considered a part of a spillover group in order to investigate whether and how programmatic inputs diffuse through a beneficiary population. Diagrammatically, the d-i-d with spillover model may be expressed as follows:

11 It is critical to show that any treatment effect remains stable despite the inclusion or exclusion of control variables. However, care must be taken in handling control variables, as researchers may also be tempted to manipulate different combinations of controls to spuriously maximize the treatment effect.

12 A more recent development in CITIES programming calls for capacity building training in citizen engagement to all 100 municipalities (see CITIES Year 2 Workplan, received 5 Dec 2017). This activity will be tracked closely to assess implications for the evaluation design.

iX υ

Figure 6 Difference-in-differences with spillover group

Analytically, the d-i-d with spillover model may be expressed as follows:

This model is identical to the d-i-d model, but also adds two terms. The term represents differences at baseline between citizens in comparison municipalities and citizens in treated municipalities who are not direct beneficiaries of CITIES activities.13 The term represents the spillover effect, if any, of

CITIES activities on municipal residents. Citizens subject to spillover effects may not be direct beneficiaries of CITIES activities, but may nevertheless show progress on CITIES outcomes as a result of observing the effects of activities, or hearing about the implementation of activities through media and/or social networks. Note that a member of the spillover group would be undefined until the endline data collection when beneficiary households, citizens, or neighborhood/blocks are identified.

This specification has the effect of estimating three distinct trend lines with two separate treatment effects. The primary treatment effect is the difference in trends between the treatment and comparison

13 Note that “CITIES activities” is shorthand for any activity attributed to CITIES. A part of the CITIES theory of change is that capacity building will induce more and/or better quality investment in public services and operations.

Therefore, “CITIES activities” may include municipality activities or investments that are thought to be induced by CITIES involvement.

0 1 2 0 1 2 i iy Treat Spill End Treat End Spill Endβ β β δ δ δ υ= + + + + ⋅ + ⋅ + ΒΧ +

2β

2δ group. The secondary (spillover) treatment effect is the difference in trends between the spillover and comparison group. In the event that spillover effects are found, then the spillover and treatment groups can be combined into a single treatment group that estimates a general equilibrium treatment effect that is a weighted average of the direct and indirect beneficiary groups.

These base estimation models assume independent repeated samples of the population of interest, with analysis conducted at the individual or household level such that the individual units of measurement are not repeated. In the event that subsequent measurements interview the same individuals or households, or the data are aggregated to track larger units such as neighborhoods or entire municipalities, then the estimation model can be modified to include the baseline outcome measure for each measured unit.

This is a more efficient model in that it can control for the baseline outcome for each measured unit, while the estimation models described above take only the group means of the baseline for all measured units. This more efficient model is expressed as:

In this model, is the baseline outcome for each individual, is the treatment effect, and allows for more efficient estimates of the treatment effect in that it statistically adjusts for the level of the outcome at baseline, while the base specifications mentioned above do not statistically adjust for the level of the outcome measure at baseline.14 This more efficient specification can be extended to include spillover groups by adding terms for the baseline level of the spillover group and subsequent gain score of the spillover group.

Control variables

Control variables in an estimating equation serve two purposes. First, given that treatment status is not randomly assigned, there will inevitably be sources of bias in the data that obscure the true treatment effect of CITIES activities. Wherever those sources of bias are observed, they must be included as control variables so that the obscuring effect may be partitioned away from the estimate of the treatment effect. Second, even if they don’t pose a threat to the validity of the treatment effect estimate, adding other control variables into the model helps improve precision of the estimates, making it easier to detect a treatment effect.

For the CITIES evaluation, dynamics that would need to be included in the model include current and past government and donor activity in the area of local governance, decentralization, and service delivery. Dynamics to improve estimate precision would be background variables at the individual level such as gender, age, and education, or background variables at the cluster level such as average levels of income or level of services. All variables used to apply the matching routine should also be included to statistically adjust for any remaining differences between treatment and comparison.

The evaluation team may develop additional controls as the evaluation proceeds. For example, municipal service delivery activity types and timelines induced by CITIES participation will be unique to each

14 Note however that the two different models have different ways of accounting for baseline differences between treatment and comparison groups, which can sometimes result in substantially different treatment effect estimates.

See McKenzie (2011) and Allison (1990) for discussion.

ii i PRE i iy y Treatδ θ γ υ= + + + iδ iγ iθ municipality. The estimating equations may benefit from tuning parameters that control for varying activity types and timelines captured within the same survey wave. Another example is that the progress of Jordan’s national decentralization agenda will likely depend on the nature and quality of relationships between government officials, MoI, MoMA, and MoPIC. Similarly, the progress of CITIES may depend on the quality of relationships between CITIES leadership and municipal leadership and staff. Additionally, the quality of relationships is also likely correlated with the relationships between municipalities and other layers in the local governance structure including governorate and executive councils, and with partner ministries. The evaluation team will also attempt to control for other donor activities. While respondents may not be aware of which donors fund specific activities, collecting data on and controlling for the presence of other programs will be critical for understanding potential variation between municipalities and understanding how the presence of other donors may affect key indicators.

As the opportunity arises, the evaluation team will gather qualitative information about the political economy of relationships between institutions and officials in order to help understand the implementation environment and assess its potential effects on the inferential design. This information will be collected through a qualitative data collection template and then compiled into a factor or some other quantitative data type to allow for the inclusion of these critical features of decentralization.

EVALUATION INDICATORS

MESP proposes a set of 10 evaluation indicators measuring seven primary hypotheses. Each of these have been mapped to the evaluation questions (presented here by their themes for brevity), the data source, disaggregation approach, and analysis method. Evaluation question four, Learning, is captured by the evaluation learning agenda. Annex 3 provides more detail on the specific survey questions that will measure each indicator. The proposed indicators are as follows:

Table 3 Evaluation indicators Evaluation

Question/ Theme

Hypothesis Indicators Data Source Disaggregation Data Analysis Method

1. Effectiveness Hypothesis 1: Institutional capacity building will improve municipal operations.

- MICA Services Score

- MICA GESI Score

- Municipality Revenue

- Budget allocation to salaries

- MICA assessment data

- Available municipality administrative data

- KIIs

Municipality, district, governorate, region

- d-i-d

1. Effectiveness Hypothesis 2: Institutional capacity building, combined with incentive grants, will improve the quality of public services.

- Availability of services

- Perceived quality/availability of services

- Beneficiary survey

- Population survey

- KIIs

- FGDs

Service, municipality, district, governorate, region, gender, age, education

- d-i-d

- d-i-d with spillover

2. Sustainability

Hypothesis 3: Institutional capacity building, combined with proactive outreach by municipalities, will increase public perception that municipalities are responsive to citizen needs.

Perceived municipal responsiveness15

- MICA assessment data

- Beneficiary survey

- Population survey

- FGDs

- KIIs

Municipality, district, governorate, region, gender, age, education

- d-i-d

- d-i-d with spillover

- Pattern/Content

Analysis

Hypothesis 4: Increased attention to marginalized groups such as youth, gender, nationality, and disability will strengthen a sense of community cohesion and resilience.

Perceived municipal effectiveness16

- Beneficiary survey

- Population survey

- FGDs

- KIIs gender, age, education

- d-i-d

- d-i-d with spillover

- Pattern/Content

Analysis

1. Effectiveness

3. Synergies

Hypothesis 5: Institutional capacity building, combined with proactive outreach and increased / improved public service delivery, will increase public perception that municipalities are effective.

Perceived ability to address common problems17

- Beneficiary survey

- Population survey

- FGDs

- KIIs

- Activity and municipality

Municipality, district, governorate, region, gender, age, education

- d-i-d

- d-i-d with spillover

15 Municipal responsiveness can be defined as municipality governance listening to and addressing peoples’ needs in an open and efficient manner.

16 Municipal effectiveness can be defined by perceptions of the degree to which municipal leaders and administrators can formulate and implement policies that address citizen needs, the perceived quality and maintenance of existing and requested public services, and the perceived credibility of the government to follow through on citizen needs.

17 The ability to address common problems refers to the perception of how political, tribal, gender, religious, or other affiliations influence a municipality’s response to citizen needs and requests, as well as the ability of municipality leadership to coordinate without inequitable concession to these affiliations.

Evaluation Question/ Theme

Hypothesis Indicators Data Source Disaggregation Data Analysis

Method documents

1. Effectiveness

2. Sustainability

3. Synergies

Hypothesis 6: Improved public service delivery, increased public perception of municipal responsiveness and effectiveness, and a stronger sense of community cohesion and resilience will lead to an increase in public confidence in municipal / local governance.

- Confidence in local government

- Trust in local government

- Beneficiary survey

- Population survey

- FGDs

- KIIs

- Other activity documents gender, age, education

- d-i-d

- d-i-d with spillover

- Pattern/Content

Analysis

1. Effectiveness

2. Sustainability

3. Synergies

Hypothesis 7: More effective public service delivery, confidence in local leadership and performance, and strengthened community cohesion and resilience will lead to a more stable and prosperous Jordan.

- Quality of life

- Composite measures

- Beneficiary survey

- Population survey

- FGDs

- KIIs

- Other activity documents gender, age, education

- d-i-d

- d-i-d with spillover

Subsequent analytical reports (baseline, midline, endline) will go into additional detail about the construction of measures and indices. For example, not all data collected in the municipal institutional assessments have direct attributional links to CITIES performance, but are rather gathered for contextual purposes. In this case, the evaluation team will construct suitable indices from a subset of MICA data and use the non-performance measures as control variables (see Annex 4). Similarly, high-level measures such as effectiveness, confidence, and stability may be measured as composite indices of multiple survey items. In this case, the analytical reports will provide a detailed explanation of index construction and analysis.

LEARNING AGENDA

The CITIES Learning Agenda operationalizes the overall Learning evaluation question, and is meant to go beyond questions of whether or not the activity worked to include insight on the mechanics of how or why outcomes were or were not achieved, and whether outcomes are heterogeneous across variation in implementation or beneficiary characteristics. According to the Office of Management and Budget:

A learning agenda is a set of broad questions directly related to the work that an agency conducts that, when answered, enables the agency to work more effectively and efficiently, particularly pertaining to evaluation, evidence, and decision-making. Once the questions are identified, a learning agenda also prioritizes and establishes a plan to answer short- and long-term questions of the highest value across relevant program and policy areas.

The USAID Bureau of Program Planning and Learning (PPL) extends this definition to describe a learning agenda as 1) a set of questions, prioritized and often organized around thematic areas, 2) a set of learning activities such as research, evaluation, literature review, stocktaking sessions, etc, and 3) a set of learning products that review and synthesize the evidence. Learning products may be accompanied by targeted and pithy communications such as infographics to highlight specific findings.

In this context, the CITIES evaluation may be seen as its own learning activity, or as a program of ongoing learning activities throughout the period of performance that will culminate with impact assessment. As learning activities cumulatively gather data, context, and insight, the evaluation team will seek to address questions that are meaningful to USAID managers and leaders and practically useful for understanding the implementation environment and supporting subsequent activity design. The table below presents the primary learning agenda questions, followed by secondary learning agenda questions.

Table 4 Primary learning agenda questions

Question(s) What is the level of citizen awareness and knowledge of Jordan’s decentralization agenda? At what level of governance does subsidiarity [decentralization] most effectively reside—local councils, municipal councils, or governorate councils? What is the locus of control for all three levels of government?

Analytical Approach and Output

The general population survey will ask where Jordanians would turn when they require assistance, as well as other data on actual citizen preferences and behavior. Reflection on this data will include a comparison between the functions assigned to these councils by law and regulation, the functions actually executed by councils, and the actual preferences of citizens.

Question(s)

What is the sufficient dose of programmatic inputs before we see changes in municipal operations, improved service delivery, and citizen perception of service delivery? Does the combination of capacity building and incentive grants induce more, and/or more effective, investment in public services? What is the sufficient dose of institutional capacity building to induce additional investment? To what extent are these investments a function of the political context with higher levels of government or a response to programmatic inputs?

Analytical Approach and Output

In addition to a binary 1/0 treatment variable to measure the average effect on beneficiaries, the evaluation team will develop a multi-valued treatment variable that reflects the scope and intensity of programmatic inputs. Such a variable could be as simple as an activity count per area or municipality, or could be a more sophisticated construct that captures more nuance of the types and magnitudes of inputs, such as provision of office and other equipment. The analysis will then indicate a minimum level of input before outcomes are realized, or the level of input needed to reach a desired magnitude of outcome. Qualitative work and political economy analysis will provide an assessment of the decision making context that affects service delivery.

Question(s)

Is there a “J-curve” in which awareness raising drives citizen expectations, which lower perceptions of responsiveness and effectiveness until citizens see tangible improvements in service delivery? What are the relative risk/rewards in pursuing such a strategy? If there is a “J-curve,” what is the time frame before citizen perceptions improve over baseline as a result of tangible improvements in service delivery?

Analytical Approach and Output

The evaluation team will seek multiple post-treatment measurements in order to trace out a time-path of outcome trajectories – for example, the general public survey and institutional assessments will be conducted in both 2020 and 2021. This will serve two purposes: first, to identify whether there may be short to medium term deficits in outcome measures based on, for example, unrealized citizen expectations or disruption of bureaucratic processes before improvements kick in; and second, multiple post-treatment measurements will provide insight as to the time frame at which outcomes increase or attrite over time.

Data collection to respond to this learning agenda question will need to occur over a minimum of three time periods to capture potential changes.

Question(s)

Does varying levels of female and youth membership in local and municipal councils affect municipal operations or service delivery? What about varying levels of female and youth employees in the municipal bureaucracy? Is there a relationship between either of these dynamics and citizen perceptions of council responsiveness and effectiveness? Does more female and youth membership of local council bodies engender increased engagement and participation among women and youth?

Analytical Approach and Output

The evaluation team will collect information on the gender composition of local and municipal councils and cross-reference this information against citizen perception and institutional capacity scores. This process will repeat for municipal bureaucratic staff. Household and beneficiary survey data will also be disaggregated by gender to see if there are differential effects of council gender composition.

Question(s) What is the “design space” of CITIES implementation? Does it vary within or across municipalities?

Analytical Approach and Output The evaluation team will track differences in activity implementation across municipalities and attempt to isolate whether these different design choices affected outcomes.

In addition to the…