Attachment_J-14_Draft_PMP_for_the_2016-2021_CDCS.pdf

USAID/UGANDA PERFORMANCE

MANAGEMENT PLAN 2016-2021

Last Updated: February 21, 2018

(DELETE THIS BLANK PAGE AFTER CREATING PDF. IT’S HERE TO MAKE FACING PAGES

AND LEFT/RIGHT PAGE NUMBERS SEQUENCE CORRECTLY IN WORD. BE CAREFUL TO

NOT DELETE THIS SECTION BREAK EITHER, UNTIL AFTER YOU HAVE GENERATED A FINAL

PDF. IT WILL THROW OFF THE LEFT/RIGHT PAGE LAYOUT.)

I

PMP UPDATE INFORMATION: READ ME FIRST

All iterations of the Performance Management Plan (PMP) will be stamped with a “Last Updated” date on the front cover of the document. Before reading this document further, ensure you are working from the most recent version of the PMP.

ACCESSING THE PMP

Users can access the latest version at (insert URL) or by contacting the Performance Management Point of Contact, FirstName LastName, at nameid@usaid.gov. The Program and Project Development Office will ensure that only the most recent version is linked at the file path above, and that archived versions will be clearly marked. A list of updates to the PMP will be included in the text of the final PMP document (see below). External users should contact the Performance Management Point of Contact, FirstName LastName, at nameid@usaid.gov to obtain the most recent version. The following table notes all archived versions of the PMP.

WHEN TO USE THE PMP

Project and Activity Design: This PMP provides key definitions, indicators at IR and Sub-IR levels, and details of planned evaluations and research as a planning tool for project and activity design teams.

Collaborating, Learning, and Adapting: This PMP contains a CLA plan to apply throughout the Program Cycle and an explanation of the Mission’s approach to Learning and the program cycle including Portfolio Reviews and CDCS Stocktaking.

Managing an Integrated1 Strategy: Tools and methods are proposed to explore geographic co-location, integration across funding streams, and the application of the Mission’s architectural approach to managing performance. In addition, resources are provided to help Implementing Partners (IPs), the Government of Uganda (GOU), and other partners understand the Program Cycle and their role in achieving the goal of the CDCS to reach its 2040 goal of sustainable and inclusive Uganda-led development.

1 See ‘Integration and Geo-focus Spectrum’ in the 2016-2021 CDCS and a definition in the glossary.

PMP UPDATES LOG

DOCUMENT DATE UPDATES ARCHIVED

Text Goes Here Table Text Goes Here

Text Goes Here Table Text Goes Here

Are you reading the most recent version of the

PMP?

mailto:nameid@usaid.gov mailto:nameid@usaid.gov

II

I

ABBREVIATIONS, ACRONYMS, AND INITIALISMS

ADS Automated Directives System

AOR Agreement Officer’s Representative

CDCS Country Development Cooperation Strategy

CDCS 2.0 Country Development Cooperation Strategy 2016-2021 – The current strategy USAID/Uganda

CLA Collaborating, Learning and Adapting

CO Contracting Officer

COR Contracting Officer’s Representative

CSO Civil Society Organization

DO Development Objective

DRG Democracy, Human Rights and Governance

EG Economic Growth (refers to Mission technical team)

ENR Environment and Natural Resources

EPOC Evaluation Point of Contact (in PPD)

EYCD Education, Youth, and Child Development

FTF Feed the Future

GCC Global Climate Change

GIS Geographic Information System

GOU Government of Uganda

HH Household

HL Health (refers to Mission technical team)

HRH Human Resources for Health

HSS Health Systems Strengthening

HWC Human-Wildlife Conflict

IR Intermediate Result

LGBTI Lesbian, Gay, Bisexual, Transgender, and Intersex

M&E Monitoring and Evaluation

MCH Maternal and Child Health

MEL Monitoring, Evaluation, and Learning

MLC Mission Leadership Council

MLQ Multifactor Leadership Questionnaire

NRM Natural Resource Management

PAD Project Appraisal Document

PEPFAR U.S. President’s Emergency Plan for AIDS Relief

PM Permanent Methods

II

PMI Presidential Malaria Initiative

PMP Performance Management Plan

PMPOC Performance Management Point of Contact (in PPD)

PPD Program and Project Development Office

PPR Performance Plan and Report

PPL/LER Policy, Planning, and Learning’s Office of Learning, Evaluation, and Research

PRS Performance Reporting System

QCA Qualitative Comparative Analysis

QED USAID/Uganda Monitoring, Evaluation and Learning Program (The Learning Contract), implemented by the QED Group LLC

RF Results Framework

SRGBV School Related Gender Based Violence

STIP Science, Technology, Innovation, and Partnership

USG US Government

WASH Water, Sanitation, and Hygiene

WFP United Nations World Food Programme

WHO World Health Organization

CONTENTS

PMP UPDATE INFORMATION: READ ME FIRST I

ACCESSING THE PMP I

WHEN TO USE THE PMP I

ABBREVIATIONS, ACRONYMS, AND INITIALISMS I

CONCEPTUAL FRAMEWORK 5

BACKGROUND 7

STRATEGIC LEARNING FOR PERFORMANCE MANAGEMENT 11

PERFORMANCE MEASUREMENT AND MANAGEMENT ARCHITECTURE 17

DEVELOPMENT OBJECTIVE 1 MONITORING PLAN 23

DEVELOPMENT OBJECTIVE 1 LEARNING QUESTIONS 24

DEVELOPMENT OBJECTIVE 1 APPROACH TO MEASUREMENT AND ANALYSIS 25

DEVELOPMENT OBJECTIVE 1 INDICATORS 27

MONITORING OF PROGRAMMATIC ASSUMPTIONS 29

DEVELOPMENT OBJECTIVE 2 MONITORING PLAN 32

DEVELOPMENT OBJECTIVE 2 LEARNING QUESTIONS 33

DEVELOPMENT OBJECTIVE 2 APPROACH TO MEASUREMENT AND ANALYSIS 34

DEVELOPMENT OBJECTIVE 2 INDICATORS 36

MONITORING OF PROGRAMMATIC ASSUMPTIONS AND CONTEXT 38

DEVELOPMENT OBJECTIVE 3 MONITORING PLAN 41

DEVELOPMENT OBJECTIVE 3 LEARNING QUESTIONS 42

DEVELOPMENT OBJECTIVE 3 APPROACH TO MEASUREMENT AND ANALYSIS 43

DEVELOPMENT OBJECTIVE 3 INDICATORS 47

MONITORING OF PROGRAMMATIC ASSUMPTIONS AND CONTEXT 50

GUIDING PRINCIPLES MONITORING PLAN 53

EVALUATION PLAN 58

REQUIREMENTS SUMMARY 58

COLLABORATING, LEARNING, AND ADAPTING (CLA) PLAN 68

EXPANDING THE LOCUS OF LEARNING 68

COLLABORATION PRIORITIES 69

LEARNING PRIORITIES 69

ADAPTIVE MANAGEMENT PRIORITIES 70

ORGANIZATIONAL CULTURE 71

COLLABORATING, LEARNING, AND ADAPTING PLAN TASK AND RESOURCES 72

RESOURCES FOR PERFORMANCE MANAGEMENT TASKS 76

ANNEX A: GLOSSARY OF PMP TERMS AND DEFINITIONS 79

ANNEX B: LEARNING QUESTIONS 98

ANNEX C: THE 14-YEAR-OLD GIRL 111

ANNEX D: LEARNING IN THE PROGRAM CYCLE 114

ANNEX E: STANDARD INDICATOR ALIGNMENT 115

ANNEX F: TRUTH TABLE 127

ANNEX G: BAYESIAN NETWORK MODEL 128

ANNEX H: PERFORMANCE MANAGEMENT WHITE PAPERS 129

WHITE PAPER 1: THE USE OF MEASUREMENT MODELS TO BUILD INDEXES FOR MEASURING

PROGRESS ON THE CDCS 129

WHITE PAPER 2: QUICK OVERVIEW OF QUALITATIVE COMPARATIVE ANALYSIS AND ITS

RELEVANCE TO USAID/UGANDA 135

CONCEPTUAL FRAMEWORK

The Performance Management Plan (PMP) is USAID/Uganda’s plan for adaptive management. This PMP represents a break from tradition. Functioning as a “living document” per Automated Directives System (ADS) 201 guidance, this PMP combines traditional performance monitoring with learning-driven research to support continuous adaptive management, as shown by the intersections of the Venn Diagram below:

Collaboration, Learning, and Adaptation (CLA) sit at the intersection of this convergence2. The CLA Plan offers a platform to bring together the various learning agenda items, identify a collaborative plan for how the issues and questions will be addressed through various types of action-driven research and performance monitoring, and in line with the Mission’s Guiding Principles, it informs adaptive management. The CLA Plan is complemented by a methodological toolkit of indicators which balance qualitative, quantitative, and mixed analytical perspectives.

Moreover, this PMP is designed to enhance traditional performance monitoring with real-time context monitoring and a more comprehensive performance management architecture which considers data support for decision-making. With these elements combined, monitoring and

2 The intention here is to show those major overlapping elements and concepts contained within this Performance Management Plan. Elements which fall in more than one circle, example Learning Agenda, are those elements which are relevant to multiple areas of performance management. This diagram is not exhaustive and could be conceived in variant forms.

evaluation (M&E) becomes a powerful tool for adaptation in the shifting contexts defining USAID’s implementation environment.

One key locus of change the Mission has already identified is the concept of the “14-year-old girl,” who sits at the intersection of many of Uganda’s most persistent development challenges. Research will identify the underlying drivers that influence her decisions and preferences, which in turn will guide programming and help to design interventions that are at the core of USAID/Uganda’s development objectives and strategic success. Finally, periodic evaluation will also allow the Mission to take a step back and review progress for both accountability and to emphasize and enable continuous learning.

These three overlapping concepts of performance monitoring, learning agenda research, and adaptive management underpin the PMP structure. They help the user understand how a single PMP activity – such as collecting context data – can be used to support multiple purposes.

Thus, the PMP does not only measure if we do things right, but also if we do the right things.

MEASURING THE LAYERING OF ACTIVITIES

The CDCS assumes that USAID will be able to achieve greater results through the complementarity of programs: if we co-locate activities and layer programs with different interventions on the same subject, target population or geographic area, we’ll achieve better results. The PMP tests cross-sectoral, cross-activity, and even cross-project assumptions, to assess the impact of USAID’s approach, using compatible units of measurement. The PMP selects some of the highest level results and lays out a management plan for monitoring, evaluating, and learning at various levels to ultimately assess the Mission’s progress towards the longer term objectives and goals. Implementing partners will monitor short-term progress and performance; whereas USAID projects, technical offices, and MEL support mechanisms will be charged with analyzing effects between multiple activities, long-term impacts, indices of development objective (DO) performance, and projections of changes in context.

When USAID activities address issues across sectors on community, district, and national level, the combined results will be measured by looking at changes in context data3, assuming USAID impact is

3 To distinguish context data from performance data, we define context data as observations on the general status or trend of certain variables within the geographic, or theoretically connected space of our certain variables of interest. For example, we may have an intervention and program targeting early grade reading in five schools in one district, and we monitor the results for our students in our schools. We measure our performance on the schools and students whom we helped, but we also need to know if those interventions together with better sanitation facilities at school, changed socio-economic behaviors and a youth-focus in Family Planning activities could have improved girls’ retention in school. The data for girls’ retention will be pulled from the Ministry of Education’s database. Context data could be gathered for national surveys, district level data beyond our targeted program area, or data on variables related to our target population - but which we are not directly affecting.

greater than the aggregated sum of performance indicators. For example, a number of activities across projects will contribute to child well-being. Activities will collect data on the standard indicators, outputs, and outcomes. At the Sub-IR level, a combined result of a few activities is expected, sometimes at the household, community, district, or regional level. Each of these activities then contribute to the IR, which feeds into the larger development objective, where USAID may impact the related development context.

The visual below depicts that there are various levels of analysis used. The highest level of complexity, Level 4, doesn’t replace Basic Level and Level 2 single and bi-variate measures, but complements the lower levels and builds on these levels to better analyze an integrated, more complex framework.

BACKGROUND

The United States Agency for International Development (USAID)/Uganda developed this PMP as a single, Mission-wide resource to assist USAID/Uganda in achieving its highest development goal and objectives.

This PMP reflects USAID/Uganda’s current 2016-2021 CDCS, which was approved on December 6, 2016 and covers the years 2016-2021. The Mission has a 5-year goal under this CDCS which will serve as the foundation for the longer-term achievement of the 25-year goal of the RF:

5-Year Goal: Uganda’s Systems Are Accelerating Inclusive Education, Health, and Economic Outcomes

25-Year Goal: Uganda-Led Inclusive and Sustainable Development

Level 1. Basic

•Mostly outcome level •Directly influenced by a single activity •Short-term (1-2 years)

Level 2. Measure effect

•Mostly outcome level •A result of 2-3 activities •Semi complex •Medium time frame

(2-3 years)

Level 3. Complex attribution

•Mostly long-term/impact level •A result of multiple activities •Complex •Longer-term time frame (3-5 years)

Level 4. Very complex attribution

•Mostly long-term outcome/impact level •A result of multiple actors and context sensitive •Composite and complex •Longer-term timeframe (3-5 years)

Multi-variate analysis

(Regression, ANOVA, etc.)

Bi-variate analysis.

Cross -tabulation

Single variable tabulation

The CDCS articulates three Development Objectives (DOs) which reflect the key development hypotheses that will inform USAID/Uganda’s projects and activities over the CDCS period. These DOs are:

DO 1: Community and Household (HH) Resilience Increased

DO 2: Demographic Drivers Affected to Contribute to a Long-Term Trend Shift

DO 3: Key Systems More Accountable and Responsive to Uganda’s Development Needs

USAID/Uganda’s strategic approach in CDCS 2016-2021 is designed to generate immediate returns that are linked to long-term sustainable development. Root causes and constraints affecting Ugandans’ everyday lives are inherently interlinked, calling for a more holistic approach to development. USAID/Uganda has elected a systems approach, recognizing that interrelated factors affect each other within and across sectors. Thus, the CDCS directs development support to leverage points within and between systems to multiply the effects of results. Building on experience and lessons learned from implementing the previous CDCS, including various integrated interventions and evaluations of results, the CDCS 2016-2021 presents a fully integrated framework that aims at structuring and building long-term partnerships to ultimately achieve Uganda-led inclusive and sustainable development.

Results Framework and Development Hypothesis

The main development hypothesis in the strategy is twofold. First, if community and household resilience in select areas and target populations is increased, and demographic drivers are affected to contribute to a long-term trend shift including a positive demographic dividend, and key country systems are more accountable and responsive to Uganda’s development needs, then Uganda’s systems will accelerate inclusive education, health, and economic development. This hypothesis assumes mutual interdependency among the development objectives. If the key systems (health, social protection, education, market systems governance and accountability, and natural resources management) are more accountable and responsive, then community and household resilience will increase and the demographic drivers will be affected. Conversely, if household and community resilience is increased and the demographic drivers are affected, the systems will be strengthened to become more responsive and accountable.

The relationships between the DOs have implications for the RF.4 The reinforcing relationships among the DOs flow down to their constituent IRs across the entire RF both vertically and horizontally. For example, child well-being will be accelerated and change will be more sustainable if leadership is developed and improved. These linkages are visualized on the front page of this PMP.

Relationships between IRs within each DO are also important and complementary. Each IR and Sub- IR build upon each other to achieve a higher outcome, the DO.

The development hypothesis is further underpinned by the assumption that the Mission’s approach to development collaboration, transparency and accountability, inclusivity, and the ‘how’ of implementation is just as important as resource allocation.

This framework encourages adaptive management for program leaders to identify opportunities where specific linkages (depicted by arrows on the front page) show promise to accelerated results of positive change. All the while, it enables USAID priority shifts in response to political considerations, specifically changing or fluid Government interest in program areas due to patrimonial dynamics. Thus, because USAID can adapt and shift priorities, intractable roadblocks can be avoided while continuing to pursue similar outcomes.

The Guiding Principles

The Guiding Principles are a set of 15 key concepts that will help the Mission to govern how USAID conducts business throughout the implementation of the 2016- 2021 strategy. USAID will learn, through monitoring the application of the Guiding Principles, what effect these have on achieving results. For example, USAID will track the number of programs changed due to collaboration and learning to ensure that CLA drives adaptive management. However, to move beyond accountability, USAID will observe those partnerships in relation to specific IR or DO outcomes and ask: what contribution, if any, does the adherence to these Principles have on result achievement? USAID will continuously track the application of the Guiding Principles as they are recorded in Mission processes, and in projects and activities. 5 Projects and activities in implementation will reflect and learn from their application of the Principles while conducting activity-based learning reviews. The entire

4 See Pardee Center (2015), “Understanding development patterns, trends, and relationships in Uganda to support the USAID Country Development Coordination Strategy 2016-21,” internal report, for a clear illustration of how the interrelations of development objectives interact over time across a number of plausible scenarios.

5 See Guiding Principles Monitoring Plan.

Select example Guiding Principles

Harness youth-appropriate approaches so that they are included in all that we do

Infuse inclusive development – including, but not limited to, gender empowerment – throughout the program cycles

Prioritize partnerships that enable Ugandan-led development, while leveraging USAID’s limited resources

Pursue integrated approaches at various levels – program, project, activity, intervention – when and how it makes sense in achieving greater effectiveness in our work

Mission will collect data on the application of the Guiding Principles through regular portfolio reviews, and internalize and reflect on lessons learned during annual retreats. Pause and reflect moments around accountability and learning will enable better adherence to the Guiding Principles and ultimately improved performance. The select example Guiding Principles in the box above are also directly aligned to distinct, implicit theories of change which for strategic choices in the 2017- 2021 CDCS. As such, they are further explored below.

The 14-Year-Old Girl

The 14-year-old girl has a special role in the CDCS and her vulnerable situation will form a lens through which we measure progress across the DOs. The PMP has been designed with her in mind, in that if she is able to better access services, benefit from improved policies, and enjoy more prosperous economic opportunities, her household and community will also benefit. But the 14-year-old girl’s situation is fragile and structural biases will continue to allow her to fall behind.

Development policies, as they currently operate, run the risk of failing to adequately address her situation. In contrast, if programming decisions focus on the 14-year-old girl attention will be paid to the mechanisms that allow her, her household, and her community to thrive. The 14-year-old girl is an important marker for measuring progress throughout Uganda. A more detailed description of the role the 14-year-old girl plays in relation to the CDCS can be found in Annex C.

STRATEGIC LEARNING FOR PERFORMANCE MANAGEMENT

The PMP “is a Mission-wide tool for planning and managing the processes of monitoring strategic progress, project performance, programmatic assumptions, and operational context; evaluating performance and impact; and learning and adapting from evidence,” (ADS 201.3.2.16). By initially identifying the learning questions and then developing methods and indicators to find answers to our questions, the PMP will truly inform adaptive management. Below are the Mission’s main learning questions pertaining to the CDCS and the new integrated approach. Each DO section also has learning questions outlined in each of the relevant sections that outline questions that drive the learning of the RF.

Overall, we want to test the theories of change6. The main questions that need to be explored are:

Strategic Learning Questions for Adaptation

Data required and methods of analysis

What are the entry points for interventions that trigger change and promote development outcomes for adolescent girls in target households?

USAID will use project and activity data to conduct qualitative comparative analysis and factor analysis. By using performance and context data, the Mission will understand what interventions and conditions improve the 14-yearold girl’s situation. This analysis will be conducted annually and inform adaptive management in both projects and activities.

What are the circumstances or conditions under which USAID activities promote local and national solutions?

USAID will use activity implementation data and descriptive analysis. In particular, data from IR 3.1: Leadership Developed, and a number of Guiding Principles will be used in analysis of quantitative data, including QCA analysis. A research question will also be included in mid-term evaluation on leadership development.

How and to what extent did projects or activities remove barriers for marginalized groups to participate in their own development outcomes?

USAID will use primary data collection at project and activity level to communicate descriptive statistics and conduct comparative analysis.

Outcome data that has been disaggregated by sex, age, and wealth quintile will be used as well as data from the Guiding Principles. Findings will inform portfolio reviews and qualitative studies in geographic areas where key activities in DO1 and DO2 operate.

Did USAID integrate programming at all levels (strategy, project, activity), and how did that integration improve outcomes and enhance effectiveness?

USAID will use project and activity implementation data and context data to conduct configuration analysis and QCA. The Whole of Project Evaluation should also provide findings that helps answering the question.

6 Each of the ‘Strategic Learning Questions for Adaptation’ in the left column are aligned to distinct, implicit theories of change, as articulated in the 2016-2021 CDCS. They, respectively, correspond to concepts regarding: the 14-year-old girl;

working through Uganda’s local systems; marginalization and inclusive development; and the hypothesis on the added value of integration.

Methods Overview

The 2016-2021 CDCS requires a PMP that combines traditional M&E methods with a new set of approaches to data collection, measurement, and analysis. This PMP lays out a framework that monitors progress, unpacks and manages causal complexity, and ultimately builds on evidence that suggests cross-cutting, integrated approaches are more likely to produce greater results than traditional sector-based ones.

Since this approach is new to the USAID context, this section is dedicated to laying out a range of non-traditional techniques and methods that will be used to support these efforts, and will form the conceptual backdrop for most of the DO- and IR-level indicators described below. Crucially, these techniques will require rigorous testing before they can be expanded – not only in terms of their efficacy on the ground, but also in terms of their goodness of fit within the Mission’s current capabilities, analytical needs, and learning culture. Every approach should therefore first be piloted to a smaller group, area, or set of activities, and be immediately followed by Mission reflection and decision-making on the level of investment required to take each method to scale.

New methods require new ways of thinking about how impacts are achieved. Although experimental design impact evaluations are effective ways to determine if outcomes can be attributed to a project, the realities and challenges of the Ugandan context seldom make these feasible.7 It is oftentimes difficult and unrealistic to determine if a given intervention makes the difference, when in fact it is only possible to know if it contributed to making a difference. It is therefore necessary to also consider more nuanced forms of contribution analysis that recognize that there are many relevant factors which fall outside of an intervention’s control. Contribution analysis identifies, unpacks and, in some cases, quantifies the “contribution claims” an intervention makes on its associated impacts in ways that are oftentimes more appropriate for complexity-aware monitoring than experimental designs.8

This PMP will require a suite of approaches and tools that will be applied across varying levels and to varying sets of questions (around hypothesis testing, performance management, testing our assumptions, etc.), and may be combined in various ways. There is no ‘one size fits all’ tool or approach, but several options are presented below with explanations of when and how they will be used.

7 One of the leading references on this point in the field of monitoring and evaluation is found in the “Stern paper,” which argues that limits to impact evaluations are due, among other things, to the fact that they are conducted after the intervention is completed (and thus offer little opportunity for program learning); to ethical considerations around random assignment to control and treatment groups; to increased complex programming that either defies counter factual designs or is not cost effective (e.g. the sample size needed to adequately “power” the design is too expensive). See Stern et. al.

(2012). Broadening the Range of Designs and Methods for Impact Evaluations, DFID Working Paper 38, London: Department for International Development.

8 Here, contribution analysis is used in reference to a broad range of approaches that capture the elements of social change under conditions of uncertainty, while the latter’s use is more dependent upon firmly-established and tested theories of change. See Mayne, J. (2008) Contribution Analysis: An approach to exploring cause and effect, ILAC methodological brief, available at: http://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf http://www.cgiar-ilac.org/files/ILAC_Brief16_Contribution_Analysis_0.pdf

On the quantitative side, traditional approaches to collecting indicator data and analysis will be enhanced by the following approaches:

• Mobile Digital Solutions: Mobile Digital Solutions have increasingly demonstrated their value in current programs and will be scaled up to match the needs of the CDCS.9 These solutions are relevant to tracking progress on the shared Sub-IR 1.x.3: Access to Tools and Technology that Reduce Risks or Mitigate Shocks; IR 3.2: Citizens Actively Participate in Development; and Sub-IR 3.2.2: Access to and Utilization of Information will be Increased.

• Indexing: Many of the goals of the CDCS cannot be represented by a single indicator, but are multi-faceted, interconnected, and don’t easily aggregate into higher-level composite scores. As a result, this PMP describes various indices that combine multiple indicators representing multiple dimensions. Indices, when rigorously constructed, have the advantage of demonstrating how relationships among contributing factors differ, and thus present opportunities to identify strategic entry points to improve program design and facilitate learning and adaptation.10 Indices will be especially relevant for DO1, but will also be used in other specific circumstances across the other DOs.

• Case management (moving from beneficiaries to partners): We will utilize technology (for direct, consistent beneficiary contact) that tracks how beneficiary lives change as a result of USAID support and that opens a new window for data gathering and analysis.

Direct case management focuses implementing partners to layer interventions and enables measurement of the integrated effect of multiple interventions at the household level.

• Forecasting Models: Forecasting models use a systems’ approach to combine the influence of multiple variables across numerous outcomes simultaneously to make projections about the impact of interventions over time. These models are especially useful for understanding the longer terms impacts of activities whose benefits may not be immediately apparent. For this CDCS, forecasting models will be used to estimate how the combined effects of various projects and activities will lead to greater gains in the future.11 These types of models are useful for tracking how the various IRs under DO2 lead to longer term demographic shifts.

9 Recent analysis of mobile phone meta data in particular has established the link between food security and food behavior, human mobility and calling patterns to track the spread of shocks created by infectious disease, and concentrations of traffic to understand if improvements to infrastructure have been successful. See, for example, Decuyper et al. (2015) “Estimating Food Consumption and Poverty indices with Mobile Phone Data.”

10 USAID currently does not provide guidance on how to construct an acceptable index. Until an approach is provided, the OECD’s “Handbook on Constructing Composite Indicators,” is strongly recommended. It can be found at:

https://www.oecd.org/std/42495745.pdf 11 For an example, see Moyer, Jonathan D., Alex Porter, Shelby Johnson, Jason R. Moyer, Dave K. Bohl. 2015. "Advancing development in Uganda: evaluating policy choices for 2016-21 and selected impacts to 2040." Invited Research Paper for USAID, Uganda Mission. Pardee Center for International Futures, University of Denver, Denver, CO

It must be noted that although these approaches are primarily quantitative in nature, they also perform valuable qualitative functions. For example, mobile solutions can also provide geo-located pictures. Indexes can be helpful in categorizing disparate qualitative concepts, and forecasting can use mixed methods. In other words, these methods hold the potential to work in dialogue with qualitative and other methods to advance a more integrated, mixed method approach.

Not all questions about meaningful change can be answered solely through quantitative analysis. On the qualitative side, the following complexity-aware approaches will provide additional insight to progress on the CDCS:

• Outcome Harvesting: Outcome Harvesting (O/H) identifies “defining moments” in the chain of causal events that explain the underlying processes of social change. The relationships between these elements are mapped out into a plausible contribution narrative of the key changes that led to the outcome in question, and are verified with stakeholders on the ground through collaborative discussions. In Uganda, O/H was applied to the recent final evaluation of the District Operation Plan (in which more than 200 qualitative outcomes were harvested),12 and will be relevant for many sub-IRs, such as 3.1.1 and 3.4.2.

• Process Tracing and Bayesian Updating: Process Tracing is an intensive qualitative method ideally suited for questions around the contribution of an activity to policy change and shifts to the enabling environment.13 Process tracing relies upon a rigorous logical testing schema based upon four logical tests of the contribution.14 Once these tests are completed, the team may also use a technique called “Bayesian updating” to assess the quality of evidence to understand if it should be used to support decision-making. Process tracing will be relevant to Sub-IR 3.4.2: Number of USAID-Supported Policy Improvements Strengthened, as well as other IRs in which policy changes are implicated.

• Configuration Analysis: Configuration analysis manages the complexity of multiple factors contributing to an outcome by identifying which combinations of activities and external factors can be logically associated with a given outcome. Oftentimes, configuration analysis involves the application of logical rules of comparison to reduce the complexity of these combinations into more manageable blocks, as well as to identify in which contexts each block is found. Configuration analysis is relevant to address the larger integration question of how the cross-cutting nature of the CDCS contributes to greater results.

• Bayesian Uncertainty Modeling: Uncertainty modeling is a graphical approach to modeling project theories of change in which the complete understanding of how the desired change will occur is still incomplete. Like traditional approaches, these models

12 See DOP Final Evaluation at http://pdf.usaid.gov/pdf_docs/PBAAE808.pdf 13 For an example of Process Tracing, see Befani, B. et. al (2016), “Clearing the fog: new tools for improving the credibility of impact claims,” IIED Briefing Paper. Find at http://pubs.iied.org/pdfs/17359IIED.pdf 14 The four logical tests are: Straw-in-the Wind, Hoop, Smoking-Gun, and Doubly Decisive.

delineate results chains and identify key assumptions, but go further by assigning probabilistic ratings to their presence/absence in the chain. The results of these models yield more realistic predictions of how the CDCS DOs will be achieved and also alternative approaches to learn from bottlenecks and barriers. Uncertainty modeling will be used to understand if all key elements of the system have been identified and understood.15 This type of modeling will be used to gauge progress on understanding how USAID/Uganda’s efforts under DO3 lead to better outcomes.

There are also important mixed-method approaches that will be used to capture a variety of quantitative and qualitative information:

• Network Analyses: As noted by the CDCS, Uganda is characterized by dense webs of social relationships (both formal and informal) that can undermine as well as enable effective service delivery. Network Analyses study the structure of these relationships across individuals, households, communities, organizations, institutions, and systems. Network Analysis has already been used in Uganda to understand the distribution of authority and influence within the environmental management network,16 and can also be used to quantify the degree to which networks are characterized by spheres of influence or corruption (as well as identify key entry points to disrupt these types of networks). Network Analysis will for example be used to measure progress on the joint Sub-IR 1.x.2: Community Social Capital Strengthened, and IR 1.2: Capacity to Manage Risk Increased.

• Truth Tables will also be used to monitor and manage the complexity of the RF.

Configuration analysis tools identify which sets of combinations of activities are relevant to the outcomes produced and which play a less relevant role. This will be done through the use of programmatic “Truth Tables,” tabular representations of all the relevant factors in the model. Truth Tables will be used to present contributions to outcomes, as well as to illustrate where the Mission’s activities and projects are currently operating or have been implemented, and will cut across multiple levels at the household, community, district, etc.17 These factors include, but are not limited to, the presence and implementation of various activities that fall under and cut across the DOs, context variables that are relevant to achieving the outcome, and other relevant factors, such as the presence or absence of other donor and GOU service delivery programs. These factors will be benchmarked against multiple DO outcomes, such as girls’ education outcomes, improved youth productivity, 15 See IOD PARC Concept Note #23 “Use of Bayesian Networks in Evaluation” 26 Feb 2015.

http://www.iodparc.com/get_resource.html?resource_id=23 16 See “An Organizational Network Analysis of Expertise Exchange and Stakeholder Collaboration With USAID/Uganda’s Environmental Management for Oil Sector Activity,” conducted by the QED Group, LLC under the Learning Contract, March 2015.

17 The representation identifies if a factor is present, absent, or some gradation in between (e.g. more present than absent, more absent than present, etc.) through a 0 to 1 coding scheme. The coding scheme is applied through a process of interaction with stakeholders and implementers around what is determined to be “good enough” to be considered successfully implemented/present and what does not make the cut.

http://www.iodparc.com/get_resource.html?resource_id=23 reductions to demographic drivers, and health outcomes. See an example of a Truth Table in Annex F..

The methods listed will be used in addition to measuring progress at activity and project levels.

These methods will help to determine and validate whether targets were met, and whether goals were achieved. It should be noted that there is no “one size fits all” method, and all of the above offer strengths and also limitations. Crucially, none of them establish that program attribution has been made, and the relationships they uncover may in fact be plausible coincidences that can be replaced by stronger alternative explanations. The ongoing goal then is for data users and Mission staff to increasingly and continuously learn about both the positive aspects and the shortfalls of each method.

PERFORMANCE MEASUREMENT AND MANAGEMENT ARCHITECTURE

PROJECT MEL PLANS AND PMP MANAGEMENT GUIDANCE

One of the primary goals of this PMP is to ensure that Project MEL plans demonstrate how their IRs and Sub-IRs contribute to the overarching goals of each DO and to the larger CDCS. Since the CDCS is integrated and works horizontally across projects as well as vertically within projects, the implications are that multiple projects will contribute to the realization of one DO, just as the achievement of a single project purpose will contribute to multiple DOs. As such, it is crucial for projects to clearly articulate their contribution claim to the achievement of the DOs they support, not just the DO where it is housed. For example, while reducing the incidence levels of infectious diseases is a high priority for the Health Team, the PMP requires that projects supporting the cross-cutting DO1 demonstrate how this may contribute to improving community and household resilience (e.g. by measuring the average number of income-earning days lost in a vulnerable household due to malaria infection). This PMP should therefore be understood as a master analytical framework that ensures these varying contributions come together to form stable hypotheses that connect activities and projects at the DO level.

CDCS 2016-2021 will be implemented through five projects:

• Resilience Project

• Demographic Drivers Project

• Health Systems Strengthening Project

• Democracy, Human Rights, and Governance Project

• Market Systems Project

DATA PROTOCOLS

The PMP results framework listed above demonstrates how the interconnections running across DOs and IRs, both horizontally and vertically, are assumed to yield greater development outcomes.

Moreover, the CDCS also assumes that these outcomes will intensify when USAID’s activities are strategically concentrated in the same areas. To sufficiently monitor, test and refine these propositions, USAID requires PMP data that considers dimensions of Geography, Time, Scale, and Quality (GTSQ).

Geography: Uganda differs from region to region depending on socio-cultural factors, conflict legacies, politics, and climate. What works in one region might not work in another area depending on the root causes. USAID will therefore consider regional differences when measuring progress, performance, and outcomes, and manage programs to take advantage of co-location and integration.

This requires that activities and implementing partners remain diligent in geo-coding performance and outcome data, and identify where their activities operate as precisely as possible.

Time: Changes to demographics and systems require a long-term view to affect outcomes which may take decades to appear. However, each of the long-term processes can be divided into “time blocks” with indicators that measure shorter-term progress leading to change. IPs and M&E staff should therefore identify timebound processes that indicate progress towards long-term development objectives. Indicators and other relevant data should include information around their expected duration over time. For example, the benefits from a specific training activity cannot be assumed to occur indefinitely, but lose potency over time. Snapshot context indicators should also be considered to provide up to the minute information of the situation on the ground. The minimum time required to achieve meaningful quantifiable results will also be built into project design.

Scale and Tipping Points: USAID programming must consider scale and investment size to assess progress. In doing so, activities should consider “tipping points,” or thresholds, at which their efforts are expected to produce a substantive change. For example, simply increasing the number of people trained within a given institution does not provide enough information to test that activity’s development hypothesis. Rather, activities must identify how many people need to be trained before that institutions capacity is expected to substantively improve. In many interventions, there is a minimum threshold at which point the investment starts generating results that can cause change to the target population, thus leading to sustainable outcomes.

Quality: USAID will consider the quality of interventions when assessing results and examining the development context for impact. The quality of intervention design and execution will directly impact how, why, and to what extent results are achieved. Although both relevant for determining impact, quality of intervention and quality of data used to monitor those interventions are separate constructs.

As per ADS 201, periodic data quality assessments (DQAs) must be conducted so that USAID is aware of the strengths and weaknesses of the data collected. DQAs are used to help inform program and M&E staff on the extent to which the data integrity can be trusted to influence their management decisions. Over time, M&E staff will develop a data quality ranking system that attempts to quantify varying levels of confidence in the data reported. Activities should refrain from using lower quality data for reporting purposes, but should strive to use low quality data ratings as an opportunity to improve their systems.

The multiple dimensions of GTSQ require a multilayered monitoring system. This monitoring system aims at providing information and analysis that will allow the Mission to assess if each of these elements is appropriate and contributing to the larger objective, and what combinations of these elements are creating the desired results. By analyzing these aspects, the Mission can adapt programming to be more effective and efficient.

The unit of measurement will vary since the Implementing Partners are operating at, and context data is collected at, varying levels. To make the analysis useful for learning and adaptation and to connect the performance of USAID investments with Ugandan development, the Mission will use both performance data and higher level context data. Disaggregation of data and coding of geolocation will be crucial to understand what works and what doesn’t. A review of all activity indicators will be undertaken and new templates for collecting data with disaggregation instructions will be developed.

REAL-TIME MONITORING AND SHORTER FEEDBACK LOOPS

Real-time monitoring allows both implementers and managers to view implementation data on interventions through any device with a screen (computer or smartphone). Implementers and activity managers will use this real-time monitoring to shorten feedback loops for learning and adaptation through USAID activities, even prior to receiving formal (traditional) reports or accountability reports. USAID technical offices will use dashboards based on systems like the national district level health reporting system (DHIS2), PEPFAR reporting system (DATIM) and the national orphan and vulnerable children reporting system (OVC-MIS) to provide real-time monitoring and shorter feedback loops that will feed into design, programming, and implementation.

This form of monitoring will also create learning opportunities without having to wait for formal and structured learning moments. For this the Mission will also require easy access to a large range of data, analytics, and visualization of multiple results.

DATA MANAGEMENT

Adhering to the GTSQ standards above will require a more integrated data management system.

The indicator data needed for this PMP is currently stored in USAID databases such as the Performance Reporting System (PRS), Feed the Future MIS (FTFMIS), PEPFAR (DATIM), possible Pardee Center subnational data, and in Uganda-owned reports and databases such as the Education Management Information System (EMIS), the Health Management Information System (HMIS2), Population and Agriculture census summaries, various sector league tables, Uganda National Household Survey (UNHS), Uganda Demographic and Health Survey (UDHS), Ministry of Finance and Ministry of Local Government data, and other national sources of information. These sources need to be made accessible through a common interface so that their data can be analyzed and visualized through common tools and techniques without negatively impacting the integrity of any of those data sources. The Learning Contract is currently exploring how to access, extract, link, and/or integrate relevant data from these databases to be made available in the new PMP architecture, and how GTSQ protocols can be integrated into this interface through more dynamic meta-tagging. This interface may also guide the construction of a more integrated database management system that combines performance and context data to allow analysis that goes beyond simply meeting targets and identifies how the various pathways to results are achieved.

ROLES AND RESPONSIBILITIES

Project management teams will lead their own learning and evaluation initiatives, supported by the Program Office. Project-based learning and analysis should inform the DO level analysis throughout the Learning Cycle and specifically during DO level portfolio reviews, led by DO leads. Project management teams will guide the overall management and adaptation needed to implement their projects and contribute to Mission-wide DOs. The Mission Leadership Council and the Front Office will provide direction on Mission priority adaptations which will significantly affect the achievement of DOs and the larger CDCS goal.

The Program Office will provide tools, activities, and fora for applied learning and adaptation. The CLA Plan outlines the various key moments of reflection and evidence based decision making which will inform adaptive management. The annual Performance Plan and Report (PPR) preparation will provide data that is analyzed in activity learning reviews, used for project level course correction decision-making, and in testing the CDCS theories of change to allow for adaptive management on all levels across the Mission.

The project management teams will be accountable to results in the IRs that have been identified to contribute to each project purpose across the RF. In strategy level theory of change, it is assumed that several projects contribute to each IR. For example, all projects will contribute to IR 3.1:

Leadership Development Supported which also means that all projects are accountable to IR 3.1.

The DO lead is responsible for ensuring that learning is generated when aggregating the projects and that course correction decisions are in line with the intentions of the CDCS.

The Project MEL Plans will inevitably be similar to the PMP and indicators will often be the same.

However, the PMP takes into consideration the combined outcomes of several projects and will generate higher level analysis that will inform Mission-wide adaptations.

Responsibilities for Monitoring and Analysis

• The Program Office, with support from technical sector M&E specialists, will ensure the application of Geography, Time, Scale, and Quality information to monitoring, evaluation, and implementation decisions throughout the program cycle for improved analysis. The Program Office will coordinate the Mission tracking of the Guiding Principles and the implementation of its CLA Plan. The Program Office will collaborate closely with DO leads and the Front Office to ensure overall PMP implementation.

• AORs, CORs, and activity managers will ensure timely, quality secondary performance data collection from implementers, primarily contributing to the monitoring of Sub-IRs, IRs, and some indicators at the DO level. They are responsible for analyzing results and engaging stakeholders (external and internal) to ensure course correction measures are taken.

• Project management teams, project managers, the Program Office, and some technical…