J.2_SOCY_May_2017_Baseline_Report.pdf

May 7, 2017

This publication was produced for review by the United States Agency for International Development. It was submitted by the USAID/Uganda Monitoring, Evaluation and Learning Program, AID-617-C-13-00007 It was prepared by SoCha,llc under subcontract to QEDGroup,llc.

SUSTAINABLE OUTCOMES FOR

CHILDREN AND YOUTH BASELINE

REPORT

SUSTAINABLE OUTCOMES FOR CHILDREN

AND YOUTH BASELINE REPORT

Submitted to:

USAID/Uganda

Prepared by:

SoCha,llc, under subcontract to QED Group,llc

Disclaimer:

The authors’ views expressed in this document do not necessarily reflect the views of the United

States Agency for International Development or the United States Government.

i

Acknowledgements We would like to thank USAID/Uganda, QED Group,LLC, CRS Uganda and MEEPP for the generous time and support they offered during the preparation for the baseline survey. First and foremost, we are grateful to Katharina Erxleben, Christine Muwanga, Phil Greene, and Douglas

Blanton for endorsing the use of QCA as a rigorous impact assessment tool and inviting SoCha to participate. We also are grateful to the QED Group team – particularly Anant Singh, Johan

Kleinhans, Gorretti Kiiza Mbabazi, Augustine Wandera and Ruth Ankunda for their support throughout Phase 1 of our visit. Staff at CRS and MEEPP – especially Helena Dalton, Richard

Ekodeu, Sasha Angelevski, Fred Werikhe, Olive D’Mello and Barbara Amuron – generously offered their time straight away, and have since been crucial to the development of our understanding of Sustainable Outcomes. We’d also like to recognize the insightful ethics training offered by Dr. Mastula Nanfuka-TASO IRB. Finally, we’d also like to thank Dr. Leyla Ismayilova and her team, Nathan Roter, Lu Yang, and Young Han, from the University of Chicago, School of

Social Service Administration and Dr. Elias Dinas from Brasenose College, Oxford for their technical inputs and guidance on experimental design.

SoCha,LLC ii

Acronyms

ACODEV Action for Community Development

AMELP Activity Monitoring, Evaluation and Learning Plan

ANPPCAN African Network for Prevention & Protection Against Child Abuse & Neglect

ART Antiretroviral therapy

CAO Chief Administration Officers

CDO Community Development Officers

CRS Catholic Relief Services

CSO Civil Society Organizations

DCDO District Community Development Officers

DID Difference in Difference Design

DHMIS2 District Health Information System 2

DOVVC District OVC Coordination Committees

ECD Early Childhood Development

HIV Human Immunodeficiency Virus

HVAT Household Vulnerability Assessment Tool

HVPT Household Orphans and Vulnerable Children Vulnerability Prioritization Tool

IPs Regional Implementing Partners

MEEPP Monitoring and Evaluation of Emergency Plan Progress

MER Monitoring, Evaluation and Reporting

MGLSD Ministry of Gender Labor and Social Development

ODK Open Data Kit

OLS Ordinary Least Squares

OVC Orphans and Vulnerable Children

OVCMIS OVC Management Information Science

PEPFAR President's Emergency Plan for AIDS Relief

PSWs Para Social Workers

QCA Qualitative Comparative Analysis

SAGE Social Assistance Grants for Empowerment

SCORE Sustainable Comprehensive Responses for OVC and their Families

SDS Strengthening Decentralized Systems

SILCs Savings and Internal Lending Communities

SOCY Sustainable Outcomes for Children and Youth (aka Sustainable Outcomes)

SOVCCs Subcounty OVC Coordination Teams

SRMP Sub Recipient Management Policy

TPO Transcultural Psychosocial Organization

UNICEF The United Nations Children's Fund iii

UPDF Uganda People’s Defense Force

USAID United States Agency for International Development

VHTs Village Health Teams

VI Vulnerability Index

WASH Water, Sanitation and Hygiene iv

Table of Contents

Acknowledgements ............................................................................................................................................................ i

Acronyms............................................................................................................................................................................. ii

Executive Summary ........................................................................................................................................................... v

Background ..................................................................................................................................................................... v

Baseline Goals and Design .......................................................................................................................................... v

Baseline Implementation ............................................................................................................................................ vii

Findings ......................................................................................................................................................................... viii

Conclusions and Recommendations ........................................................................................................................ xi

Introduction

Evaluation Purpose, Aims and Objectives

Research Strategy

Research Design: Impact Evaluation

Research Design: QCA

Outcome: Defining Vulnerability

Power Analysis and Sample Size Calculations

Sampling Strategy

Survey Implementation

Baseline Survey Results

Survey Data

Group Comparisons of Household Socio-Demographic Characteristics

Group Comparisons of Secondary Outcomes

Vulnerability Assessment and HVAT Scoring

HVAT Group Comparisons

HVAT Distribution

Relative Biases

Mapping the QCA System

Sustainable Outcomes MIS

Suggested Improvements to the System

Conclusions and Recommendations

Annex 1: Scope of Work

Annex 2: Survey Instrument

Annex 3: HVAT Scoring Method

Annex 4: List of Sampled Parishes v

Executive Summary

Background USAID/Uganda’s Orphans and Vulnerable Children (OVC) programming employs an integrated approach to reducing OVC and Youth vulnerability in Western, Southwestern and Central Uganda. Specifically, Sustainable Outcomes for Children and Youth aims to achieve the following three Results:

Result 1: Orphaned and vulnerable children, youth and their caregivers are better economically empowered to access core services;

Result 2: Local government and CSOs and informal community structures increase and improve core services for orphaned and vulnerable children, youth and their caregivers;

Result 3: Improved coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care.

These three results are designed to be mutually supporting, in which SILC households not only benefit from the expected increase in knowledge, access, and support, but also experience improved access to quality OVC and services across an improved body of coordinated service providers operating at local

(CSO and informal), social service worker, local government and other service provider (other programs and NGOs) levels.

Sustainable Outcomes for Children and Youth has already entered its second year of implementation and have reached a significant number of households. System strengthening interventions under Result 2 have been rolled out in all target communities since October 2015. Under Results 1 and 3, households will be reached across three Cohorts:

Cohort 1 (since May 2016): 85% of communities

Cohort 2 (planned for January 2017): 5% of communities

Cohort 3 (planned for January 2018): 10% of communities (tbd. after review)

This staggered implementation approach presents the opportunity to conduct an impact evaluation of how effective Sustainable Outcome’s is at reducing household, orphan and youth vulnerability. Specifically, the impact evaluation will systematically compare the difference in outcomes between Cohort 2 and Cohort

3.

Baseline Goals and Design The purpose of this baseline is to help USAID/Uganda answer two overarching questions regarding the effectiveness of Orphans and Vulnerable Children (OVC) programs; understand how different interventions or combinations of interventions of the program contribute to program outcomes; and generate evidence to inform decisions about future OVC programs. The aim is to assess and understand the impact of participating in programs that seek to improve household socioeconomic security through savings and strengthen institutional childcare service delivery on OVC caregivers, households and children over time.

The primary objectives of this baseline are to lay the foundation to answer two complementary and inter-related questions:

1. Has caregiver participation in Sustainable Outcomes reduced household vulnerability and improved OVC well-being?

2. What combinations of factors (Including those within and outside of Sustainable Outcomes’ control) best identify how this may or may not have happened?

vi

Design The first question will be answered use a quasi-experimental, difference in difference, impact evaluation design in which the outcomes of Cohort 2 will be compared with a control group that will later be eligible to participate in Cohort 3. The second question will be answered using Qualitative Comparative Analysis

(QCA). QCA is a comparative research approach that combines the diversity of implementation with data on relevant external factors into an analytical model that identifies how various pathways to the outcome can be achieved. This combined QCA/quasi-experimental design approach presents a unique opportunity to test Sustainable Outcomes’ underlying assumption that caregiver participation in a SILC group, combined with additional support services and an improved referral system, improves household economic status which in turn leads to improvements in child well-being.

This baseline report is designed to lay the groundwork to address three different impact evaluation questions:

Question 1: Does caregiver participation in activities under Results 1 and 3 reduce household vulnerability and improve OVC well-being?

Question 2: Does caregiver participation in activities under Results 1 reduce household vulnerability and improve OVC well-being?

Question 3: Do the combined activities under Results 1, 2 and 3 reduce rates of HIV prevalence and incidence in participating subcounties?

Moreover, it is also designed to address the following sets of evaluative questions using QCA:

1. What combinations of factors best identify how Sustainable Outcomes reduced household vulnerability and improved OVC/Youth outcomes?

2. What combinations of factors explain where Sustainable Outcomes did NOT reduce household vulnerability and improved OVC/Youth outcomes?

3. What external factors, especially additional services provided as the result of Sustainable

Outcomes’ efforts to improve the referral system, are relevant for reducing household vulnerability and improving OVC/Youth outcomes?

4. Which activities implemented under Sustainable Outcomes may not be relevant for reducing household vulnerability and improving OVC/Youth outcomes?

5. How did households and OVC/Youth improve in subcounties where Sustainable Outcomes did not (yet) implement R1 and R3 activities?

6. What are the key factors of R2 that explain how HIV prevalence and incidence rates may have declined in Sustainable Outcomes subcounties relative to subcounties outside of the program?

The primary impact indicator used for this evaluation is the vulnerability score assessed with the

Household Vulnerability Assessment Tool (HVAT). Twenty-four secondary outcome measures were captured at the OVC and Youth wellbeing level for children aged 0-9 years (with questions directed to the primary caregiver) and youth aged 10-17 years. These include:

Percent of children whose primary caregiver knows the child’s HIV status

Percent of children < 5 years of age who are undernourished

Percent of youth too sick to participate in daily activities

Percent of children too sick to participate in daily activities

Percent of youth who have a Birth Certificate

Percent of children who have a Birth Certificate

Percent of youth enrolled in school

Percent of children enrolled in school vii

Percent of youth regularly attending school

Percent of children regularly attending school

Percent of children <5 years with recent diarrhea

Percent of children <5 years with recent fever

Percent of youth >10 to 17 years reporting irregular food intake

Percent of children >2 to 9 years reporting irregular food intake

Percent of youth who progressed in school during the last year

Percent of children who progressed in school during the last year

Percent of children <5 years of age who recently engaged in stimulating activities with any household member over 15 years of age

Percent of caregivers who feel harsh physical punishment is appropriate means of discipline in the home or school

Percent of households able to access money to pay for unexpected expenses

Percent of households able to access money to pay for health expenses in past 3 months

Percent of households able to access money to pay for school expenses in past 3 months

Percent of households able to access money to pay for food expenses in past 3 months

Percent of children 1-5 years fully immunized

Percent of youth aged 10-17 years reporting basic support

Two HIV-specific measures HIV prevalence and HIV incidence rates, will be captured at the sub county level over time. These can be broken down into specific subgroups, such as girls ages 10-17. We will use data from the District Health Information System 2 (DHMIS2) database.

Baseline Implementation The survey team conducted household survey for 2,629 households across 82 parishes in 15 districts across three regions. The survey was adequately powered (using the standard power of 80% at a significance level of 5%) to detect changes in each impact indicator that fall between 5%-10%. The control group included 834 households, Treatment group 1 (R3 group) assigned to receive the village Para-social work training included 863 households and the Treatment group 2 (R1R3) that will receive the combined

SILC intervention included 932 households. Households are distributed across the three implementation regions as follows:

HH Group Distribution by Region

Region Control Group R3 group

(Treatment 1)

R1R3 group

(Treatment 2) Total

Central 834 460 0 1294

Western 0 317 0 317

Southwest 0 86 932 1018

Total 834 863 932 2629

Control and treatment groups were selected according to the Sustainable Outcomes implementation cycle. The R3 treatment group runs across all three regions, with a higher concentration in Central and the fewest in the Southwest. None of these households have received any R1 services or are slated to receive any services under Cohort 2. All households for the R1R3 group are from the Southwest Region, as this is the main region in which Sustainable Outcomes program staff will be providing R1 services (as well as receive R3 services). Households selected under the Control group are those districts which have not received any R3 or R1 services to date, but are eligible for participation in the program under the viii upcoming Cohort 3. These households therefore allow us to avoid any ethical issues associated with observing households in need but denying them any subsequent assistance.

Household sampling selection was aligned to Sustainable Outcomes’ enrollment target areas across

Cohorts 2 and 3, and used a two-stage cluster sampling approach to select households. In the first stage, 31 parishes were selected from each treatment and control group’s sampling frame. In the second stage, 27 to 30 households in the control group and R3 treatment were selected in each Parish using the “random route” technique. In the R1R3 treatment group, the survey team received lists of potential vulnerable households from Sustainable Outcomes local representatives and randomly selected a household from this list as the enumeration starting point. Households were first screen for consent and eligibility using the HVAT screening tool.

The baseline survey was designed to collect sampling information at three levels of analysis: Household, OVC (0-9), and Youth (10-17). Information was collected across three dimensions: Vulnerability, OVC/Youth Status and Situational Analysis. Each dimension uses a questionnaire module that was already tested and IRB approved in Uganda. Data was collected using handheld tablets (with power banks) using

Survey CTO. Survey instruments and consent forms were translated into local languages Luganda, Runyankole, and Runyoro/Rutoro. Consent forms were developed in the respective local languages for primary care givers, emancipated youth respondents (age 10-17), and assent forms for children below 12 years (signed by the primary care giver).

During the baseline survey period, the study team also met with Sustainable Outcomes staff to work to establish a QCA-system within the program’s M&E system. This entailed the following steps:

1. Socialized the Sustainable Outcomes teams on QCA: The study team presented to

Sustainable Outcome staff on how QCA functioned, it’s applicability to the program and next steps;

2. Confirmed the relevance of the QCA questions to be answered: The study also presented the QCA questions to the implementation team to discuss and confirm their relevance to management decision making.

3. Identified data inputs and modify, if needed and possible, the current MIS system: The study team held several meetings with the M&E team on the design and implementation of the program’s MIS. Several changes were suggested (see below);

4. Understand the “human systems” behind the data: The team also met individually with program staff to identify data flows, reporting and roles/responsibilities associated with the collection of implementation data.

Findings For the QCA design, the baseline survey team worked with Sustainable Outcomes staff to identify which sets of relevant implementation data will be used to answer the “how” questions of program effectiveness.

Overall, Sustainable Outcomes has put into place a new system that sufficiently captures a significant amount of data to answer a wide array of implementation questions. However, the following limitations were observed:

There are no fields to capture GPS coordinates.

The current Household ID system holds the potential to generate duplicate numbers. Moreover, villages and Parishes that have the same name across the 16 districts cannot be distinguished from each other in the Household ID system. The implications are that users may incorrectly assign households to other districts.

The current system cannot capture multiple referrals.

ix

These concerns aside, the new system is comprehensive, user friendly and well designed. The only other major potential challenge involves capacity, as Sustainable Outcomes currently employs only a small number of M&E experts, none of which have database management expertise.

The QCA system has strong potential to enable Sustainable Outcome implementer to use the results to provide ongoing program feedback, learning loops, mini-cost effectiveness analyses and midcourse program adaptations. These steps are not yet formally adopted in the program’s workplan.

For the 24 secondary outcomes, five had no significant differences across the three groups. These all revolved around children under five on such matters as involvement in stimulating activities, irregular food intake, school attendance and progression, health and caregiver knowledge of HIV status. All remaining secondary indicators had significant differences.

For the primary outcome indicator, the baseline survey found that household vulnerability using the HVAT tool was distributed according to the following:

Distribution of Vulnerability across Control and Treatment Groups

HVAT Category Control R1 R1R3 Total

Not Vulnerable 64 (8%) 60 (7%) 23 (2%) 147 (6%)

Slightly Vulnerable 628 (75%) 667 (77%) 630 (68%) 1925 (73%)

Moderately Vulnerable 142 (17%) 136 (16%) 279 (30%) 557 (21%)

Critically Vulnerable 0 0 0 0

The differences between groups are significant (*** p<0.001) and can most likely be attributed to geography.

Moreover, the HVAT’s six constituent “Core Program Areas” (CPAs, i.e. economic strengthening, food and nutrition, health and water, sanitation and hygiene (WASH), education, child protection and psycho-social) also held significant differences. Using data from a previous “Vulnerability Index,” analysis revealed that the Central region (which contains the control group) tends to have a lower rate of vulnerability compared to the Southwest and Western region (which mostly hosts R1 and R1R3 treatment groups).

These differences are not an issue for the difference-in-difference design.

The distribution of household vulnerability scores does not pose a threat to the impact evaluation design, and also improves the explanatory power of the QCA analysis by providing more evidence of the counter-factual within each treatment group. However, from a programmatic view, the seemingly low number of more vulnerable households as well as the percentage of “not vulnerable households” may be a cause for concern. This raised questions about the household selection approach and about the design of the HVAT tool; both of which required further investigation. In regard to household selection, households that were selected through pure random selection without the assistance of Sustainable Outcome stakeholders had no significant differences from those who were selected from lists generated by the Sustainable Outcomes team.

However, two of the HVAT’s CPAs – psycho/social and child protection - most likely suffers from a positive social response bias (especially in regard to self-reporting of abuse and mental well-being), which may drive vulnerability scores downward (i.e. households appear less vulnerable than they actually are).

Dropping these two CPAs pushes overall vulnerability upward (households are scored as more vulnerable).

x

During the implementation, the survey team also discovered that Sustainable Outcomes program staff had also dropped these two CPAs, and were working from a leaner version known as the Household

Assessment Tool (HAT) for Cohort 1. The survey team found that Cohort 1 has a higher percentage of

“Not Vulnerable” and a slightly higher “Critically Vulnerable” representation, while Cohort 2 has a much higher percentage of “Moderately Vulnerable” and very similar “Slightly Vulnerable” distribution. As such, there do appear to be systematic differences between the two Cohorts in which Cohort 1 is thicker on both ends of the vulnerability spectrum while Cohort 2 has a higher concentration of mid-level vulnerability. Of particular concern though, is the high number of “not vulnerable” households currently receiving support under Cohort 1 (more than 14,000).

In regard to the design of the HVAT tool, it was found that the way the tool is scaled has an impact on the resulting vulnerability scores and status. More precisely, a review of the HVAT tool scaling structure reveals the following:

Contradictions in scoring, due in part to inconsistent units of measurement;

Arbitrary spaces in the scoring of each item can bias household vulnerability scores downward

(i.e. they are less vulnerable);

Many of the choices found under each question/item do not fall on the same spectrum (i.e. some represent different concepts than the one represented by the question);

The items on the HVAT are not equal in relevance and therefore should not be scored equally;

The items fall across different underlying dimensions and therefore are not easily aggregated using the current scoring method.

Principle Component Analysis (PCA)1 was used to test the internal consistency of each item across the

HVAT index and its six constituents “Core Program Areas” (CPAs, i.e. economic strengthening, food and nutrition, health and WASH, education, child protection and psycho-social). The results suggest a very poor fit in terms of how well all HVAT items combine to form a uni-dimensional scale. Less than 16% of all variance could be explained on a combined HVAT component, and 9 of the 17 items drop out with scores below a minimum .35 threshold. These results suggest that the HVAT cannot be easily reduced to a single dimension score. The implications are that HVAT scores can be a misleading proxy for comparing underlying vulnerability across households (for example, two households that hold the same overall vulnerability score but differ in their CPA distribution in fact may differ significantly).

To improve the scale, items and options on the scale were separated and tested separately to generate total CPA scores instead of individual items. The results lead to the creation of two underlying dimensions associated with household vulnerability: “material vulnerability,” i.e. economic strengthening, food and nutrition, health and WASH and education; and “functional vulnerability,” i.e. child protection and psycho-social support. Using this approach to assign weighted factor scores, new household vulnerability scores were generated and compared with the original raw scores. However, while these more accurate results revealed that the relative weights of each CPA shifted and were more accurately represented than using the original raw scoring method, no substantial differences in the distribution of household vulnerability between the two methods could be found.

The survey team also explored how geography may help explain the seemingly lower level of household vulnerability across both Cohort 1 and the baseline. Using data from a previous “Vulnerability Index” that was reported in the Ministry of Gender Labor and Social Development (MGLSD)’s 2010 OVC Situational

Awareness Report (p. 4), a quick comparison reveals that the Central region (which contains the control

1 PCA is a statistical technique used to identify a smaller number of uncorrelated variables, i.e. components, from a large set of variables. The goal is to explain the maximum amount of variance with the smallest number of components.

xi group) tends to have a lower rate of vulnerability compared to the Southwest and Western region (which mostly hosts R1 and R1R3 treatment groups):

Critically

Vulnerable

Moderately

Vulnerable

Generally

Vulnerable

Total

Vulnerability

Central 7.8 33.6 52.7 94.1

Eastern 7.5 45.5 43.8 96.8

North 9.3 53.6 35.9 98.8

Western 8.1 41.1 45.9 96.1

Average 8.1 42.9 45.1 96.1

However, the survey team was unable to obtain the primary data behind these numbers, as well as the scaling methods used to derive them. As such, one should be wary of making direct comparisons with this data and further research is most likely required.

Conclusions and Recommendations These results lead to the following high-level conclusions and recommendations:

The HVAT is an adequate tool to provide an overall snapshot of various dimensions of household vulnerability and is therefore useful for program decision making and addressing higher level evaluative questions of effectiveness. However, potential positive social response bias on the psycho-social and child protection CPAs suggest that results from these CPAs should be met with caution. Moreover, when comparing changes in vulnerability status using rigorous statistical approaches, the impact evaluation should break down the HVAT scores by CPA and item scores and search for statistical differences of each. Doing so will provide a more nuanced picture of what impacts have been achieved.

While the distribution of household vulnerability poses no threat to the design of the impact evaluation (and improves explanatory power of the QCA component), the lower seemingly low number of more vulnerable households as well as the percentage of “not vulnerable households” may be a cause for concern. While there are some scaling biases built into the way vulnerability is scored, once accounted for they do not reveal any substantial differences in the distribution of vulnerability across the sample. Rather, the differences may be better attributed to Ugandan economic geography, but this conclusion requires further investigation. Regardless, the high number of “not vulnerable” households enrolled in Cohort 1 does suggest that Sustainable

Outcome staff review their selection procedures.

The embedded QCA system is comprehensive, user friendly and well designed. Sustainable

Outcomes staff should consider improving the system to capture GPS, avoid duplicates and capture ongoing referrals by implementing partners. Moreover, the program may also wish to consider employing additional M&E staff to manage the wealth and quality of data that is being captured. The QCA system has strong potential to enable Sustainable Outcome implementer to use the results to provide ongoing program feedback, learning loops, mini-cost effectiveness analyses and midcourse program adaptations. The program should consider formally including these steps in its current workplan.

Introduction

Uganda is currently the second youngest population in the world and the third fastest growing nation in

Africa. However, deep and extensive vulnerabilities exist that especially affect children: 96% of children are considered vulnerable, and 62% of those living in poverty are children. Vulnerabilities take different forms, including poverty, malnutrition, and exposure to violence. Low levels of education and high prevalence of HIV/Aids among children themselves as well as within their families exacerbate these vulnerabilities.

USAID/Uganda’s Orphans and Vulnerable Children (OVC) programming employs an integrated approach to address these factors. The Sustainable Outcomes for Children and Youth (Sustainable Outcomes) activity aims to economically empower children, youth, and their caregivers to access core services, strengthen systems to provide core services, and improve coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care. The activity is a five-year cooperative agreement implemented by a consortium led by Catholic Relief Services (CRS) across

17 districts in Central and Western Uganda. These districts were identified by the United States Agency for International Development (USAID) due to high prevalence rates of HIV and availability of trained community workers to deliver program services. Sustainable Outcomes’ goal is to help a minimum of

625,000 OVC, youth and 101,500 households to access core services for improving health, nutrition, education, and psychosocial well-being to reducing abuse, exploitation and neglect.

Sustainable Outcomes deploys an integrated approach to reduce OVC and Youth vulnerability that operates across multiple levels. Specifically, Sustainable Outcomes aims to achieve the following three

Results:

Result 1: Orphaned and vulnerable children, youth and their caregivers are better economically empowered to access core services;

Result 2: Local government and CSOs and informal community structures increase and improve core services for orphaned and vulnerable children, youth and their caregivers;

Result 3: Improved coordination of community-based clinical and socio-economic services for efficiency and effectiveness along the continuum of care.

Each Result corresponds to a different level of implementation, beneficiary and support package. For

Result 1, OVCs and their care givers at the household level are the main beneficiaries of support, primarily through the formation of savings and internal lending communities (SILCs).2 SILCs are hypothesized to increase household income and savings, which will enable the household and OVC caregiver to better provide for OVCs as well as better access core OVC services outside of the home.3 Activities include

2 For a description of SILCs, see Vanmeenen G. (2010). Savings and Internal Lending Communities (SILC) Voices from Africa: The benefits of integrating SILC into development programming. Baltimore, MD: Catholic Relief Services. Drawing upon various descriptions in SILC evaluations, here SILCs can be summarized as model developed by CRS for user-owned, self-managed savings and credit groups that offer households a way to protect assets, smooth cash flow, and increase income. Community members are encouraged to self-select into groups of 15-30 people they know and trust. Compared to traditional accumulating savings and credit associations, SILC groups are more accessible, transparent, and flexible. Each group determines its meeting schedule, regular contributions to a loan fund, loan duration, interest, and maximum loan amount. Members can borrow from the fund at this predetermined interest rate and term. Interest and fines allow the common fund to grow. At the end of a predetermined time period, all or part of the common fund returns to the group members based on the total amount saved by each member.

SILC groups are initially supported by project “field agents,” but the goal is institutional and financial independence (note: this was taken from the SILC Impact Evaluation of the STEPS OVC Project in Zambia Final Evaluation Report, found at www.silcevaluation.com).

3 The key assumption in this argument is that if SILCs do in fact increase household income and savings for members who are caregivers of OVCs, some of the benefits will also reach the OVC, e.g. caregivers will use these resources to benefit the child and not use them for something else.

increased access to temporary consumption support (by linking them to external resources such as SAGE, Give Directly, and WFP, as well as to groups within the community); enroll households in SILCs (in which

15-30 households pool savings on a weekly basis to create social funds and access loans; the returns from which are paid out to members after 12 months); train SILC members on a variety of topics, such as financial management, HIV prevention and treatment, skills building, and better parenting/caregiving; link girls to access to better education opportunities; and provide additional technical assistance, market access, and training to farmer-focused SILCs to improve income.

For Result 2, district government and CSOs are the main beneficiaries and here sustainable outcomes work with relevant district staff to create functioning OVC coordination mechanism, allocate resources for OVCs and youth, fill key OVC-support positions, conduct performance appraisals of relevant staff, use the OVCMIS for planning, create alternative care panels and transition OVCs from institutional to family based alternative care. The program will also build the capacity of CSOs to comply with national OVC and youth standards, conduct protection pilots and implement improved OVC service delivery.

Finally, for Result 3, the primary beneficiary is the referral system, in which the Para-social workers (PSWs) and village health teams (VHTs) sit as the main actors who refer youth and OVCs to the appropriate provider in the service network. Here, activities include a mapping of district level services; training PSWs and VHTs on how to make referrals to multiple types of services, track those referrals, and assist vulnerable households with the development of household action plans; and conducting pilot projects on an Early Childhood Development (ECD) tracking component, ECD development spaces at health centers, and creating very young adolescent groups to discuss sexual reproductive health issues.

Figure 1: Sustainable Outcomes Graphic Representation

These three results are designed to be mutually supporting, in which SILC households not only benefit from the expected increase in knowledge, access, and support, but also experience improved access to quality OVC and services across an improved body of coordinated service providers operating at local

(CSO and informal), social service worker, local government and other service provider (other programs and NGOs) levels.

Sustainable Outcomes will be considered successful when 100% of the 101,500 targeted households have left their status as vulnerable behind. System strengthening interventions under Result 2 have been rolled out in all target communities since October 2015. Under Results 1 and 3, households will be reached across three Cohorts:

• Cohort 1 (since May 2016): 85% of communities

• Cohort 2 (planned for January 2017): 5% of communities

• Cohort 3 (planned for January 2018): 10% of communities (tbd. after review)

Evaluation Purpose, Aims and Objectives The purpose of this evaluation is to help USAID/Uganda answer overarching questions regarding the effectiveness of Orphans and Vulnerable Children (OVC) programs; understand how different interventions or combinations of interventions of the program contribute to program outcomes; and generate evidence to inform decisions about future OVC programs.4 The aim of this evaluation is to assess and understand the impact of participating in programs that seek to improve household socioeconomic security through savings and strengthen institutional childcare service delivery on OVC caregivers, households and children over time.

The primary objectives of the evaluation are to answer two complementary and inter-related questions:

1. Has caregiver participation in Sustainable Outcomes reduced household vulnerability and improved OVC well-being?

2. What combinations of factors (including those within and outside of Sustainable Outcomes’ control) best identify how this may or may not have happened?

Answering the first question suggests a counterfactual research design that compares changes in vulnerability status across various control and treatment groups to demonstrate “what would have been the change if households had NOT participated in Sustainable Outcomes.” This is commonly referred to as an impact evaluation. Answering the second question suggests a comparative research design that combines the diversity of implementation with data on relevant external factors into an analytical model that identifies how various pathways to the outcome can be achieved. Qualitative Comparative Analysis is the method commonly associated with this approach. In what follows below, we review the overall design strategy, the outcome(s) the Sustainable Outcomes will be evaluated against, the type of quasi-experimental design that has been used, the approach to QCA, and finish with a report on the baseline results.

Research Strategy Combining a QCA-approach with an impact evaluation presents a unique opportunity to test Sustainable

Outcomes’ underlying assumption that caregiver participation in a SILC group, combined with additional

4 An earlier version of this report was submitted in the form of a concept note. The creation of the note involved a document review of Sustainable Outcomes documents (including the ToR, the proposal, the workplan, the AMELP, regular reports, the baseline survey, and various tools), MEASURE Evaluation OVC tools and evaluations, a baseline study conducted by MEEPP, Government of Uganda OVC strategy, OVCMIS and Vulnerability Index documents, PEPFAR guidance documents, USAID/Uganda portfolio review of the OVC program, and literature on SILCs implemented elsewhere. The study team also spent one week in

Kampala in mid-August visiting MEEPP, USAID/Uganda staff and CRS staff. The team leader was supported by the senior M&E education advisor and the education communications officer at the Learning Contract.

support services and an improved referral system, improves household economic status which in turn leads to improvements in child well-being. The combination of this approach yields a design strategy oftentimes referred to as “lumping and splitting”. For the sake of policy decision making, it is oftentimes necessary to “lump” a wide array of outcomes and pathways into two groups (control and treatment) represented with summary statistics based upon measures of central tendency (e.g. the mean, the regression line, etc.). This allows us to make clear, testable statements around program attribution and causal inference (aka internal validity). While useful, lumping seldom includes information on how the multitude of household pathways converged on these results across contexts, and so we are left with little information on how to reproduce the same results elsewhere. It is therefore necessary to “split” the control and treatment groups up into more refined subgroups to identify how local context matters and how various aspects of a program combine and interact in different ways. This then allows us to better understand how various pathways to the outcome occurred and can occur elsewhere (i.e. external validity). While also useful, splitting does run the risk of identifying pathways that may have been the result of random chance, and therefore our ability to attribute the outcome to these explanations is reduced. As such, combining the strengths of both approaches compensates for their respective weaknesses.

This evaluation will use a longitudinal, quasi-experimental and QCA design using a multi-stage cluster sampling approach in which a representative group of households that participate in Sustainable Outcomes will be compared with households that have not yet participated (but will be eligible in the future). These groups will be selected from Cohorts 2 and 3, respectively.5 The evaluation will also incorporate a wide array of implementation and environmental data. Participants and households in both groups will take part in an annual interviewer-administered survey for three study years. Systematic follow up during and after Sustainable Outcomes implementation will allow for retrospective analysis of dose-response, sub-group analysis, and evaluation of the sustainability of both SILC and outcomes of participation.

Research Design: Impact Evaluation This baseline report and dataset contains outcome and other data collected from two treatment and control groups to estimate the impact of Sustainable Outcomes on vulnerability and childhood outcomes.

The difference between these groups will be the estimates of Sustainable Outcomes’ effects. Additional data collection waves are anticipated in midline (Fall 2017) and endline (Fall 2018).6

This baseline report is designed to lay the groundwork to address three different impact evaluation questions:

Question 1: Does caregiver participation in activities under Results 1 and 3 reduce household vulnerability and improve OVC well-being?

Question 2: Does caregiver participation in activities under Results 1 reduce household vulnerability and improve OVC well-being?

Question 3: Do the combined activities under Results 1, 2 and 3 reduce rates of HIV prevalence and incidence in participating subcounties?

In designing the impact evaluation, the evaluation team considered a variety of experimental and quasi-experimental designs. Many aspects of the Sustainable Outcomes program – such as the voluntary-based

SILC group consisting of both direct and indirect beneficiaries – defy random assignment and thus we were unable to advance a randomized control trial design and instead turned to quasi-experimental

5 In terms of household identification, we will use the same numbering system as CRS and merge our dataset with theirs. We will also use the same confidentiality protocols, with limited rights to the datasets in which household locaters are restricted, and keep this information on a secure server. Any analysis of the data will exclude this information.

6 Note: these times are subject to change according to the implementation cycle.

designs. Originally, we had considered a regression discontinuity design based upon the program’s enrollment criteria, but further exploration revealed that these criteria served more as guidelines than as rigid standards that were enforced consistently across all areas. Finally, we had reviewed the option of using genetic matching techniques across the control and treatment groups, but ruled out this approach as the potential differences in geography separating the groups may undermine the matching assumptions of the model.

This baseline was designed to support a difference-in-difference (DID) impact evaluation design. DID designs compare two types of changes: The changes in outcome of a treatment group (or groups) before and after an intervention has occurred with the changes in outcome of a control group (or groups) over the same period of time. This design puts in place two controls: The difference in the before-and-after outcomes for the group(s) enrolled in the program controls for factors that are constant over time in that group (as it is comparing the same group to itself over time); Then, this difference is compared with the difference in before-and-after outcomes for a group that was not enrolled in the program but was exposed to the same set of environmental conditions over the same time period to control outside time-varying factors. DID thus combines these two counterfactuals (before-and-after comparisons, and comparisons between those who were enrolled and those who were not enrolled) to produce a better estimate of the counterfactual to identify the impact of the program (see Figure 2 below).

Figure 2: Difference in Difference Design

Crucially, the DID design does not require that each group have statistically insignificant outcome differences at baseline, i.e. statistically significant differences across groups do not pose threats to the internal validity of the design (we find several statistically differences across groups, which is discussed below). Rather, the key assumption for DID is that the outcome across treatment and control groups will follow the same trend over time in absence of the treatment.

For Question1, households and OVC/Youth in Cohort 2 subcounties that receive both R1 and R3 treatments (treatment group 1) will be compared to households and OVC/Youth in Cohort 3 subcounties

(the control group) who do not receive any treatment until the end of 2017 and 2018. A simple representation of the measure of impact under the experimental design is the interaction effect of treatment and time, or the double-difference estimate, is shown in Equation (1):

Estimate of impact of R1+R3 on Vulnerability = (YR1+R3,t2 – Y R1+R3,t1) - (YC,t2 – YC,t1) where, Y = outcome of interest; T = treatment group; C = comparison group; t1 = baseline or beginning of study; and t2 = middle or end of study. As discussed above, this question will be addressed using a difference in difference design to compare differences between households enrolled in the treatment group of Cohort 2 and those who reside in the sub-counties associated with the planned implementation of Cohort 3.

For Question 2, the effects of R1 will be estimated by making comparisons between households and

OVC/Youth households that receive both R1 and R3 (treatment group 1) with those that only receive R3

(treatment group 2). The difference between the two groups will be the estimates for the R1 effect, as shown in Equation (2):

Estimate of impact of R1 on Vulnerability = (YR1+R3,t2 – Y R1+R3,t1) - (YR3,t2 – YR3,t1)

For this question, the study also uses a difference in difference design to construct the control and treatment groups. To avoid contamination in the treatment group 2 (only R3), households were selected in Cohort 2 subcounties that did not contain parishes where R1 will be implemented (see below).

For Question 3, the study will estimate the combined effects of R1+R2+R3 by comparing HIV Prevalence and Incidence rates across Sustainable Outcomes subcounties and various high risk groups, such as girls age 10-24, within those subcounties with rates in non-Sustainable Outcomes subcounties across the 17 districts, as shown in Equation (3). This data will be collected from secondary sources; namely, the

Ministry of Health DHMIS2 database.

Estimate of impact of R1+R2+R3 on HIV Prevalence and Incidence = (YR1+R2+r3,t2 – Y R1+R2+R3,t1) - (YC,t2 – YC,t1)

For this design, the study also relies upon a difference in difference approach to construct the control and treatment group.7

Figure 3: Impact Evaluation Design

All three equations will be estimated using a simple regression formula presented below:

𝑌𝑡2 − 𝑌𝑡1 = 𝛼 + 𝑇 𝛽 + 𝑋 𝛿 + 𝜀

7 The difference in difference designs are very straightforward comparisons of changes in treatment group with changes in a comparison group, but lacks random assignment. Instead, equivalence between the two groups will be found by adjusting for previous rates of change in HIV prevalence and incidence rates for control and treatment groups.

where T is a dummy for assignment to treatment group (e.g. R1+R3) and X are characteristics of the unit of analysis (households, OVCs/Youth and HIV rates). This regression can be estimated using ordinary least squares (OLS).

Research Design: QCA Whereas the DID design will confirm whether or not Sustainable Outcomes has worked, this study uses

QCA to identify how it has worked. QCA is a data analysis technique for identifying which combinations of factors best explain how an outcome was (or was not) achieved. Factors here refer to a wide range of potential variables, e.g. continuous variables, categorical variables, qualitative changes in condition, M&E indicators, program outputs, relevant external variables outside of the program’s control, context variables, etc. These factors are built into a model that identifies how every possible combination (both observed and unobserved) can be associated with the outcome of interest. QCA then applies an algorithm to this model to identify sets of necessary and sufficient combinations of conditions that can be logically associated with explaining how the outcome was achieved. Crucially, QCA typically identifies several combinations of factors simultaneously, which can be used to account for various contexts and explain how there may be multiple pathways to achieving the same outcome.

Whereas the impact evaluation is limited to testing the impact of only a few treatment arms at the Results level (i.e. R1, R1+R3, R1+R2+R3), QCA will be used to unpack how the various activities that fall underneath each result combine to achieve success.