Annex_C_-_Description_of_the_Previous_Evaluations.pdf

Annex C - Description of the Previous Evaluations

A. Baseline Evaluation

The Center for Global Health and Development (CGHD), in collaboration with Addis Ababa

University School of Social Work (AAU-SSW), conducted the baseline of a multi-year evaluation study from November 2011 through May 2012 to understand the situation of highly vulnerable children (HVC) and their households throughout the regions and city administrations of Ethiopia. This report of the study contains the findings of the baseline, which establishes bench marks for ongoing monitoring and evaluation of Yekokeb Birhan OVC project.

The evaluation was supposed to be a prospective, observational, mixed methods approach to follow children and households, measuring vulnerability and outcomes based on the receipt of care and support services over time. Additionally, the design was intended to call for observation of program implementation and documentation of key aspects of the intervention (i.e. services provided because of Yekokeb Berhan).

It was a longitudinal study design that allowed the evaluation team to document the dynamic situation of children and households throughout Ethiopia. Over time, the study was intended to identify which children and families receive support services, measure a range of impacts, examine how the program works, and examine the mechanisms whereby change occurs. The combined research activities were used to make study design maximizes internal and external validity by allowing the evaluators to triangulate data, verify findings, and confirm processes. It was also designed to allow the evaluation team to make recommendations to improve program performance at multiple points in time to maximize positive outcomes and impacts for children and families.

Methods

The evaluation study was conducted in 130 kebeles across 111 woredas throughout the 11 regions of Ethiopia. This national study was allowed the researchers to describe the situation of children and households throughout the country. The study sample was representative of the areas where Pact and Yekokeb Berhan is operational throughout Ethiopia.

Following the extensive training of the local team, the research team had conducted the following data collection activities:

Quantitative Household Survey (n=3,377)

Qualitative In Depth Interviews with Caregivers (n=90)

Community Mapping with Children (n=30)

Key Informant Interviews with Communities (n=120)

Focus Group Discussion with Community members (n=20)

Key Informant Interviews at Regional and Woredas Levels (n=22)

Next, the team had developed a detailed analytic plan to guide the quantitative and qualitative analysis. The plan was developed to prioritize analyses that answer the evaluation questions.

For quantitative data, they computed frequencies and means for all data and then disaggregated data by HVC status and poverty status in order to highlight differences and disparities between children and households in the sample. They calculated cross tabulations and means of all data by HVC and poverty status and by age and gender where appropriate, and present chi square tests for differences. They used the frequency sampling weights in all calculations to more accurately reflect the full study population of interest. In the household analysis, the finding present data by HVC household status and then, where appropriate, disaggregate data to show frequencies for all households and for the poorest households with HVC. Next, they calculated logistic regression models for key outcomes at the child and household level using Proc Survey

Logistic and Proc Survey Reg in SAS. Models account for the clustering at the kebele level and are weighted to reflect the overall population in each study area. Finally, models account for

HVC and poverty status, gender, age, and district location.

In the qualitative analysis, the researchers identified key themes for each study question and established a coding system for each qualitative component (IDIs, community maps, KIIs, and

FGDs) that allowed them to summarize relevant information for each question.

B. Midterm Evaluation

In 2013, Alliance for Better Health (ABH) Services was commissioned by Pact to undertake the midterm evaluation of Yekokeb Berhan Program to document progress against various outcome level indicators as well as changes in the lives of beneficiaries.

Methods

A combination of quantitative and qualitative methods was employed to collect data to determine the program’s achievements and to identify benchmarks that were not addressed in the baseline survey. Quantitative data were collected through a cross sectional survey of HVC households. In the qualitative assessment, focus group discussions and key informant interviews were conducted with beneficiaries, community volunteers, implementing partners (IPs), community committees and government office staff. Secondary data including the baseline survey, routine reports, supervision reports and other relevant program documents were also reviewed to complement the primary data in order to address the four midterm evaluation objectives.

The evaluation was conducted in four major target regions where the majority of beneficiaries of

Yekokeb Berhan Program are found: Amhara, Oromia, SNNPR, Tigray and the Addis Ababa

City Administration. Using a multi-stage sampling, sample size was determined to be 2,142 households for the household survey from five. Zones, woredas, kebeles and households were selected randomly.

Focus group discussions (FGDs) were conducted with caregivers, community committees (CCs), community volunteers (CVs) and saving groups in selected program sites, and a total of twenty sessions of FGDs were completed. Key informant interviews (KIIs) were conducted with thirty-four individuals representing: implementing partners; MoWCYA officers at federal, regional and woreda levels; international non-governmental organization (INGO) partners (Pact, FHI 360 and

Child Fund); and Yekokeb Berhan Program regional cluster managers.

Quantitative data were entered into computer software and analyzed; whereas, the qualitative data were transcribed and compiled under pre-defined thematic areas. Data from all sources were synchronized before producing the final report.