Annex_B_-_Yekokeb_Berhan_End_line_Evaluation_Design_Matrix.pdf

ANNEX B: Yekokeb Berhan End line Evaluation Design Matrix

Evaluation Question Illustrative Methods1 Sources for indicators/measures2 To what extent has the activity been able to build the capacity of implementing partners and regional/local government including CCC/CC to identify, plan, coordinate, and respond to the needs of highly vulnerable children and their families?

a. What evidence exists showing improved capacity and increased performance or effectiveness of these entities to implement program activities and achieve expected results? Are these capacities and resulting service delivery improvements likely to continue after closure of YB activity?

b. Which of the capacity building activities and approaches, such as different types of training and technical assistance, that were undertaken can be plausibly linked to the most significant improvements in the effectiveness of service delivery?

c. Are there any remaining gaps in technical, programmatic, management, financial and partnership capacities that were not addressed through the YB Activity and should be targeted by future activities?

Organizational Capacity Assessments (repeat of previous measures of capacity with the same organizations as baseline)

Assessment of organizational performance through review of existing PMP data and collection of additional data

FGDs with local partner organization staff and regional/local government, particularly those who have participated in YB trainings

KIIs with local partner organization staff and regional/local government, particularly those whom have participated in YB trainings

Contribution analysis, regression analysis, as feasible

Measuring the Organizational Development of NGOs Baseline Findings from Organizational Development Assessments Of Pact Ethiopia Yekokeb Berhan Partners (2013)

PMP indicators

What is the availability, accessibility, and quality of health and social services for HVC in Ethiopia?

a. Has the availability, accessibility, quality of services provided by the 34 local partner institutions improved since the beginning of YB ? (quality should be assessed based on the definition by the Quality Improvement Standards for HVC in Ethiopia (2008)

- Availability – Analysis of geographic service delivery data and resource maps

- Accessibility- Analysis of household survey data

- Quality – Direct observation of service delivery using structured observations

- Service Provision Assessment study (if available)

- PMP indicators (Annex C)

- SIMS

- DHS indicators

1 The table includes suggested methods; however, USAID Ethiopia encourages innovative methods and data collection sequencing strategies to answer the evaluation questions.

2 The offeror is encouraged to expand this table to include particular indicators/assessment criteria for each question and sampling methods.

b. What are the key remaining barriers, possibly including gender, to access high-quality health and social services for HVC?

FGDs KIIs

To what extent are economic strengthening activities effective in improving the quality of life of households and HVC within them?

a. Are the economic strengthening activities plausibly contributing to enhancing the capacity of households caring for vulnerable children to generate income and increase resiliency to shocks?

b. Are there any significant differentials by gender (female headed households compared to male headed households), and if so, what are the causes for this and what needs to be done to address these differentials?

Household Survey

FGDs KIIs

PMP indicators (Annex C)

To what extent are the services and support that are being provided improving health and educational outcomes and making meaningful change in the lives of HVC and their families?

a. Are there any significant differentials by gender in the outcomes, and if so, what are the causes for this and what needs to be done to improve this?

b. The mid-term evaluation recommended emphasizing tutorial support, including partnering with local universities - was there any success in this regard since the midterm performance evaluation?

Data abstraction of health outcomes

FGDs KIIs

Household Survey

PMP indicators (Annex C)

DHS

To what extent has Yekokeb Berhan enhanced the capacity of implementing partners to establish effective and efficient monitoring, evaluation, reporting and learning systems to inform program management?

a. How effective are the databases and systems to identify HVC in communities in need and to link them to

Review of data collection, reporting, and use processes on-site (paper, electronic)

Review of outputs of MERL processes via review of progress reports, annual reports

PMP indicators (Annex C)

SIMS

appropriate services?

b. What is the level of information sharing between NGOs and implementing partners to ensure equity in service provision?

c. To what extent have gender appropriate metrics been incorporated into the implementing partners’ monitoring and evaluation systems?

d. To what extent are data generated from routine data collection meet quality standards and used for decisions-making?

FGDs

KIIs

c. Are there any significant differentials by gender in the outcomes, and if so, what are the causes for this and what needs to be done to improve this?

The mid-term evaluation recommended emphasizing tutorial support, including partnering with local universities - was there any success in this regard since the midterm performance evaluation?