SOL-663-16-000026.pdf

SECTION B -- SUPPLIES OR SERVICES AND PRICES

B.1 TITLE

End-line Performance Evaluation: Support to Yekokeb Berhan Project for Highly Vulnerable Children, Award Number: AID-663-A-11-00005.

B.2 PURPOSE

The purpose of the End-line Evaluation is to analyze the effectiveness of the activities’ systems strengthening approach to improve access to quality services and health outcomes for HVC and their families and to inform the future project design.

B.3 CONTRACT TYPE

This is a Firm Fixed Price (FFP) type contract.

B.4 CONTRACT PRICE

The firm-fixed price for the performance of the work required is TBD. The contractor can invoice and be paid per the payment schedule.

B.5 PAYMENT SCHEDULE

Payment will be made upon successful completion of each deliverable (see Section F) in accordance with the following schedule:

• Twenty Five percent (25%) of the total contract amount upon satisfactory completion of Deliverable #1&2.

• Twenty Five percent (25%) of the total contract amount upon satisfactory completion of Deliverable #3&4.

• Twenty Five percent (25%) of the total contract amount upon satisfactory completion of Deliverable #5.

• Twenty Five percent (25%) of the total contract amount upon final satisfactory completion Deliverable #6&7 and/or the end of the period of performance of the contract, whichever is later.

Offerors can propose an alternative payment schedule, and provide a corresponding rational.

The final payment schedule will be determined by USAID at the time of contract award.

(End of Section B)

SECTION C -- DESCRIPTION/SPECIFICATIONS

End-line Performance Evaluation Activity: Yekokeb Berhan Project for Highly Vulnerable Children, Award Number: AID-663-A-11-00005

C.1 PURPOSE

USAID/Ethiopia seeks to procure and implement an end-line performance evaluation of the Yekokeb Berhan (YB) Highly Vulnerable Children (HVC) activity, the purpose of which is to analyze the effectiveness of the activities’ systems strengthening approach to improve access to quality services and health outcomes for HVC and their families and to inform the design of a follow-on activity. Results have been achieved through developing the capacity of local and regional government and community organizations, increasing access to services to HVC, and enhancing evidence utilization for program management. The evaluation will investigate results associated with this five year activity including the performance of the implementing partners in achieving the objectives as well as the capacity of local institutions to coordinate and sustainably meet the needs of HVC. Household level economic, health, and educational outcomes will be compared to pre-intervention measures and analyzed to determine the plausible contribution of this activity in reducing vulnerability and increasing resiliency among HVC and their families.

The primary audience for the results of this evaluation is USAID/Ethiopia who will use it, in conjunction with the previous baseline and mid-term evaluations and management reviews, to analyze factors contributing to improved health outcomes for HVC over the course of the activity and to draw lessons learned to inform the design of a follow-on activity. In addition, the implementing partners will use the findings to identify the areas of success and challenges in implementation and gain insights into how performance could be improved. Finally, the host country government, especially the Ministry of Women, Children and Youth Affairs (MoWCYA), HIV Prevention and Control Office (HAPCO), and other relevant local government stakeholders will use the analysis to identify opportunities to continue to improve services for HVC and their families and to inform planning for future HVC programming.

C.2 ACTIVITY INFORMATION

Activity Title: Yekokeb Berhan Program for Highly Vulnerable Children Award Number: AID-663-A-11-00005 Performance Period: April 20, 2011 – January 31, 2017 Total Estimated Cost (TEC):1 $92 million Implementing partner(s): Pact (Prime), FHI 360, Child Fund and 34 local CSO Geographic coverage: Amhara, Oromia, SNNPR, Tigray, Addis Ababa, & Gambella

C.3 BACKGROUND

According to UNICEF’s State of the World Children’s Report (2015), out of a total population of 91,728,000, there are an estimated 4.5 million orphans and vulnerable children in Ethiopia due to all causes. There are an estimated 900,000 HVCs that are orphaned as a direct result of AIDS.

There are also an estimated 170,000 children who are estimated to be living with AIDS. All

1 $92 million is the original Total Estimated Amount of the activity; however, all of the amount mentioned will not necessarily be obligated during the implementation period.

orphans and vulnerable children are at increased risk for neglect, abuse, malnutrition, poverty, illness and discrimination and, as they get older, are much more vulnerable to HIV infection.

The United States Agency for International Development (USAID), in partnership with the Government of Ethiopia is supporting the strengthening of systems to deliver needed services to Highly Vulnerable children and their families. The Yekokeb Berhan (YB) activity is implemented by Pact, in conjunction with FHI, Child Fund, and 34 local implementing partners. It is a $92 million, 5-year cooperative agreement. The activity’s performance period is from April 20, 2011 to January 31, 2017 and builds on previous HVC USAID investments. YB operates primarily in five regions and has satellite offices in these locations including Addis Ababa (located at the head office in Addis Ababa), Amhara (Bahir Dar), West Oromia (Jimma), SNNPR (Hawassa) and East Oromia (Adama). The activity established these offices to facilitate close monitoring of activities implemented by its partners and provide them with hands-on technical support.

The prime partner is Pact which has the overall mandate to implement the activity. The other two key partners are FHI360 and Child Fund which are international partners offering technical assistance in key programming areas, including design of guidelines and training materials and provision of follow-up technical support to other teams and partners. At the grass-roots level, Yekokeb Berhan staff assists implementing partners (IPs) and local governments in supervision, training and support for over 20,000 local community volunteers (CVs) who go home-to-home to share their skills and knowledge with beneficiary families, promote referrals, provide follow-up and develop one-on-one relationships with targeted children for supplementary care, encouragement and emotional support.

C.4 ACTIVITY GOALS, OBJECTIVES, KEY RESULTS, AND

IMPLEMENTATION APPROACH

The YB Activity takes a systems approach to meet the needs of HVC. The activity has four primary objectives: 1) build the capacity of stakeholders to effectively use improved data management systems and employ a national HVC supervision system; 2) employ effective and efficient family-centered, age-appropriate and inclusive HVC care management system; 3) enhance the capability of communities for coordinated and improved responsiveness towards HVC care; and 4) establish effective and efficient monitoring, evaluation, reporting and learning systems ensuring evidence- based programming and policy formulation. See Results Framework in Annex A.

The long-term goal of investments in HVC is to foster a sustainable, child-focused, social welfare framework in Ethiopia that allows all children to thrive so that HVC and their families can access appropriate and high-quality services and are empowered to lead healthy, productive, and fulfilling lives. To this end, the aims are to reduce vulnerability, increase resiliency, and strengthen systems and structures to deliver quality essential services to HVC and their families throughout Ethiopia in each year of the 5-year implementation period.

In the original activity plans, the geographic scope of the activity’s implementation was intended to include: Tigray, Afar, Amhara, Oromia, Somali, Benishangul Gumuz, Southern Nations Nationalities and Peoples’ Region (SNNPR), Gambella and Harari regions and Addis Ababa and Dire Dawa cities. However, due to declining PEPFAR funding and geographic focusing, the size of the target population for the activity has been adjusted from 500,000 to 325,000 HVC and the geographic scope has also been adjusted to account for reduced funding levels to include:

Amhara, Oromia, SNNPR, Tigray, Addis Ababa, and Gambella.

Children and families affected by HIV and AIDS, who are often the most vulnerable, are the primary target of the YB Activity. The YB Activity takes a family-centered and systems-development approach in its service delivery – breaking away from a traditional handouts approach in favor of a value-added approach that emphasizes sustainable, self-help methodologies that encourages access to locally available resources. This approach recognizes the fact that the wellbeing of a child depends on the wellbeing of other household members. HVC and their families receive age-appropriate, inclusive services, and children participate in discussions and decisions that affect their lives.

The program utilizes various approaches to provide appropriate services to HVC and their families, working with a wide range of government and civil society partners to establish and/or strengthen community capacity to coordinate and sustain the continuum of care for highly vulnerable children and their families.

C.5 DOCUMENTS FOR REVIEW

USAID Ethiopia will provide all the following reference documents for desk review, including:

o Evaluation of USAID/Pact Program for Highly Vulnerable Children: Yekokeb Berhan

Baseline Findings (September 2012) (http://www.bu.edu/cghd/files/2014/01/Yekokeb-Berhan-Baseline-Report-with-Annexes_Rev ised-May-2013.pdf) o Yekokeb Berhan Program for Highly Vulnerable Children in Ethiopia Mid-Term Evaluation Report (June 2014) (http://pdf.usaid.gov/pdf_docs/PA00K84N.pdf) o YB Mid-term Programmatic Review o Measuring The Organizational Development of NGOs Baseline Findings from Organizational

Development Assessments of Pact Ethiopia Yekokeb Berhan Partners (2013) o Annual Program Statement: Care Services for HIV-Infected and Affected Orphans and

Vulnerable Children o Quality Improvement Standard for HVC Services- Government of Ethiopia o Activity documents (Work plans, Performance Monitoring Plan) o Reporting documents (Semi-annual and annual progress reports, SIMS data, APR data) o Financial reports o Raw data as need to answer the proposed questions (including implementing partner level data, reporting data)

C.6 EVALUATION QUESTIONS

In principle, the end-line evaluation questions and methodology take as a basis the baseline and mid-term evaluations which will be used as points for comparison. However, there are some differences in both the evaluation questions and methodology used in the baseline and mid-line evaluations which this evaluation will need to take into account. In addition, this end-line evaluation will be a critical input to the design of a follow-on activity. As such the evaluation must be conducted and documented in such a way that the same or similar design, methods, and tools can be replicated in the future as part of potential evaluations of a follow-on activity. All of the necessary documentation must be created with this purpose in mind.

There are five main evaluation questions that are closely linked to the activity’s development hypothesis depicted in the Results Framework (see Annex A). The sub-questions are suggestions meant to help clarify the types of information that could be included in an analysis to respond to the main question. The evaluation team must collect data and present findings in relation to each of the five main evaluation questions.

http://www.bu.edu/cghd/files/2014/01/Yekokeb-Berhan-Baseline-Report-with-Annexes_Revised-May-2013.pdf http://www.bu.edu/cghd/files/2014/01/Yekokeb-Berhan-Baseline-Report-with-Annexes_Revised-May-2013.pdf http://pdf.usaid.gov/pdf_docs/PA00K84N.pdf

1) To what extent has the activity strengthened the capacity of implementing partners and regional/local government, including community care coalitions/community committees (CCC/CC), to identify, plan, coordinate, and respond to the needs of highly vulnerable children and their families?

a. What evidence exists showing improved capacity and increased performance or effectiveness of these entities to effectively implement program activities and achieve expected results? Are these capacities and resulting service delivery improvements likely to continue after closure of the YB Activity?

b. Which of the capacity building activities and approaches, such as different types of training and technical assistance, which were undertaken can be plausibly linked to the most significant improvements in the effectiveness of service delivery?

c. Are there any remaining gaps in technical, programmatic, management, financial and partnership capacities that were not addressed through the YB Activity and should be targeted by future activities?

2) How have the availability, accessibility, and quality of health and social services (including psychosocial services) for HVC in Ethiopia changed since 2012 (YB Activity baseline year), and how have YB interventions plausibly contributed to these changes?

a. Has the availability, accessibility, quality of services provided by the 34 local partner institutions improved? (quality should be assessed based on the definition by the Quality Improvement Standards for HVC in Ethiopia (2008))

b. What are the key remaining barriers, possibly including gender, to access high-quality health and social services for HVC?

3) To what extent are the services and support that are being provided improving health and educational outcomes of HVC and their families (comparing baseline, midterm and end-line findings)?

a. Are there any significant differentials by gender in the outcomes, and if so, what are the causes for this and what needs to be done to improve this?

b. The mid-term evaluation recommended emphasizing tutorial support, including partnering with local universities - was there any success in this regard since the midterm performance evaluation?

4) To what extent are economic strengthening activities effective in improving the capacity of households and HVC to meet their basic needs?

a. Are the economic strengthening activities plausibly contributing to enhancing the capacity of households caring for vulnerable children to generate income and increase resiliency to shocks?

b. Are there any significant differentials by gender (female headed households compared to male headed households), and if so, what are the causes for this and what needs to be done to address these differentials?

5) To what extent has Yekokeb Berhan enhanced the capacity of implementing partners to establish effective and efficient monitoring, evaluation, reporting and learning systems to inform program management?

a. How effective are the databases and systems to identify HVC in communities in need and to link them to appropriate services?

b. What is the level of information sharing between NGOs and implementing partners to ensure equity in service provision?

c. To what extent have gender appropriate metrics been incorporated into the implementing partners’ monitoring and evaluation systems?

d. To what extent are data generated from routine data collection meet quality standards and used for decisions-making?

Note: all data must be disaggregated by sex (male/female) to comply with the USAID Gender Policy and to better understand the service coverage and needs of HVC.

C.7 EVALUATION DESIGN AND METHODOLOGY

The evaluation should employ a mixed-methods design using both qualitative and quantitative measures that will document key measures of interest of the Yekokeb Berhan activity. Findings must demonstrate both achievements made and where improvements in the YB Activity theory, approach, and implementation practice are needed. These end line data collection, analysis, and interpretation efforts will also simultaneously serve as a baseline for the follow-on activity and will inform activity design. The end-line data collection efforts should use similar design, tools, and methods as were used for the mid-line data collection efforts to allow for comparability and interpretation of the contribution of these efforts in achieving results. [Use of the same methods does not necessarily mean that the same mechanism or partner will be used for subsequent measurement efforts, but simply that the same methods and tools should be employed, to the extent possible and practical].

Narrative descriptions of the illustrative evaluation design, methods, sampling, and questions are below. An illustrative evaluation design matrix which includes the questions, methods, and sources for indicators can be found in Annex B. This may be subject to change as evaluation key questions and sub-questions may be revised.

A range of varied methods, tools, and approaches were used in past evaluations and program reviews. The three previous evaluation and data collection and synthesis efforts for this HVC activity were varied in scope. These include: 1) Evaluation of USAID/Pact Activity for Highly Vulnerable Children: Yekokeb Berhan Baseline Findings (September 2012); 2) Yekokeb Berhan Program for Highly Vulnerable Children in Ethiopia Mid-Term Evaluation Report (June 2014);

and 3) YB Mid-term Programmatic Review. The first was meant to serve as a baseline to understand the economic and vulnerability status of children and families at the start of the program. The mid-term evaluation focused on programmatic results achieved at mid-point. The mid-term programmatic review was intended to compliment the mid-term evaluation by focusing on operational and management challenges and issues and providing relevant recommendations for mid-course correction. The end-line evaluation must collect data on selected indicators collected at baseline and mid-line and must assess the activity’s results after five years of implementation. Some areas of operational and management issues, particularly those that were prioritized in the Yekokeb Berhan Mid-Term Programmatic Review, must also be addressed. A description of previous evaluation and review efforts can be found in (Annex C).

Evaluation design

This is a non-experimental design that will focus on measuring and analyzing activity results before and after activity implementation using activity monitoring and survey data. The baseline data should be drawn from the baseline survey report, and may be supplemented as appropriate with retrospective methods.

Methodology

As stated above, the evaluation team will be responsible for defining an appropriate evaluation methodology. The team should consider mapping the research questions against the quantitative and qualitative data in a matrix/table to show how each research question will be answered (An illustrative evaluation matrix can be found in Annex B). Below are suggested data collection methods. In addition to these methods, the evaluators must propose rigorous methods that can add to and complement the data collected by the methods described below. Particularly, evaluators should incorporate innovative methods to assess the effectiveness of YB capacity building approaches on improved service delivery and sustainability in the absence of counterfactuals.

This evaluation employs mixed data collection and analyses methods design (quantitative and qualitative). These methods will collectively provide evidence, when triangulated, to capture current status of progress. Comparisons of midline and baseline data along with other types of analyzes (regression analysis, contribution analysis, triangulation of qualitative and quantitative data, etc.) must be able to provide plausible explanations for change in performance. While this is not an impact evaluation design and cannot definitively provide information of attribution of change as a direct result of the interventions of the activity, it must, all other factors being considered, provide plausible explanations for this change.

It is therefore critical that each of these reports, the methods used, and the data collection tools as well as the outcome indicators measured are reviewed in consultation with the Evaluation Contracting Officer’s Representative, the YB Agreement Officer Representative and in consultation with the Implementing Partner. This participatory and consultative process must be facilitated by the contractor and an agreement reached on the sampling frame, data collection measures of interest, methods employed, and data collection instruments developed. An analytical plan must be developed which will result in measures that will answer the key evaluation questions above and which can be replicated at mid-line and end-line of the follow-on activity to document progress towards achieving results.

Data collection tools

Data collection tools must be developed that will capture the information prescribed in the evaluation key and sub-questions. It is understood that some questions may not have been asked at the baseline or midline survey, and those measures may not be able to be compared, but they will serve as baseline measures for the follow-on HVC activity.

Gender disaggregation of any data collected and analyzed must take place to determine if there is a differential impact due to the intervention by gender. Further, gender sensitive indicators must be collected and analyzed as appropriate to accurately reflect gender integration and gender-related activity indicators.

Evaluation methods

It is anticipated that a range of methods will be used including but not limited to:

1) Desk Review of relevant activity and national guidance documents will be conducted;

2) Capacity Assessments of organizations according to various organizational development dimensions;

3) Non-experimental time series household survey of HVC households;

4) Focus Group Discussions of activity beneficiaries and others affected by the activity (HVC families and caregivers, Community Volunteers, Community Committees);

5) Key informant interviews of key stakeholders (USAID Mission staff, Pact, FHI 360, ChildFund, Ministry of Women, Children and Youth Affairs (MoWCYA), HIV Prevention and Control Office (HAPCO), Regional Health Office staff at relevant levels);

6) Resource mapping and field observations (update and use community committees' resource maps to assess the resources available in the community). Observations will provide first hand visualization of the results achieved on the ground.

7) Secondary data analysis of existing data including from the baseline and midline surveys as well as from routine quarterly and annual progress reports, as well as MER indicators.

Sampling frame for household survey

The sampling frame for the evaluation must be proposed by the evaluators, in close consultation with the USAID Mission and with Pact. The main purpose of the household survey is to provide data related to evaluation questions 2, 3 and 4 listed above. Efforts must be made to ensure that findings will be representative of YB intervention regions, zones, woredas, and kebeles.

The evaluators, in consultation with USAID Ethiopia and the implementing partner, must devise a sampling frame that will most closely provide data that can be generalizable to the intervention regions, zones, woredas, and kebeles. The primary sampling units will be: region, zone, woreda, kebeles, and households. The regions that were purposively selected for the mid-term evaluation were: Amhara, Oromia, SNNPR, Tigray, and Addis Ababa. This purposive selection will be discussed in light of each of the regions where YB operates, and will be determined in joint consultation. The next level sampling unit is the zones, which can also be selected purposively based on the numbers of HVC. However, the next lower level units of woredas, kebeles, and HVC households can be selected randomly based on the lists of households in geographic areas where YB is operating. Lists of HVC households will be acquired and used to do simple random sampling to mitigate bias and ensure generalizability. The sample size must be calculated using statistical software based on the size of the target population for the activity of 325,000 HVC and the geographic scope, to include: Amhara, Oromia, SNNPR, Tigray, Addis Ababa, and Gambella.

Sampling frame for qualitative data collection

Convenience samples and/or purposive sampling techniques will be used for the qualitative data collection. The selection of sites for qualitative data collection will ensure that a variety of locations, in the range of regions, zones, woredas, and kebeles where the intervention operates, taking into account inclusion of sites with varied demographic, economic, and cultural characteristics.

Focus group discussions and key informant interviews must be conducted with beneficiaries, community volunteers, community committees, and host government counterparts at the local levels.

Ethical clearances and human subjects protection

The ethical clearances and approvals needed for data collection amongst minors who are also highly vulnerable children must be sought through the proper channels within the Ministry of Women, Children, and Youth Affairs. Head of households in select households as well as HVC above the age of 14 will be interviewed. Informed consent procedures must be developed and implemented to ensure adequate protection of human subjects. Informed consent procedures may be in writing or may be verbal according to the ethical norms of data collection with minors in Ethiopia.

Analytical methods

It is required to propose analytical methods which will be reviewed and approved by the USAID Mission. These methods should include a comparative analysis of selected measures that demonstrate differences between baseline, midline, and endline measures wherever possible.

Relevant explanations must consider the long time difference between the baseline and end line measures for some of the data, and the shorter period of time difference for those data that were not captured at baseline but were captured only at midline. The analysis must disaggregate indicators by gender and/or address gender-specific dimensions of findings, as appropriate.

Limitations of the Evaluation Design and Methodology

The baseline and mid-term survey may lack some of the indicators that this final evaluation will assess and therefore the before-after analysis may not be possible for all indicators. Evaluators should reconstruct the baseline and mid-term values for such indicators using a retrospective approach when possible. However, retrospectively constructed baseline data has important data quality limitations and should be used with caution.

Also, there were some differences in the methodologies used for the baseline and mid-line surveys so evaluators will need to assess and, if possible, minimize those differences in order to compare the two data points with the end-line survey.

Finally, assessing the impact of capacity building activities on service delivery and sustainability is a challenge, particularly since this is not an impact evaluation. While this evaluation is expected to utilize innovative approaches to investigate plausible cause-effect linkages between some of the capacity building activities and improved service delivery, these relationships must be interpreted with caution as there are many other factors outside the YB Activity’s influence that may have contributed to service delivery improvements. The use of both qualitative and quantitative data collection methods will strengthen and reinforce the rigor of the data and validity of the findings. However, any data limitations associated with using either method must be clearly documented as part of the final evaluation report.

C.8 EVALUATION TEAM COMPOSITION

The evaluation team must consist of four consultants and as many data collectors/field supervisors as deemed necessary to collect survey data. The evaluation team and data collectors must comprise both males and females that have demonstrated knowledge and experience in the areas described below. The suggested team composition will include 1 team leader, 1 OVC/HVC specialist, 1 evaluation specialist, 1 organizational capacity development specialist, and approximately 20 data collectors and 4-5 field supervisors2 with collective skills that encompass the following competencies: evaluation design and implementation, quantitative and qualitative data collection and analysis, organizational development, and knowledge of USAID health programs. One of the key members of the evaluation team must have expertise in the area of gender sensitive data collection, analysis and reporting.

2 The number of data collectors and field supervisors will need to be revised to take into account the final sample size and logistical considerations to complete data collection within the specified timeframe.

Team Leader The Team Leader will oversee all aspects of the evaluation and will be the main point of contact responsible for primary communications with USAID Ethiopia staff providing oversight in the design, data collection, and data analysis process. The Team Leader will also be responsible for compiling and submitting a draft evaluation report, integrating USAID feedback into the final report, and providing an oral presentation of the preliminary results to USAID/Ethiopia prior to departure. (Note: This person may be selected from among the other key staff, and will meet the requirements of both this and the other position. This person may be an international or a local expert).

OVC/HVC Specialist The OVC/HVC Specialist will participate with the Team Leader in helping to design the study, review and finalize the data collection instruments, collect, analyze, and interpret findings, as well as draft various components of the final evaluation report under the direction of the Team Leader. (Note: This team member may be an international or local expert).

Evaluation Specialist The Evaluation Specialist will participate with the Team Leader in helping to design and structure the evaluation, review methodologies, collect, analyze, and interpret findings, as draft various components of the final evaluation report under the direction of the Team Leader.

Organizational Development Specialist The Organizational Development Specialist will focus on the questions specific to capacity building of implementing partners and host-government institutions. She/he will be expected to assist the team leader in all aspects of design, data collection, analysis, and write up of the capacity building/strengthening component of the evaluation.

In addition to the required four (4) Key Personnel positions, the offeror may choose to employ additional staff such as data collectors/clerks, research assistants, logistics personnel and administrative support personnel. Offerors must provide a detailed LOE chart showing Total LOE by task and position as shown in the illustrative table below.

Field Supervisors (approximately 5) (The number of field supervisors will depend on sampling frame and logistical plan. This number will need to be revised based on calculations at a later date)

The field supervisors will be critical to ensure the data quality of the data collection efforts for the cross-sectional household survey as well as the qualitative data collection efforts. Qualifications for the Field Supervisors include:

1. At least secondary school or equivalent required, bachelors in public health or social work preferred;

2. At least three years of experience supervising data collection efforts at household level;

3. Knowledge of Microsoft Excel and other data collection tools desired;

4. Excellent oral and written communication skills in local languages of the respective regions;

5. Willingness to travel to sites for data collection.

Data Collectors/enumerators (approximately 20) (The number of enumerators will depend on sampling frame and will need to be revised based on calculations at a later date)

The Data Collectors/enumerators will be invaluable for the cross-sectional household survey as well as the qualitative data collection efforts. Qualifications for the Data Collectors include:

1. At least secondary school or equivalent required, bachelors in public health or social work preferred;

2. Experience conducting and/or participating in data collection efforts at household level;

3. Proven ability to conduct interviews;

4. Knowledge of Microsoft Excel and other data collection tools desired;

5. Excellent oral and written communication skills in local languages of the respective regions;

6. Willingness to travel to sites for data collection.

The selected contractor may choose to hire additional staff to support the Evaluation, such as: a local logistics coordinator, interpreters/translators, and administrative assistants. If needed, offerors may propose any additional staff in their budget, which will be considered by USAID for cost reasonableness.

In addition to the members of the team listed above, USAID staff members, depending on availability and timing, may also participate on the evaluation team. Proper approvals will be sought from staff supervisors for their ability to participate as members of the evaluation team including: team planning, data collection, data analysis, and report writing.

Level of Effort Table 1: Illustrative LOE by Task and Position Activity Team

Leader

HVC

Specialist Evaluation Specialist

Org.

Development

Specialist

Data Collectors

(approx. 20)3

Field Supervisors (approx. 3)

TOTAL

Desk review, draft work plan, data collection tools

5 5 5 5 0 0 20

International Travel to Addis Ababa (if applicable)

2 2 2 2 0 0 8

Team planning meeting

5 5 5 5 0 0 20

Finalization of work plan, data collection tools

2 2 2 2 0 0 8

Data collection 15 15 15 15 300 45 390 Data analysis 10 10 10 10 200 30 270 Presentation of key findings (out brief)

1 1 1 1 0 0 4

International travel from Addis (if applicable)

2 2 2 2 0 0 8

Development and submission of draft report

5 5 5 5 0 0 20

Incorporate USAID feedback and finalize

5 0 0 0 0 0 5

TOTAL* 52 47 47 47 500 75 753

3 This LOE is based on 20 data collectors and 3 field supervisors; however, the number may be revised later depending on the sample size.

C.9 CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT

USAID Evaluation Tools, including the checklist to ensure evaluation report, will be used to assess report quality and compliance with the USAID Evaluation Policy expectations (See Checklist for Assessing USAID Evaluation Reports, http://usaidlearninglab.org/sites/default/files/resource/files/mod11_checklist_for_assessi ng_eevaluatio_reports.pdf ).

Likewise, the PEPFAR Evaluation Standards of Practice Adherence Checklist will be used to assess the report quality against the 11 PEPFAR evaluation standards. The checklist can be found here (http://www.pepfar.gov/documents/organization/221324.pdf).

C.10 USAID MANAGEMENT

The evaluation team will officially report to designated COR. The COR will provide overall direction to the evaluation team. From a technical management perspective, the evaluation contractor is responsible for all direct coordination with the USAID/Ethiopia’s Support to Yekokeb Berhan Program for Highly Vulnerable Children End line Evaluation’s COR and alternate COR.

USAID/Ethiopia:

• Meets with the evaluator prior to start of evaluation to provide input on work plan, tools, sampling, etc;

• Acts as liaison on activities and approval of deliverables with Pact, evaluation contractor, USAID/Ethiopia, and the Yekokeb Berhan team;

• Oversees cooperative agreement with Evaluation Coordinator including all communication with implementing partner;

• Reviews/Approves all deliverables

• Disseminates results via DEC

• Schedule the internal meetings

C.11 LOGISTICS

USAID/Ethiopia will not provide workspace for the evaluation team at the Mission except for initial in-briefing, debriefing and meetings with USAID staff. External meeting space and printing and photocopying services may be provided by the partner or through local hotels and printers. The implementing partner (PACT) will be responsible for assisting in scheduling site visits and the provision of internal program documents.

(End of Section C) http://usaidlearninglab.org/sites/default/files/resource/files/mod11_checklist_for_assessing_evaluation_reports.pdf http://usaidlearninglab.org/sites/default/files/resource/files/mod11_checklist_for_assessing_evaluation_reports.pdf http://usaidlearninglab.org/sites/default/files/resource/files/mod11_checklist_for_assessing_eevaluatio_reports.pdf http://usaidlearninglab.org/sites/default/files/resource/files/mod11_checklist_for_assessing_eevaluatio_reports.pdf http://www.pepfar.gov/documents/organization/221324.pdf http://www.pepfar.gov/documents/organization/221324.pdf

SECTION D -- PACKAGING AND MARKING

D.1 AIDAR 752.7009 MARKING (JAN 1993)

(a) It is USAID policy that USAID-financed commodities and shipping containers, and project construction sites and other project locations be suitably marked with the USAID emblem.

Shipping containers are also to be marked with the last five digits of the USAID financing document number. As a general rule, marking is not required for raw materials shipped in bulk (such as coal, grain, etc.), or for semifinished products which are not packaged.

(b) Specific guidance on marking requirements should be obtained prior to procurement of commodities to be shipped, and as early as possible for project construction sites and other project locations. This guidance will be provided through the cognizant technical office indicated on the cover page of this contract, or by the Mission Director in the Cooperating Country to which commodities are being shipped, or in which the project site is located.

(c) Authority to waive marking requirements is vested with the Regional Assistant Administrators, and with Mission Directors.

(d) A copy of any specific marking instructions or waivers from marking requirements is to be sent to the Contracting Officer; the original should be retained by the Contractor.

D.2 BRANDING AND MARKING POLICY

In accordance with provision D.1 above, and where applicable, the Contractor shall comply with the requirements of the policy directives and required procedures outlined in USAID Automated Directive System (ADS) 320.3.2 “Branding and Marking in USAID Direct Contracting” (version from January 8, 2007) at http://www.usaid.gov/policy/ads/300/320.pdf ; and USAID “Graphic Standards Manual” available at www.usaid.gov/branding , or any successor branding policy.

(End of Section D) http://www.usaid.gov/policy/ads/300/320.pdf http://www.usaid.gov/branding

SECTION E -- INSPECTION AND ACCEPTANCE

E.1 NOTICE LISTING CONTRACT CLAUSES INCORPORATED BY REFERENCE

The following contract clauses pertinent to this section are hereby incorporated by reference (by Citation Number, Title, and Date) in accordance with the clause at FAR "52.252-2 CLAUSES INCORPORATED BY REFERENCE" in Section I of this contract. See FAR 52.252-2 for an internet address (if specified) for electronic access to the full text of a clause.

NUMBER TITLE

FEDERAL ACQUISITION REGULATION (48 CFR Chapter 1)

DATE

52.246-4 Inspection of Services – Fixed Price. AUG 1996

E.2 INSPECTION AND ACCEPTANCE

USAID inspection and acceptance of services, reports, and other required deliverables or outputs shall take place at:

USAID/Ethiopia US Embassy Entoto Street P. O. Box 1014 Addis Ababa, Ethiopia and at any other location where the services are performed and reports and deliverables or outputs are produced or submitted. The COR, TO BE DESIGNATED DURING AWARD, will have a delegated authority to inspect and accept all services, reports, and required deliverables or outputs.

(END OF SECTION E)

SECTION F -- DELIVERIES OR PERFORMANCE

F.1 DELIVERABLES AND REPORTING REQUIREMENTS

The selected offeror will be responsible for submitting the following deliverables, all requiring final approval by USAID Ethiopia. These include:

F.1.1 Draft evaluation design and methodology /work plan and data collection tools:

Prior to start of field work, the evaluation team will read the relevant activity documents, hold several calls with the Mission to review the SOW, receive relevant documents, and prepare the protocol/work plan. The team must submit the protocol/work plan with a detailed data collection plan, analysis plan; human subject’s projection processes and tools, data integrity plan, and data collection tools. Dissemination and utilization plan /strategy must be included in the data collection plan. Documents must be sent to USAID Ethiopia and USAID/W prior to the team planning meeting for review and feedback and must provide at least 10 days for this process.

F.1.2 In brief/Team Planning Meeting (TPM): This will take place during the start of the evaluation. The evaluation team must coordinate and organize space for this team planning meeting as well as an agenda. All documents must be shared with USAID Ethiopia prior to the Team Planning Meeting. The work plan, tools, methods, sampling plan, and analysis plan will be finalized in conjunction with USAID Ethiopia during these initial planning meetings. The YB Activity contractor may also be consulted and/or invited to appropriate sessions of the TPM.

F.1.3 Progress reports: The USAID Ethiopia Mission must receive a progress report on the pre-testing of survey instruments and training of data collectors as well as bi-weekly progress updates during data collection and analysis to ensure that evaluators are on the right track, to provide guidance on issues that may arise during evaluation implementation, and to ensure that expectations are being met.

The details of when, how, and with whom this communication should occur will be discussed and agreed upon during the team planning meeting.

F.1.4 Out brief with USAID/Ethiopia and other relevant stakeholders, including

Government counterparts: The evaluation team will conduct data analysis simultaneously with data collection in order to generate preliminary results prior to finalization. Preliminary findings will be presented during the out brief meeting with USAID/Ethiopia. A separate out brief meeting with external relevant stakeholders will be organized by USAID and delivered by the Evaluation Team.

The preliminary Power Point presentation must be shared with USAID Ethiopia and USAID/W prior to the day of these presentations.

F.1.5 Draft evaluation report: The evaluation team must prepare and submit a draft report to USAID Ethiopia and USAID/W at the end of one week after the out brief presentation. Contents of the report will include the following sections: 1) Table of contents and Acronyms, 2) Executive summary, 3) Background, 4) Evaluation questions, 5) Methodology, 6) Findings, 7) Conclusion, and 8) Recommendations.

The SOW, data collection tools including informed consent forms if applicable, work plan and data sources (i.e. list of key informant interviews) must be included as an annex. Specific reporting requirements must follow the USAID Evaluation

Policy and USAID Evaluation Report Template, as well as the PEPFAR Evaluation Standards of Practice required components of an evaluation report. These will be provided to the evaluation team.

F.1.6 Provision of feedback: USAID/Ethiopia and/or USAID/W will provide comments to the evaluation team within 10 days of submission. The draft evaluation report must be submitted by the evaluation team leader through the Evaluation COR.

F.1.7 Final evaluation report: The final report must incorporate comments provided by USAID/Ethiopia. The length of the final evaluation report should not be more than 30 pages, not including Annexes and the Executive Summary. USAID comments are due within six business days after the receipt of the initial final draft. The final report must be submitted to the COR within five business days from the receipt of comments from USAID. All activity data and records must be submitted in full and should be in electronic form in an easily readable format; organized and fully documented for use by those not fully familiar with the activity or evaluation; and owned by USAID and made available to the public. Within one week of receiving comments and feedback, the evaluation team leader must revise and submit a final report to USAID/Ethiopia and USAID/W. USAID/Ethiopia will post the final approved report onto the USAID Development Experience Clearinghouse (DEC) as mandated by the Evaluation Policy and the Evaluation Mission Order/ USAID/Ethiopia.

F.1.8 One-page briefer on key qualitative and quantitative findings and conclusions related to the evaluation questions, for each municipality is included in the evaluation’s scope—to be given to the appropriate government counterpart(s) so that they have the opportunity to review evaluation findings and share them with the larger community. Each briefer will be reviewed by the Program Office and Health, AIDS, Population and Nutrition (HAPN) technical office prior to distribution.

F.1.9 Each of the reports shall be submitted electronically using Microsoft Word, Excel, or PowerPoint software. All reports must be written in plain, grammatically correct English. All reports must be approved by the COR.

F.1.10 The final evaluation report must be submitted in accordance with all the requirements of Section F and be submitted to the Development Experience Clearinghouse no later than 5 days after the approval of the final report by the COR. An electronic copy must be submitted either by:

(a) Via E-mail: docsubmit@dec.cdie.org ;

(b) Via U.S. Postal Service: Development Experience Clearinghouse, 8403 Colesville Road, Suite 210 Silver Spring, MD 20910, USA; or Online: http://www.dec.org/index.cfm?fuseaction=docSubmit.home .

F.2 REPORTING GUIDELINES

The format of the report should be consistent with the USAID branding guidelines.

The evaluation report must represent a thoughtful, well-researched and well- organized effort to objectively evaluate what worked in the activity over the given time period, what did not, and why.

mailto:docsubmit@dec.cdie.org http://www.dec.org/index.cfm?fuseaction=docSubmit.home

Evaluation reports must address all evaluation questions included in the statement of work, or as revised in agreement with USAID/Ethiopia.

The evaluation report must include the statement of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the Program Office.

Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides must be included in an annex in the final report.

Evaluation findings must assess outcomes and impact on males and females, and data must be disaggregated by gender, age group, and geographic area wherever feasible.

Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).

Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative and/or qualitative evidence.

Sources of information, including any peer-reviewed or grey literature, should be properly identified and listed in an annex.

Recommendations must be supported by a specific set of findings. They must also be action-oriented, practical, and specific, with defined responsible parties for each action.

F.3 DELIVERABLES AND TIMEFRAME

Deliverables Timeframe #1. Draft evaluation design and methodology /work plan and data collection tools

Within 10 working days of award date

#2. In brief/Team Planning Meeting Within two (2) working days of submission of Deliverable #1

#3. Progress report on pretesting of survey instruments and training data collectors

Within five (5) working days of USAID approval of Deliverable #1

#4. Progress report on data collection and analysis

Biweekly, within 30 working days of submission of Deliverable #3

#5. Out brief of preliminary findings Within three (3) working days of submission of Deliverable #4

#6. Draft evaluation report Within five (5) working days of completion of Deliverable #5

#7. Final evaluation report and one-page briefer

Within five (5) working days after receiving USAID feedback on Deliverable #6

F.4 EVALUATION SCHEDULE

A detailed evaluation timeline/schedule will be developed during the Team Planning Meeting.

The period of performance for the evaluation is expected to be from Mid of October 2016 to end of December 2016.

F.5 PLACE OF PERFORMANCE

The place of performance under this contract is Ethiopia/663.

F.6 PERIOD OF PERFORMANCE

The period of performance (PoP) for this contract is 75 days at the maximum from the date of award.

F.7 PERFORMANCE STANDARDS

Evaluation of the Contractor's overall performance toward achievement of the objectives in Section C will be conducted jointly by the COR and the Contracting Officer, and shall form the basis of the Contractor's performance evaluation record with regard to this contract. Evaluation of the overall performance will be in accordance with FAR 42.15 and corresponding USAID procedures. The Contractor’s performance will be evaluated at contract completion, utilizing the following general performance factors:

1. Quality of Product or Service, including quality and effectiveness of technical assistance provided to project clients, extent to which Contractor takes advantage of opportunities in managing for results, and degree to which key personnel provided are consistent with the names mentioned in the proposal.

2. Cost Control/Effectiveness, including actual costs incurred against the…