RFQ_No._95332419Q0086_Section_J__Attachments_J.1._-_J.12..pdf

Cote d’Ivorie Secondary Education Activity Evaluation Request for Quotation No. 95332419Q0086

Section J – List of Attachments

J.1. Evaluation Management and Review Process

J.2. Evaluator Progress Report and Work Plan

J.3. Evaluation Services Travel Scope of Work

J.4. Project Evaluability Assessment Tool

J.5. Independent Evaluation Design Report Outline

J.6. Independent Evaluation Baseline Report Outline

J.7. Independent Evaluation Final Report Outline

J.8. Public MCC Data Documentation & Materials: available online here:

https://data.mcc.gov/evaluations/index.php/catalog/155

These Include:

MCC Restricted Use Data Form

MCC Evaluation Microdata 1-page Checklist

DRB Data Package and Worksheet MCC Evaluation Microdata Guidelines

Disclosure Review Board Charter

Short Quantitative Informed Consent Statement

J.9. Nesstar Tutorial

J.10. Nesstar Metadata Template: (not attached as it is a software file, available on request from MCC)

J.11. Skills for Employability and Productivity Project Program Logic

J.12. Figure 1. Secondary Education Activity Program Logic

J.13. Past Performance Questionnaire https://data.mcc.gov/evaluations/index.php/catalog/155 https://data.mcc.gov/evaluations/index.php/catalog/155

MCC Independent Evaluations Evaluation Management Process

(EMP) Version: July 2017

Contents 1 Overview

1.1 Roles and Responsibilities

1.2 Evaluation Management Process (EMP) Objectives

2 Evaluation Management Committee (EMC)

3 Evaluation Milestones and Risk Factors

3.1 Evaluability Assessment

3.2 Evaluation SOW + IGCE

3.3 Evaluation Design Report

3.4 Baseline Materials and Data Collection

3.5 Baseline Report + Data

3.6 Endline Materials and Data Collection

3.7 Final Evaluation Report + Data

4 Summary of Risks over Evaluation Life Cycle

ATTACHMENT J.1. Evaluation Management and Review Process

Evaluation Management Process (EMP)

Version: July 2017

1 Overview The Millennium Challenge Corporation (MCC) invests in independent evaluations to measure results of MCC-financed and Millennium Challenge Account (MCA)-executed programs. The independent evaluations are conducted by third-party experts to produce high quality, credible analysis that hold MCC and country partners accountable for whether or not intended results of the program occurred and contribute directly to learning why and why not for future decision-making. Therefore, each evaluation is guided by three questions:

• Was the program implemented according to plan (in terms of quantity and quality of outputs)?

• Were there any changes in key outcomes for program participants that may be attributable to MCC’s investment?

• Why did or didn’t the investments lead to changes in key outcomes, such as income?

These guidelines outline MCC’s Evaluation Management Process (EMP) to define the (i) roles and responsibilities of the independent evaluators, MCC, and MCAs, (ii) objectives of the formal MCC review process, and (iii) the milestones and risk factors that inform how MCC manages the evaluation life cycle with these objectives in mind.

1.1 Roles and Responsibilities

MCC, MCAs, and independent evaluator - as well as other key stakeholders - play critical roles in designing, implementing, and disseminating the independent evaluations. Their roles and responsibilities are defined in the Evaluator Scope of Work (SOW). Generally, they are:

Millennium Challenge Corporation (MCC): MCC is responsible for oversight of the Independent Evaluator and quality control of evaluation activities, including the following specific responsibilities:

1. Assess when a program is ready for evaluation planning through evaluability assessment;

2. Determine what program components (Project(s), Activities, Sub-Activities) will be covered by the evaluation;

3. Set the evaluation questions to achieve accountability and learning objectives;

4. Build buy-in and ownership of the evaluation;

5. Contract and Supervise the Independent Evaluator;

6. Conduct quality reviews of all evaluation products (reports, questionnaires, etc);

7. Facilitate public dissemination efforts to inform decision-makers on learning generated by the evaluation;

8. Oversee preparation of public-use data that ensures appropriate balance of enabling verification of analysis and broad use of the data with adherence to promises of confidentiality to survey respondents;

and

9. Determine the evaluation budgets.

http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F5%5FTemplates%20and%20Examples%2F0%2E%20TORs%2FEvaluation%20Firm%20TORs

Version: July 2017

MCC M&E ensures appropriate documentation and management of its Evaluation Portfolio in the Evaluation Pipeline. Although historically excel-based, MCC M&E is working on migrating to a web-based platform with final data migration expected by December 2017:

• Evaluation Pipeline (excel database)

• Evaluation Pipeline (web-based application – December 2017 launch)

Millennium Challenge Account (MCA) and country partners: MCA and country partners are responsible for identifying users of the evaluation, building local ownership and commitment to the evaluation, oversight of the data collection firm, and quality control of evaluation activities, including the following specific responsibilities:

1. Assess when a program is ready for evaluation planning through evaluability assessment;

2. Set the evaluation questions to achieve accountability and learning objectives;

3. Build buy-in and ownership of the evaluation;

4. Manage the data collection firms (as applicable during Compact);

5. Conduct quality reviews of all evaluation products (reports, questionnaires, etc); and

6. Facilitate public dissemination efforts.

Independent Evaluator: The Independent Evaluator is responsible for the overall design, implementation, and dissemination of the evaluation, including the following specific responsibilities:

1. Finalize evaluation questions to meet needs and demand from MCC and country partners;

2. Develop the most rigorous evaluation design feasible given rules of program implementation;

3. Support MCC and MCA to build buy-in and ownership of evaluation;

4. Develop evaluation materials that are held to international standards;

5. Ensure Institutional Review Board clearance of survey protocols;

7. Manage the data collection firms (as applicable);

6. Ensure data quality during collection and entry through supervisions and management;

7. Lead data cleaning, analysis, and interpretation of results;

8. Produce high quality, credible, transparent evaluation reports;

10. Lead preparation of public-use data that ensures appropriate balance of enabling verification of analysis and broad use of the data with adherence to promises of confidentiality to survey respondents;

and

9. Lead public dissemination efforts.

http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F2%5FEvaluation%20Pipeline&FolderCTID=0x012000A21A838349721745BD00C8A6005A979C&View=%7b15A669B9-9475-4D1A-B7A4-768995B79014%7d http://evalpipeline.mcc.gov/#/viewEvaluations

Version: July 2017

1.2 Evaluation Management Process (EMP) Objectives

The objectives of the EMP stem from the core lessons learned from the “First Five Ag” impact evaluations (please also reference Sturdy, Aquino, and Molyneaux 2014):

Ensure clearly defined program. This requires clearly defining the inputs, outputs, outcomes, and goals of each intervention in a way that demonstrates how the intervention will lead to expected results. For this, there must be common understanding across stakeholders on the problem diagnostic, program logic, targeted populations and their geographic locations, metrics for measuring results, and risks and assumptions.

Ensure early integration of evaluation with program design and implementation. The evaluation methodology and timeline is closely linked to program design and implementation. This requires close collaboration between M&E and Operations (both in MCC and MCA) as early as possible in the process once the essential elements of the project are defined (who will be targeted, where, how, and what are the essential project design elements), and then continuously throughout Compact development, implementation, and Post-Compact.

Ensure evaluations are structured for learning and work toward establishing a feedback loop. This requires identifying the consumers of evaluation results and ensuring a feedback loop is established so that evaluation results can inform future decision-making.

Ensure appropriate quality control mechanisms are in place. This requires making sure that the appropriate stakeholders are reviewing key evaluation deliverables, such as evaluation design reports, and assessing risks to the interventions and their evaluations, in a timely manner.

To meet these objectives, the EMP defines:

Who is involved? The MCC EMP defines who from Management, Operations, M&E, and Economic Analysis needs to review critical evaluation products/deliverables to ensure accountability and learning objectives are met.

When do they need to be involved? The MCC EMP defines the critical milestones in every evaluation life cycle when the EMC reviews and provides feedback on evaluation products/deliverables.

What are the risk factors? The MCC EMP defines the risk factors facing each evaluation at each milestone to facilitate MCC assessment as to whether the expected benefits of the evaluation are worth the costs. These risk factors often accumulate over time.

https://assets.mcc.gov/content/uploads/2017/05/issuebrief-2012002123401-principles-impact-ag.pdf http://www.tandfonline.com/doi/full/10.1080/19439342.2014.975424

Version: July 2017

2 Evaluation Management Committee (EMC) An EMC is established for each independent evaluation comprised of one Chair and five or more Committee members. The EMC should be established as early in Compact Development as necessary for each Evaluation, and members should expect to participate in the EMC for the duration of the evaluation life time (5+ years). If an EMC member must leave the EMC, the Chair, in collaboration with DPE and DCO management, must identify the appropriate replacement.

Chair. The MCC M&E Managing Director (or designated representative) is responsible for decisions related to investments in the independent evaluations. Using input from EMC members, the Chair decides if the expected benefits of the evaluation outweigh the costs.

Members1 Monitoring and Evaluation (M&E) Lead. The M&E lead is responsible for overall management of the Compact’s independent evaluation activities and ensuring common understanding of the program logic(s) and evaluation questions, contracting Evaluation Firm(s), and ensuring quality control for evaluation materials and reports.

Evaluation Technical Lead. MCC has designated the following Technical Leads in each of the sectors for the evaluation portfolio. These Leads are responsible for sharing best practices in their sector(s) across different country evaluations:

• Jack Molyneaux: Agriculture & Irrigation; Transportation; Financial, Investment, and Trade; Capacity

Building and Institutional Development;

• Shreena Patel: Health, Energy, and Water and Sanitation;

• Ryan Moore: Education; and

• Jennifer Lisher: Land

Economic Analysis (EA) Lead. The EA lead is responsible for assessing technical evaluation design elements, including: understanding of program logic, assumptions, evaluation questions, identification strategy, analysis plan, and alignment with the cost-benefit and beneficiary analysis.

Sector/Technical Operations Lead(s). The Sector/Technical Operations Lead(s) is responsible for defining program logic, assumptions, evaluation questions, and identifying users of evaluation results. Please note, in the Post-Compact period, the DCO Sector Practice Lead will be requested to identify appropriate Sector Lead(s) for reviewing materials Post-Compact.

1 Given that participation on the EMC will require staff time and is crucial to the evaluation quality assurance process, MCC should include participation on the EMC as part of Committee members’ performance plans.

Version: July 2017

Country Relations. One representative from the Country Team Lead (CTL)/Resident Country Mission (RCM) is responsible for assessing policy relevance, understanding country context, and contributing to risk assessments. In the Post-Compact period, the DCO Regional Managing Director will be asked to identify a designated representative.

In addition to above, MCC staff must always reference the Evaluation Review Matrix to confirm who should see what at each milestone. MCC M&E is responsible for documenting review dates, decisions, and feedback using tools like the Evaluation Master Tracker.

As noted below, MCA and other country partners (policymakers, Implementing Entities, implementers, other) should be given the opportunity to review and comment on key evaluation products/deliverables.

Ideally, the MCC EMC will be able to consider any remaining issues and/or concerns raised by the MCA and country partners during its review of the evaluation product/deliverable. To facilitate this, MCAs are encouraged to establish their own, parallel EMCs to ensure appropriate local stakeholder review of the materials.

As per the Evaluation Firm SOW, the Independent Evaluator is responsible for documenting all feedback and their response to that feedback in an Annex of the evaluation deliverable (Evaluation Design Report, Baseline Report, Interim/Final Report).

http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F1%5FEvaluation%20Management%20Process&FolderCTID=0x012000A21A838349721745BD00C8A6005A979C&View=%7b15A669B9-9475-4D1A-B7A4-768995B79014%7d http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F1%5FEvaluation%20Management%20Process&FolderCTID=0x012000A21A838349721745BD00C8A6005A979C&View=%7b15A669B9-9475-4D1A-B7A4-768995B79014%7d

Version: July 2017

3 Evaluation Milestones and Risk Factors The EMP defines several key milestones in the evaluation life cycle when the EMC meets to review and assess the independent evaluator and evaluation products/deliverables. These milestones are critical points in the evaluation life cycle for the EMC members to assess the cost-benefit of continuing, or course correcting, the independent evaluation under review. The milestones2 are:

1. Evaluability Assessment

2. Evaluation SOW

3. Evaluation Design Report

4. Baseline Materials and Data Collection

5. Baseline Report + Data

6. Endline Materials and Data Collection

7. Final Report + Data

At each milestone, it is preferred that the country partners/local stakeholders have an opportunity to review and comment prior to submission to the MCC EMC. This is because there may be remaining issues/challenges between the local stakeholders and evaluator that the MCC EMC should be aware of in its review and feedback. Parallel MCC EMC and local stakeholder review is possible, it is just not preferred as it may result in multiple MCC EMC reviews to address issues raised by local stakeholders.

The MCC M&E lead and/or Independent Evaluator will distribute or present the evaluation product/deliverable to the MCC EMC for review and comment (see standard email templates here). The MCC M&E lead is responsible for collating all comments into one comprehensive summary (see Master Tracking tool here).

Following any review/feedback with the Independent Evaluator, the MCC EMC will provide either Clearance or No Clearance on the evaluation product/deliverable. In the event of disagreement on how to proceed - as documented in the comments by MCC EMC Members and/or any remaining disagreement with local stakeholders - the Chair is the final decision maker and will confer with any relevant management/staff to reach agreement on next steps. The MCC M&E lead is responsible for documenting all related discussions and decisions. The material evidence, such as emails, memos, other documents, should be saved appropriately in S drive or Sharepoint with a clear link/reference.

In addition to the milestones defined here, the MCC M&E lead and M&E MD will also lead frequent Risk Assessments of the independent evaluation portfolio. MCC M&E Management will trigger a Risk Review round 1-2 times per year, however it is the responsibility of the MCC M&E lead to continuously monitor

2 Some evaluations may include one or more midline/interim studies as well. Any midline/interim reports should follow the same review and clearance procedures as endline/final reports.

http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F1%5FEvaluation%20Management%20Process%2FEMC%20Review%20Emails http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F1%5FEvaluation%20Management%20Process&FolderCTID=0x012000A21A838349721745BD00C8A6005A979C&View=%7b15A669B9-9475-4D1A-B7A4-768995B79014%7d http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F1%5FEvaluation%20Management%20Process&FolderCTID=0x012000A21A838349721745BD00C8A6005A979C&View=%7b15A669B9-9475-4D1A-B7A4-768995B79014%7d http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F3%5FEvaluation%20Risk%20Assessment&FolderCTID=0x012000A21A838349721745BD00C8A6005A979C&View=%7b15A669B9-9475-4D1A-B7A4-768995B79014%7d http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F3%5FEvaluation%20Risk%20Assessment&FolderCTID=0x012000A21A838349721745BD00C8A6005A979C&View=%7b15A669B9-9475-4D1A-B7A4-768995B79014%7d

Version: July 2017 and assess risks to raise with Management and the EMC as necessary. These Risk Assessments may identify additional points that require an MCC EMC decision meeting, such as:

• Program re-scoping or re-structuring;

• Significant deviations from original work plan;

• Independent Evaluator contract modifications; and

• Loss of a counterfactual/Treatment group non-compliance/threats to internal validity.

The following sections detail the milestones, as well as the risk factors and objectives the EMC should consider at each milestone.

3.1 Evaluability Assessment

MCC uses an Evaluability Assessment tool to determine whether or not the Project is ready for evaluation planning based on available documentation on the five core areas of any project design:

• Problem diagnostic with sufficient supporting data/evidence;

• Project objectives and theory of change/logic with sufficient supporting data/evidence;

• Project participants clearly defined and justified in terms of geographic scope and eligibility criteria;

• Metrics for measuring results for both accountability and learning clearly defined; and

• Risks and assumptions clearly defined with potential risk mitigation strategies.

The main risks facing the independent evaluation at this stage are lack of understanding of or lack of agreement on the program logic and unclear definition of project participants resulting in:

1. Incorrect definition of the Project components that should be covered by the evaluation (for example, given program logic and common target populations, El Salvador I Formal Technical Education may have been one evaluation instead of three);

2. Incorrect identification of type of evaluation that should be used, and

3. Hiring the Independent Evaluator too early resulting in unnecessary costs.

There are TWO different points for Evaluability Assessment. First, MCC staff (M&E, DCO, EA) will use the Evaluability Assessment tool to guide input into the Investment Memo for the Investment Management Committee (IMC) during Compact Development. In addition to the IM, the tool will be used by M&E staff and management to inform when the Project(s) is ready for evaluation.

Second, the Evaluability Assessment is also the first deliverable of the Independent Evaluator. At that stage, the Independent Evaluator’s Evaluability Assessment is sent to the EMC for review to ensure the Independent Evaluator has the necessary information regarding Project design and implementation to design the evaluation.

http://intranet.mcc.gov/department/DPE/Team/ME/Evaluability/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluability%2FEvaluability%20Assessment%20Tool&FolderCTID=0x01200049142986B4777144ABC49316FB26AC9A&View=%7bFBD426C3-CD46-470A-8183-35D8692314AB%7d https://data.mcc.gov/evaluations/index.php/catalog/67

Version: July 2017

Evaluability Assessments Stage: Produced By: Reviewed By: Primary Function:

Compact Development MCC M&E and Country Team

Investment Management Committee (IMC)

Assess project readiness along five dimensions of Evaluability

Evaluation Design Independent Evaluator Evaluation Management Committee (EMC)

Assess project readiness for evaluation design and Independent Evaluator’s understanding of program logic

3.2 Evaluation SOW + IGCE

Once the project is determined to be “ready” for evaluation planning, MCC M&E will take the lead on developing the Independent Evaluator SOW (see standard template here). The SOW will be developed with the following objectives in mind:

• Define which Project components (Project(s), Activities, Sub-Activities, etc) will be covered by the evaluation;

• Define Evaluation Type (Impact or Performance) if possible given understanding of Project design and implementation rules;

• Identify consumers of the evaluation results to set priority research questions and outcomes;

• Define research team qualification, with particular attention to possible evaluation methodology and sector expertise; and

• Define expected costs for evaluation through the Independent Evaluator Independent Government

Cost Estimate (IGCE) and the data collection budget for MCA (example of IGCEs here).

In addition to the risks listed above, the risks facing the independent evaluation at this stage include a lack of understanding of the program logic resulting in:

1. Incomplete/inaccurate prioritization of research questions and outcomes,

2. Incomplete/inaccurate definition of research team qualifications, and

3. Underestimating the costs of the evaluation.

The EMC should review and clear on the Independent Evaluator SOW, ensuring that the above risks have been addressed and/or have a mitigation strategy, prior to submission to MCC CGM to issue the Task http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F5%5FTemplates%20and%20Examples%2F0%2E%20TORs%2FEvaluation%20Firm%20TORs http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F5%5FTemplates%20and%20Examples%2F0%2E%20TORs%2FEvaluation%20Firm%20TORs%2FIGCE%20EXAMPLES

Version: July 2017

Order. At this stage, the EMC should also determine who will serve on the Technical Evaluation Panel (TEP) to facilitate selection of a high quality independent evaluator.3

3.3 Evaluation Design Report

At this milestone, MCC M&E transfers ownership and leadership of the independent evaluation to the Independent Evaluator. The role of MCC and MCA/country partners is to provide timely feedback and comments to the Independent Evaluator to design the most rigorous, high quality, credible evaluation given project design and implementation rules. The MCC EMC will review the Evaluation Design Report (see template here) with the following objectives in mind:

• Prioritize research questions and outcomes that meet demand from key decision-makers;

• Ensure that the program Objective and all key accountability metrics modeled in the cost-benefit analysis are measured or justification is provided as to why not;

• Apply the most rigorous evaluation methodology feasible given project design and implementation rules;

• Clearly define the analysis plan to ensure consensus on outcomes – their definitions and measurement;

• Clearly define sample population and sampling strategy that aligns with project target populations;

• Clearly define exposure period that maps data collection timelines with project start date timelines;

and

• Update costs as necessary.

The risks facing the independent evaluation at this stage are a lack of understanding of the program logic and program implementation resulting in:

1. Infeasible evaluation methodology (for example, proposing a randomized construction schedule for health centers to create an RCT for the evaluation, when it is not possible to randomize construction schedules given other field realities – see Lesotho Health; or randomly selecting farmers based on selection criteria that does not align with how the farmers are selected for treatment – see Honduras Farmer Training);

2. Lack of alignment between the target population and the evaluation sample population (for example, treatment groups not being treated and/or control groups receiving the program – see Namibia

CBRLM);

3. Lack of agreement or understanding on required exposure period (see several examples of randomized roll-outs that did not allow for sufficient exposure period in the First Five Ag such as Ghana Farmer Training), and

4. Sample sizes that are not large enough to detect impact.

3 A TEP is a group that reviews proposals from bidders and rates each one against the requirements in the SOW in order to identify a preferred bidder. It is helpful to represent a range of perspectives on a TEP. At a minimum, the TEP should include the PM, COR, and Sector Lead. It can also be helpful to include the EA Lead.

http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F5%5FTemplates%20and%20Examples%2FDesign%20Report https://data.mcc.gov/evaluations/index.php/catalog/163 https://data.mcc.gov/evaluations/index.php/catalog/52 https://data.mcc.gov/evaluations/index.php/catalog/52 https://data.mcc.gov/evaluations/index.php/catalog/138 https://data.mcc.gov/evaluations/index.php/catalog/138 https://data.mcc.gov/evaluations/index.php/catalog/77 https://data.mcc.gov/evaluations/index.php/catalog/77

Version: July 2017

Once the Independent Evaluator finalizes the Evaluation Design Report based on MCA, country partners, and MCC EMC feedback (ensuring that the above risks are addressed and documenting the comments and responses to comments) the Independent Evaluator must prepare documentation for an entry in the MCC Evaluation Catalog to post the Evaluation Design Report, and all subsequent evaluation products/deliverables.

EMC Quality Control – Independent Evaluator Following the review and feedback round for the Evaluation Design Report or later deliverables, the MCC EMC can determine the Independent Evaluator is unable to produce a high quality, credible evaluation that meets MCC’s objectives for accountability and learning. In these cases, the MCC M&E lead must work in close coordination with CGM to cancel the contract and begin a new contract process. M&E will also produce a Cancellation Memo that is posted on the Evaluation Catalog for transparency and accountability.

3.4 Baseline Materials and Data Collection

The EMC will provide review and feedback on the Baseline Materials (questionnaires, informed consent, survey protocols) with the following objectives in mind:

• Monitor and assess if project is being implemented according to plan (quantity, quality, and timing of outputs) to ensure evaluation methodology still follows project design and implementation;

• Ensure that baseline data collection begins and ends as close as possible to the start date of the Project;

• Review questionnaires to ensure there is agreement on how outcomes of interest will be measured and who will be surveyed/interviewed to report on these outcomes;

o This is informed by monitoring best practices in questionnaire design across MCC to facilitate standardization in outcome measurement across evaluations in a sector.

• Review data quality plan for training, field work management, and data entry;

• Ensure Independent Evaluator obtains all appropriate and required clearances (both local clearances and ethical review clearances);

• Review informed consent statement(s) to ensure appropriate promises of confidentiality

(see Microdata Guidelines for more details)

In addition to risks raised above, the risks facing the independent evaluation at this stage are:

1. Survey instruments that do not capture the data required to answer the research questions;

2. Excessively long surveys that affect data quality;

3. Missed opportunities for standardization in outcome measurement for MCC meta-analysis in a sector;

and

4. Incomplete/inaccurate definition of survey team quantity and/or quality.

https://data.mcc.gov/evaluations/index.php/catalog https://data.mcc.gov/evaluations/index.php/catalog http://intranet.mcc.gov/department/DPE/Team/ME/Data%20Protection/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FData%20Protection%2F2%2E%20Data%20Documentation%20and%20De%2DIdentification&FolderCTID=0x012000E8DD24BBB41BEA4C912DEC5C47B1B7E6&View=%7b4FB707CB-47A6-47D4-91FF-232589B8CE80%7d

Version: July 2017

Once the Baseline Materials are cleared by the MCC EMC, MCC M&E will post the Questionnaires on the MCC Evaluation Catalog.

3.5 Baseline Report + Data

Within 12 months of the completion of baseline data collection4, the Independent Evaluator will produce a Baseline Report (see template here). The MCC EMC will review the Baseline Report with the following objectives in mind:

• Assess if evaluation is being implemented according to plan (threats to internal validity if an impact evaluation; appropriate sample sizes and response rates);

• Monitor and assess if project is being implemented according to plan (quantity, quality, and timing of outputs);

• Monitor and assess if project risks have materialized and if assumptions are holding;

• Ensure appropriate management of data (see Microdata Guidelines for more details)

In addition to risks raised above, the risks facing the independent evaluation at this stage are:

1. Treatment and control groups that are not appropriately balanced (for example consumption, income, etc.);

2. Significantly different context that can affect expected results (for example, if an outcome of interest in the cost-benefit analysis (CBA) for the ERR is incidence of child diarrhea, but the baseline data shows low to no incidence of diarrhea, this should be raised as a results risk for management and trigger a new CBA, particularly if this outcome is being used to drive evaluation sample sizes – see El Salvador I WASH);

3. High non-response rates for a particular survey instrument (more than typical levels); and

4. Problems with data quality (missing values and outliers).

Once the Baseline Report is cleared by the MCC EMC, MCC M&E will post on the MCC Evaluation Catalog.

Public Release of Baseline Data Only Following the Microdata Management Guidelines, de-identification and preparation of baseline data only should be based on specific demands for the data. Unless there is explicit demand for the public-use baseline data, the Independent Evaluator should prepare the data package with the Data De-Identification Worksheet for documentation purposes, but will NOT conduct the de-identification and DRB review process until the Final Data Package.

4 Some evaluations do not collect baseline data. For these evaluations, the evaluation firm should ensure the Evaluation Design Report validates the evaluation design using available trend data and other sources to justify the decision to use end-line data only.

http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F5%5FTemplates%20and%20Examples%2FBaseline%20Report http://intranet.mcc.gov/department/DPE/Team/ME/Data%20Protection/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FData%20Protection%2F2%2E%20Data%20Documentation%20and%20De%2DIdentification&FolderCTID=0x012000E8DD24BBB41BEA4C912DEC5C47B1B7E6&View=%7b4FB707CB-47A6-47D4-91FF-232589B8CE80%7d https://data.mcc.gov/evaluations/index.php/catalog/107 https://data.mcc.gov/evaluations/index.php/catalog/107 http://intranet.mcc.gov/department/DPE/Team/ME/Data%20Protection/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FData%20Protection%2F2%2E%20Data%20Documentation%20and%20De%2DIdentification&FolderCTID=0x012000E8DD24BBB41BEA4C912DEC5C47B1B7E6&View=%7b4FB707CB-47A6-47D4-91FF-232589B8CE80%7d

Version: July 2017

3.6 Endline Materials and Data Collection

The EMC will provide review and feedback on the Endline Materials (questionnaires, informed consent, survey protocols) with the following objectives in mind:

• Assess if evaluation is being implemented according to plan (threats to internal validity if an impact evaluation; appropriate sample sizes and response rates);

• Monitor and assess if project is being implemented according to plan (quantity, quality, and timing of outputs);

• Ensure that endline data collection begins AFTER the minimum agreed exposure period – i.e. the time required for the target population to be exposed to the program to produce the expected improvements/impact on outcomes.

• Review questionnaires/survey instruments to ensure there is agreement on how outcomes of interest will be measured and who will be surveyed/interviewed to report on these outcomes;

• Review data quality plan for training, field work management, and data entry;

• Ensure Independent Evaluator obtains all appropriate and required clearances (both local clearances and ethical review clearances); and

• Review informed consent statement(s) to ensure appropriate promises of confidentiality

(see Microdata Guidelines for more details)

At this point, the evaluation needs to be monitored for all risks listed.

Once final Endline Questionnaires are cleared by the MCC EMC, MCC M&E will post on the MCC Evaluation Catalog.

3.7 Final Evaluation Report + Data

Within 12 months of the completion of endline data collection, the Independent Evaluator will produce a Final Report. MCC has an expanded five-step review process to manage the public release of Independent Evaluator reports that present results, whether Interim and/or Final.

In addition to all previously listed risks, an additional risk to a high quality, credible evaluation to be considered when reviewing the Final Report:

1. Error or bias in analysis and/or interpretation of results. (Please reference Honduras Farmer Training

Management Response and other documentation related to concerns with evaluation quality).

http://intranet.mcc.gov/department/DPE/Team/ME/Data%20Protection/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FData%20Protection%2F2%2E%20Data%20Documentation%20and%20De%2DIdentification&FolderCTID=0x012000E8DD24BBB41BEA4C912DEC5C47B1B7E6&View=%7b4FB707CB-47A6-47D4-91FF-232589B8CE80%7d http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/1_Evaluation%20Management%20Process/Evaluation%20Report%20Review%20Process.pdf https://data.mcc.gov/evaluations/index.php/catalog/52

Version: July 2017

Step 1 – Country Partner(s) Review. As with all previous key evaluation products/deliverables, the Independent Evaluator is responsible for sharing the Evaluation (Interim and/or Final) Report(s) with the MCA and country partners for review and comment prior to formal submission to the MCC EMC. This may be done virtually or through a local stakeholder workshop. The purpose of this review is to ensure the Independent Evaluator has addressed any technical or factual errors in the report. For country stakeholders, during the Compact period this should include the MCA CEO, MCA Sector Lead and MCA M&E lead, implementers, and Implementing Entities. In Post Compact period, the MCA should designate appropriate representatives to review the report. Special consideration should be made on who these designated representatives are in order to review the report for technical and factual accuracy and they should be identified in the Post-Compact M&E Plan. The MCA and any other necessary local stakeholders will provide written feedback that will be publicly documented in an Annex of the Report. The Independent Evaluator will provide written response to any comments and may elect not to take comments into consideration for the Final Report. Any key stakeholder will then be given the opportunity to provide a public “Statement of Difference or Support” based on the evaluation firm’s report and response to feedback. The submission of the Evaluation Report(s) to MCC will include the following:

• Annex 1: Documentation of the comments/feedback from country stakeholders.

• Annex 2: Any “Statement of Difference or Support” from country stakeholders.

Step 2 – MCC EMC Review. As with all previous key evaluation deliverables, the Independent Evaluator will submit the Final Report to the EMC to review for (i) completeness, (ii) technical rigor, (iii) policy relevance, and (iv) appropriate dissemination activities.. The EMC will ensure that DPE and DCO management is invited to participate in the review. If any factual or technical errors are found in the report, written comments will be submitted to the Independent Evaluator for consideration. In addition, the EMC may elect to request an external peer review of the Evaluation Report, in which case peer reviewer comments will also be provided to the Evaluation Firm (see template for Peer Reviews here). The MCC M&E lead is responsible for collating all comments into one comprehensive summary. The Independent Evaluator will provide written response to any comments and may elect not to take comments into consideration for the Final Report. These comments and responses must be documented by the Independent Evaluator and submitted with the Final Report as part of the deliverables package.

• Annex 3: Documentation of the comments/feedback from MCC stakeholders.

http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F1%5FEvaluation%20Management%20Process%2FPeer%20Reviewer

Version: July 2017

At this stage, the Final Report is complete and will not be altered. During this stage, MCC M&E will also draft the MCC Management Response5 and Summary of Findings6 for the evaluation to accompany the public dissemination of the Evaluation Report. These drafts should be shared with the EMC prior to Step 3.

Step 3 – MCC submits package for final Country Review. Next, MCC M&E will send the following to the MCA, or designated representative: (i) Final Report with Annex 1-3, (ii) MCC Summary of Findings, and (iii) Draft MCC Management Response. The MCA will review and have the option to provide a public Official Country Response7 on the Final Report, if necessary.

Step 4 – Final submission to MCC VP Review. The MCC M&E Managing Director, or designated representative, will send the following to MCC VPs for DCO, DPE, OGC, CPA, as well as the Chief of Staff: (i) Final Report with Annex 1-3, (ii) MCC Summary of Findings, (iii) Draft MCC Management Response, and (iv) Official Country Response (if provided). MCC VPs will finalize the Official MCC Response Statement to be posted with the public Final Report.

Step 5 – Public Dissemination. Following the formal review of the Final Report with Annex 1-3, MCC Summary of Findings, Country Response and MCC Response, the documents will be publicly posted on the MCC Evaluation Catalog.

Public Release of Complete Data Package Following the Microdata Management Guidelines, de-identification and preparation of the Complete Data Package should be developed within 6 months of the completion of the Final Report.

5 The MCC Response is an official MCC management statement that confirms MCC acceptance of the final report and documents any outstanding differences between MCC and the Evaluation Firm as it relates to (i) factual and/or (ii) technical issues.

6 The MCC Summary of Findings places the evaluated program in context of the overall Compact, summarizes the program logic, assumptions, monitoring indicators and results, as well as evaluation questions and key results. The Summary of Findings also summarizes key lessons learned by MCC resulting from the evaluation findings.

7 The Official Country Response is an official host-country statement that either confirms their acceptance of the evaluation findings or documents any outstanding differences between the host country and the Evaluation firm as it relates to (i) factual and/or (ii) technical issues.

http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F4%5FPublic%20Posting%2F3%5FMCC%20Official%20Response http://intranet.mcc.gov/department/DPE/Team/ME/Evaluations/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FEvaluations%2F4%5FPublic%20Posting%2F2%5FMCC%20Summary%20of%20Findings http://data.mcc.gov/evaluations/index.php/catalog http://intranet.mcc.gov/department/DPE/Team/ME/Data%20Protection/Forms/AllItems.aspx?RootFolder=%2Fdepartment%2FDPE%2FTeam%2FME%2FData%20Protection%2F2%2E%20Data%20Documentation%20and%20De%2DIdentification&FolderCTID=0x012000E8DD24BBB41BEA4C912DEC5C47B1B7E6&View=%7b4FB707CB-47A6-47D4-91FF-232589B8CE80%7d

Version: July 2017

4 Summary of Risks over Evaluation Life Cycle

Risk Evaluability Assessment

Evaluation

SOW

Evaluation Design Report

Baseline Materials

+ Data Collection

Baseline Report +

Data

Endline Materials

+ Data Collection

Final Report +

Data

Incorrect definition of the Project components that should be covered by the evaluation; X X X X X X X

Incorrect identification of type of evaluation that should be used X X X X X X X Hiring the Independent Evaluator too early resulting in unnecessary costs X X X X X X X Incomplete/inaccurate prioritization of research questions and outcomes, X X X X X X X Incomplete/inaccurate definition of research team qualifications X X X X X X Underestimating the costs of the evaluation X X X X X X Infeasible evaluation methodology X X X X X Lack of program implementation fidelity X X X X X Lack of alignment between the target population and the evaluation sample population X X X X X

Lack of agreement or understanding on required exposure period X X X X X Sample sizes that are not large enough to detect impact X X X X X Survey instruments that do not capture the data required to answer the research questions X X X X

Missed opportunities for standardization in outcome measurement for MCC meta-analysis in a sector X X X X

Incomplete/inaccurate definition of survey team quantity and/or quality X X X X Treatment and control groups that are not appropriately balanced X X X Significantly different context that can affect expected results X X X High non-response rates for a particular survey instrument X X X Problems with data quality X X X Error or bias in analysis and/or interpretation of results X

Evaluation Firm Compact Project(s) Task Order Contract Authorized Amount Labor $ - Billed Labor Through Reporting Period $ - Contract Authorized Amount Other Direct Costs $ - Billed ODCs Through Reporting Period $ - Contract burn rate #DIV/0!

Progress Reporting Period MM/YY-MM/YY

5. Monitoring of Evaluation Costs (Identify any anticipated changes to the original budget, including cost overruns for originally defined tasks/deliverables and additional tasks/deliverables, Note pending travel or consultant costs that have yet to be billed)

1. Brief Summary of Progress Made (Describe who did what and the progress made on deliverables and any justification for adjustments to the work plan)

2. Risks and Risk Mitigation (Identify risks facing the evaluation design and implementation and describe mitigation measures)

4. Requests to MCC, MCA, or other stakeholders (Describe any required actions by the stakeholders to facilitate progress and risk mitigation for the evaluation, including trips, contract modifications, adjustments to personnel, etc)

3. Any MCC requests for deliverables not included in original work program

ATTACHMENT J.2. Evaluator Progress Report and Work Plan

Total Estimated Cost Total Actual Cost Deliverable % of Task Completed

Estimated Due Date Date Submitted

1 Assess Evaluation Plan Written assessment of program logic, review of evidence, ERR and beneficiary analysis SOW, Trip Report for each country visit Agenda, minutes of each local stakeholder workshop/meeting Draft Evaluation Design Report Local Stakeholder feedback with response MCC feedback with response Final Evaluation Design Report (updated as needed) Draft data collection firm TORs Draft English questionnaires, training manuals SOW, Trip Report for each country visit Summary of pilot test Written review of back-translation Final English questionnaires, training manuals

IRB approval/clearances with informed consent statement(s)

Documentation of local stakeholder and MCC feedback and response SOW, Trip Report for each country visit

Written minutes of meetings with data collection firm(s)

Written summary of quality control checks SOW, Trip Report of each country visit Written minutes of meetings with local stakeholders Draft Baseline Report Documentation of local stakeholder and MCC feedback and response Final raw and analysis files, anonymized following MCC guidelines; STATA do files Final Baseline Report SOW, Trip Report of each country visit Agenda, minutes from local stakeholder workshop PPTs Written risk assessments included in quarterly reports (for impact evaluations, include summary of any risks to internal validity) Written status of implementation in treatment and control groups Draft data collection firm TORs Update/revise English questionnaires, training manuals SOW, Trip Report for each country visit

Summary of pilot test, Written review of back-translation

Final English questionnaires, training manuals

IRB approval/clearances with informed consent statement(s)

Documentation of local stakeholder and MCC feedback and response SOW, Trip Report for each country visit

Written minutes of meetings with data collection firm(s)

Written summary of quality control checks SOW, Trip Report of each country visit Written minutes of meetings with local stakeholders Draft Evaluation Report Local Stakeholder feedback with response; Public Statement of Difference/Support MCC feedback with response Final raw and analysis files, anonymized following MCC guidelines; STATA do files Final Evaluation Report SOW, Trip Report of each country visit Agenda, minutes from local stakeholder workshop PPTs

4 Supervise Baseline data collection

Task

2 Develop Evaluation Design Report

3 Develop Baseline Evaluation Materials

5 Develop Baseline Report

6 Disseminate Baseline Results

7 Monitor program implementation

11 Disseminate Final Results

8 Revise Interim/Final Evaluation Materials

9 Supervise Interim/Final Data collection

10 Develop Interim/Final Report

1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 Assess Evaluation Plan

2 Develop Evaluation Design Report

3 Develop Baseline Evaluation Materials

4 Supervise Baseline data collection

5 Develop Baseline Report

6 Disseminate Baseline Results

7 Monitor program implementation

8 Revise Interim/Final Evaluation Materials

9 Supervise Interim/Final Data collection

10 Develop Interim/Final Report

11 Disseminate Final Results

Year 5 Task

Year 1 Year 2 Year 3 Year 4

SCOPE OF WORK

Name, Organization

COUNTRY

Project/Activity

DATE

1. Travel Dates and Objective(s)

2. Team Roles/Responsibilities

3. Proposed Activities

4. Proposed Meetings

5. Expected Deliverables

6. Requested Background Documentation (from MCA or other stakeholders)

7. Itinerary

Name(s) Carrier and #

Mode (Air, Car, Train) Class (Business/Coach) Departure

Location Departure Date/Time Arrival Location Arrival

Date/Time Hotel Name

Justification for Premium Class (if applicable):

8. Estimated Budget (airfare, hotel, per diem)1

Name(s) Total Days Airfare ($) Hotel ($) Per Diem ($) Misc ($) TOTAL

ESTIMATE ($)

1 All travel must be made in accordance with Federal Travel Regulations (FTR) available online here. In accordance with the Fly America Act, contractors must use a U.S. flag air carrier service for all travel funded by the government beginning or ending in the U.S. unless a specific exemption to the “Fly America” rule applies. Flights on U.S. air carrier pairings with foreign carriers (i.e., code share flights) are regarded as meeting Fly America requirements if the ticket is issued on the American carrier and there is an American carrier flight number.

ATTACHMENT J.3. Evaluation Services Travel Scope of Work http://www.gsa.gov/portal/content/104790

TOTAL

9. Country clearance – MCC staff and contractors are required to receive country clearance through the embassy prior to travel. To complete the clearance form, travelers should provide their full itinerary and name of hotels (required in table 7), passport number, passport country of issue, and country of birth. These details can be provided separately to the M&E lead.

Project Evaluability Assessment (PEA)

Background As a results-based institution, MCC is dedicated to using rigorous methods to identify the core constraints to growth and design and implement evidence-based Projects in every Compact and Threshold country to address those core constraints.

For this reason, the concept of evaluability is central to MCC’s results agenda, as it is impossible to determine the results achieved by MCC projects unless they are designed up-front to be evaluable downstream. MCC defines evaluability as the ability of an intervention to demonstrate in measurable terms the results it intends to deliver.