Annexs_1_3_4_5_6_7_8_10_11_12_14_15_16.pdf

ANNEX 3 ‐ RFQ – MCC‐17‐RFQ‐0010 Liberia Energy Project

Project Evaluability Assessment (PEA)

Background As a results-based institution, MCC is dedicated to using rigorous methods to identify the core constraints to growth and design and implement evidence-based Projects in every Compact and Threshold country to address those core constraints.

For this reason, the concept of evaluability is central to MCC’s results agenda, as it is impossible to determine the results achieved by MCC projects unless they are designed up-front to be evaluable downstream. MCC defines evaluability as the ability of an intervention to demonstrate in measurable terms the results it intends to deliver. An evaluable intervention uses data to identify and verify the problem(s) it intends to address and underlying causes, has an evidence-based design to address the problem(s) identified, clearly identifies assumptions and risks associated with the intervention to address the problem(s), including identifying means for verifying and mitigating risks, and has clear and time-bound metrics for the results of the intervention in terms of outputs and outcomes.

Objective The objective of an evaluability assessment is to use specific, transparent standards and best practices for assessing the following five dimensions of a project: (i) Problem Diagnostic, (ii) Project Objectives and Logic, (ii) Risks and Assumptions, (iii) Project Participants/Beneficiaries, and (v) Accountability and Learning Metrics.

Evaluability in Practice A tool has been developed to help guide the iterative process of evaluability assessment.1 The Project Evaluability Assessment tool draws on and expands on the standards and best practices identified in the MCC Due Diligence Book in order to assess the five dimensions of a project through Compact development and into Compact implementation. The tool is intended to help guide teams to answer five big questions:

Is the problem clearly defined and is there sufficient evidence to support the problem diagnostic?

Are the project objectives and theory of change/logic clearly defined?

Are the risks and assumptions clearly defined with potential risk mitigation strategies?

Are project participants clearly defined and justified in terms of geographic scope and eligibility criteria?

Are the metrics for measuring results for both accountability and learning clearly defined?

In each of the five dimensions, the evaluability assessment tool lists a series of questions based on standards and best practices. While a Project may not meet all of the standards and best practices, and it is possible that the list of questions does not capture all potential factors that should be flagged, the purpose is to highlight the strengths and weaknesses of each intervention so that they can be addressed prior to Compact signature, in the development of the supplemental agreements, and during implementation as appropriate.

The tool may be applied throughout Compact development and implementation, and it is important to note that the assessment is intended to clarify and improve project documentation. For this reason, the assessment is only conducted on specific project documents as they are developed: (i) Concept Notes, (ii) Concept Papers, (iii) Investment Memo, (iv) Other Project and Evaluation Design Document (s).

1 The evaluability assessment tool was developed through a yearlong pilot from 2012-2013, tested by the MCC M&E Team on the following 7 Compact and Threshold programs: Threshold (Honduras); Implementation (Cape Verde II and Lesotho); Compact Development (El Salvador II, Georgia II, Ghana II, Benin II).

The current proposed process for assessment of project evaluability is intended to (i) maximize integration of the evaluability assessment within the Country Team, and (ii) ensure quasi-independent assessment for quality assurance. The proposed process is:

1. Concept Note: Country team members submit Project(s) Evaluability Memo to Country Team Lead with a cc to respective MDs.

2. Concept Paper: Country team members submit Project(s) Evaluability Memo to Country Team Lead with a cc to respective MDs.

3. Investment Memo: Country M&E lead works with Country Team to produce Investment Memo, reflective of current state of Project(s) Evaluability. Once submitted, an independent peer review of the IM, using the evaluability assessment tool, will be conducted. The composition and timing of this peer review is TBD.

At each stage, the intention is to provide feedback to the Country Team and Management on the current status of the Project(s) which is one of several inputs into project and evaluation selection, design, and implementation decisions.

Project2 – Evaluability Assessment Assessment Categories Strengths, Weaknesses, and Next Steps

Dimension 1: Is the problem clearly defined and is there sufficient evidence to support the problem diagnostic?

Standards and best practices

1. Is there quantitative evidence regarding constraints to and sources of economic growth?

2. Is the problem(s) clearly defined and understood with sufficient evidence/quantitative (baseline) data available to support claims?

3. Is there evidence to support root causes identified?

4. Will all root causes be addressed by the proposed intervention or other, complementary intervention(s)?

5. Is there a public good rationale and/or market failure that necessitates government intervention and funding? This is linked to sustainability – how will recurrent costs be covered in the future, how will private sector investment be triggered, etc?

6. Is the institutional context understood, the political economy understood, and does the team clearly understand how the proposed intervention(s) link to other initiatives by the gov’t and/or other partners?

7. Is there a clear understanding on how different social and cultural dynamics (gender, poverty, race, ethnicity, etc.)

may be influenced by or influence the problem identified?

Strengths:

Weaknesses:

Next steps:

Dimension 2: Are the project objectives and theory of change/logic clearly defined?

Standards and best practices

1. Is the objective of the Project clearly stated with a link from the problem diagnostic?

2. Is the project logic and economic model clear, plausible and based on existing evidence and literature? If there is limited evidence, is there commitment to generating evidence via an impact evaluation (linked to Section 5)?

3. Is there a clear logic that links different projects within the Compact program that is then linked to accelerating economic growth?

4. Is it clear which component(s) of the problem diagnostic will be addressed by the proposed intervention, which will be addressed by complementary activities and which will remain risks for the MCC intervention to achieve proposed results?

5. Are the inputs, outputs, outcomes clearly defined and linked to the economic analysis (ERRs)?

6. Is the timeline for expected results clear and based on evidence?

2 It is anticipated that an evaluability assessment will be conducted for each Project; however this may vary depending on Compact composition. For example, the country team may prefer to conduct the evaluability assessment for individual Activities rather than Projects.

7. Is it clear whether or not benefits are expected to be sustained beyond the life of the compact?

Dimension 3: Are the risks and assumptions clearly defined with potential risk mitigation strategies?

Standards and best practices

1. Are the risks to achieving expected results clear, with clearly defined risk mitigation strategies?

2. Does the ERR reflect these assumptions and risks? Has sensitivity analysis been used to select key risks and assumptions?

3. Is it clear how risks will be monitored?

4. Is it clear how design and implementation may be altered as information on new risks/realization of risks occurs?

5. Does the project team make a critical assessment of the degree to which there may be blind spots or unknown unknowns in a project of this nature (e.g. how foreseeable are the potential risks that may arise in new sectors)?

Dimension 4: Are project participants clearly defined and justified in terms of geographic scope and eligibility criteria?

Standards and best practices

1. Is the selection criteria for project participants clearly defined and based on the problem and evidence in the program logic?

2. Is program participants’ selection based on credible, quantifiable selection criteria?

3. Are specific demographics (age, gender, poverty status) defined where necessary?

4. Are the geographic location(s) for the Project defined and based on the problem listed above and evidence in the program logic?

5. Will the Project design and implementation plan vary by different sub-groups and/or geographic locations based on the problem listed above and evidence in the program logic?

6. Can the selection be replicated for the purposes of an impact evaluation (linked with Section 5)?

Dimension 5: Are the metrics for measuring results for both accountability and learning clearly defined?

Standards and best practices

Are there clearly defined indicators and data sources identified for monitoring project implementation?

1. Are there clearly defined indicators for measuring expected performance (processes, outputs)?

2. Are the indicators linked to the ERR?

3. Is it clear which indicators will be disaggregated by gender, age, income as appropriate?

4. Is there a clear understanding of the time frame for expected results of each indicator (if varies)?

5. Is there sufficient information to set appropriate and feasible baseline and annual/quarterly targets?

6. Are there sufficient human and financial resources in the MCA and IEs to conduct the necessary data collection/reporting during the life of the intervention?

Are data collection costs known and budgeted for?

7. Is it clear who will use the data and for what purpose(s)?

Next steps:

Are there clearly defined indicators and data sources identified for monitoring project results?

1. Are there clearly defined indicators for measuring expected performance (processes, outputs)?

2. Are the indicators linked to the ERR?

3. Is it clear which indicators will be disaggregated by gender, age, income as appropriate?

4. Is there a clear understanding of the time frame for expected results of each indicator (if varies)?

5. Is there sufficient information to set appropriate and feasible baseline and annual/quarterly targets?

6. Are there sufficient human and financial resources in the

MCA and IEs to conduct necessary data collection/reporting during the life of the intervention?

Are data collection costs known and budgeted for?

7. Is it clear who will use the data and for what purpose(s)?

Is the evaluation clearly defined for maximizing learning and accountability?

1. Is there commitment by all key stakeholders to implement the independent evaluation?

2. Are evaluation questions and outcomes clearly defined and prioritized?

3. Is it clear which outcomes will be disaggregated by gender, age, income as appropriate?

4. Is it clear who will use the evaluation results and for what purpose(s)?

5. Is the evaluation methodology the most rigorous and feasible possible?

6. Is it clear how an evaluation (performance or impact) will contribute to the evidence base in the sector?

7. Are there interim/continuous evaluation results which could help inform decisions during the compact life? If so, is such an evaluation built into the evaluation plan?

8. Do the potential benefits and learning from an evaluation of the program outweigh the costs?

9. Are there sufficient human and financial resources in the MCC, MCA and IEs to conduct necessary data collection/reporting during the life of the evaluation? Are data collection costs known and budgeted for?

Annex 4 – RFQ MCC‐17‐RFQ‐0010 Liberia Energy Project

SCOPE OF WORK

Name, Organization

COUNTRY

Project/Activity

DATE

1. Travel Dates and Objective(s)

2. Team Roles/Responsibilities

3. Proposed Activities

4. Proposed Meetings

5. Expected Deliverables

6. Requested Background Documentation (from MCA or other stakeholders)

7. Itinerary

Name(s) Carrier and #

Mode (Air, Car, Train)

Class (Business/Coach) Departure Location

Departure Date/Time

Arrival Location Arrival

Date/Time

Hotel Name

Justification for Premium Class (if applicable):

8. Estimated Budget (airfare, hotel, per diem)1

Name(s) Total Days

Airfare ($) Hotel ($) Per Diem ($) Misc ($)

TOTAL

ESTIMATE ($)

1 All travel must be made in accordance with Federal Travel Regulations (FTR) available online here. In accordance with the Fly America Act, contractors must use a U.S. flag air carrier service for all travel funded by the government beginning or ending in the U.S. unless a specific exemption to the “Fly America” rule applies. Flights on U.S. air carrier pairings with foreign carriers (i.e., code share flights) are regarded as meeting Fly America requirements if the ticket is issued on the American carrier and there is an American carrier flight number.

Annex 4 – RFQ MCC‐17‐RFQ‐0010 Liberia Energy Project

TOTAL

9. Country clearance – MCC staff and contractors are required to receive country clearance through the embassy prior to travel. To complete the clearance form, travelers should provide their full itinerary and name of hotels (required in table 7), passport number, passport country of issue, and country of birth. These details can be provided separately to the M&E lead.

Annex 5 ‐ RFQ MCC‐17‐RFQ‐0010 Liberia Energy Project

MCC Independent Evaluation Design Report Outline Template

i. List of Acronyms

ii. Table of Contents

LIST OF ACRONYMS

TABLE OF CONTENTS

1. INTRODUCTION

1.1. Background

1.1.1. Country context

1.1.2. Objectives of this report

2. OVERVIEW OF THE COMPACT AND THE INTERVENTIONS EVALUATED

2.1. Overview of the project and implementation plan

2.1.1. Program participants (who is being targeted and why)

2.1.2. Geographic coverage (where it is being implemented and why)

2.2. Key program indicators and how they lead to expected outcomes. (Here we will summarize the program logic rather than discuss its merits since the program logic assessment is coming in the assessment report.)

2.3. Link to ERR and Beneficiary Analysis

3. LITERATURE REVIEW

3.1. Summary of the existing evidence

3.2. Gaps in literature

4. EVALUATION DESIGN

4.1. Policy relevance of the evaluation

4.2. Overview of an impact evaluation that includes a quantitative impact analysis and qualitative implementation analysis

4.3. Evaluation questions

4.4. Key Outcomes

4.5. Methodology/ Impact evaluation design

4.6. Study Sample (to include required sample size, sample frame, process of selection of treatment and control groups, stratification, design omission in the sample, level of representation, strategy for absent respondents and dropouts)

4.7. Analysis Plan

4.8. Timeframe of exposure. Will include justification for survey timing relating back to program logic and when impacts are anticipated to take effect.

4.9. Limitations and Challenges. Will include strategies to minimize problems with randomization compliance and strategies to deal with deviation from assignment status.

5. DATA SOURCES AND OUTCOME DEFINITIONS

5.1. Data collection plans. (We are thinking it would be good to discuss survey rounds and when different types of data collection would take place.)

5.2. Proposal to measure implementation fidelity

5.3. Data needs

5.3.1.1. Data sources for quantitative

5.3.1.2. Data sources for qualitative

6. ADMINSTRATIVE

6.1. Summary of IRB requirements and clearances

6.2. Preparing data files for access, privacy and documentation

6.3. Dissemination Plan

6.4. Evaluation Team Roles and Responsibilities

6.5. evaluation timeline

6.6. reporting schedule

7. REFERENCES

8. ANNEXES

8.1. Stakeholder comments and Evaluator Responses

8.2. Evaluation Budget. Given sensitivities around future procurements, the budget should not be a public document. However, the evaluator and MCC M&E should have a clearly defined budget for the evaluation, including planned evaluator and data collection costs which should be tracked over time with actual evaluator and data collection costs.

Annex 6 - RFQ MCC-17-RFQ-0010 - MCC Independent Evaluations Evaluation Management and Review Process

DRAFT: 1/12/2017

Contents 1 Introduction

1.1 What are the different roles and responsibilities?

1.2 What are the objectives?

1.3 How will we meet these objectives?

2 Evaluation Management Committee

3 Evaluation Management Process

3.1 Evaluability Assessment

3.2 Evaluation Plan

3.2.1 Contract Evaluation Firm(s)

3.3 Evaluation Design Report(s)

3.3.1 Survey Firm TOR and Evaluation Materials

3.3.2 Baseline Report

3.4 Final (and Interim) Evaluation Report

1 Introduction The Millennium Challenge Corporation (MCC) invests in independent evaluations to measure results of MCC-financed and Millennium Challenge Account (MCA)-executed interventions in order to hold MCC and MCAs accountable for results and to produce continuous learning. The independent evaluations are conducted by third-party independent experts to ensure that the evaluations represent an unbiased assessment of the interventions. Ideally, evaluations should be designed to answer three fundamental questions related to accountability and transparency:

Was MCC’s investment implemented according to plan?

Were there any changes in key outcomes, such as income, for program participants that are attributable to MCC’s investment?

Why did or didn’t the planned investments lead to changes in key outcomes, such as income?

However, it is widely recognized that these three separate objectives cannot always be met through one evaluation, and there are necessary trade-offs depending on priorities. Prioritizing the objectives of each evaluation is something that MCC, and many other organizations, continue to struggle with when deciding how to guide independent evaluators during evaluation design. MCC is currently planning or actively managing over 150 independent evaluations, and this issue is expected to be raised for each evaluation continuously throughout the evaluation review process.

This Guidance Note outlines how to strengthen MCC’s independent evaluations by summarizing the (i) roles and responsibilities of the independent evaluators, MCC, and MCAs, (ii) current challenges MCC is facing to ensure high quality, independent evaluations, (iii) objectives of the formal MCC review process to address these challenges, and (iv) its structure and the critical milestones that will be established to review evaluation activities and deliverables.

1.1 What are the different roles and responsibilities?

Each institution, the MCC, MCA, and independent evaluator, as well as other key stakeholders, play critical roles in designing, implementing, and disseminating the independent evaluations.

Their roles are as follows:

Independent evaluator. The independent evaluator is responsible for the overall design, implementation, and dissemination of the evaluation, including the following specific responsibilities:

o Develop a rigorous evaluation design given rules of implementation and feasibility of options. Using the program logic as the foundation, independent evaluators must assess the program design and implementation to develop the most rigorous evaluation design feasible, whether it is a performance or impact evaluation, and identify the most appropriate evaluation methodology feasible given the context. The independent evaluator is responsible for developing an evaluation design report, in consultation with

MCC and MCA, which summarizes the evaluation design, sampling, analysis plan, and other critical design elements of the evaluation.

o Support MCC and MCA to build buy-in and ownership of evaluation. The independent evaluator will meet with MCC and local stakeholders to develop the evaluation design based on program design and implementation. When possible, the independent evaluator will work with MCC and MCA to advise on a program implementation roll-out plan that enables a rigorous evaluation design. In addition, the independent evaluator will continuously present the evaluation objectives, materials, and interim and final results for MCC and local stakeholders to maintain commitment to the evaluation.

o Develop evaluation materials that are held to international standards. Using the program logic as a foundation, the independent evaluators define how key outcomes will be measured and develop the survey instruments or other data collection tools in order to answer critical evaluation questions.

o Ensure appropriate review of evaluation materials and research protocols.

Independent evaluators are responsible for ensuring all evaluation materials, including the evaluation design, survey instruments, sampling strategy, data collection and entry protocols are reviewed by appropriate local stakeholders prior to submission to MCC.

The goal of this review is to ensure that local stakeholders are able to comment on the feasibility of the proposed methodology, as well as the technical and factual accuracy of the materials.

o Ensure Institutional Review Board clearance. The Independent Evaluator is responsible for submitting evaluation materials to an institutional review board (IRB).

IRB review and approval is required before any data collection commences. The independent evaluators will document all approvals and informed consent procedures throughout the evaluation time period.

o Manage the data collection firms. This responsibility may differ in the Compact period compared to the Post-Compact period. However, as appropriate, the evaluator may develop Terms of Reference for the data collection firm, with technical support from MCC and the MCA. The independent evaluator may be responsible for managing the contracting process, including Request for Proposals, Proposal Review, selection of the data collection firm, development of a work plan, and supervision of the data collection firm in line with the work plan and agreed deliverables.

o Supervise data collection. Whether the MCA, MCC or independent evaluator contracts the data collection firm, the independent evaluator is responsible for quality control over data collection activities, including interviewer training, pre-testing, and implementing data quality review checks during data collection.

o Lead data cleaning, analysis, interpretation of results. The independent evaluator is responsible for leading any necessary data cleaning and consultation with the data collection firm to produce final analysis files. The independent evaluator is responsible for leading all analysis in line with the agreed analysis plan developed with MCC, and consulting with MCC should the analysis plan need to be revised or adapted.

o Produce evaluation reports. The independent evaluator is responsible for sharing initial evaluation reports with local stakeholders and MCC for review and feedback. However, given the independent evaluator is the author of the evaluation reports, local stakeholder and MCC feedback may or may not be taken into consideration for the final report. The independent evaluator is responsible for documenting all feedback and evaluator responses.

o Lead public dissemination efforts. The independent evaluator is responsible for leading public dissemination efforts facilitated by MCA and MCC (such as local workshops and conferences), but also additional conferences or other opportunities to publicly disseminate the results of the evaluation. The evaluator will advise MCC on any public dissemination opportunities and collaborate as appropriate.

MCC. MCC is responsible for oversight of the independent evaluator and quality control of evaluation activities, including the following specific responsibilities:

o Set the evaluation priorities and evaluation questions. Using the program logic and priorities of key users of the evaluation results as the foundation, MCC will develop the evaluation priorities and evaluation questions based on existing evidence, the evidence gap, and the learning priorities for the sector and agency. These will be developed in consultation with the MCA and local stakeholders to maximize use of the evaluation results and their contribution to future decision making.

o Build buy-in and ownership of the evaluation. MCC will work with the independent evaluator and MCA to ensure the evaluation design follows the rules of program design and implementation. When possible, MCC will work with the independent evaluator and MCA to develop a program implementation roll-out plan that enables a rigorous evaluation design. When it is not possible to develop a coordinated roll-out plan, the evaluator will ensure the evaluation design is based on feasible methodologies given program design and implementation. MCC will work with MCA to ensure commitment to the roll-out plan by local stakeholders, and if is not possible to adhere to the roll-out plan, will inform the independent evaluator in order to adapt the evaluation design.

o Determine the evaluation budgets. MCC will develop an evaluation budget, including independent evaluators, expected data collection requirements, and any other activities related to the evaluation.

o Manage the independent evaluators. MCC will develop Terms of Reference for the independent evaluators and manage the contracting process, including Request for Proposals, Proposal Review, selection of the independent evaluator, development of a work plan, and supervision of the independent evaluator in line with the work plan and agreed deliverables.

o Quality review of evaluation materials. Using the program logic and evaluation priorities as a foundation, MCC is responsible for reviewing evaluation materials, including the design report and survey materials, for feasibility, technical and factual accuracy, and adherence with international standards (when applicable).

o Quality review of evaluation reports. MCC is responsible for reviewing independent evaluators’ reports to ensure technical and factual accuracy. While MCC will provide feedback, the independent evaluator is ultimately the author of the evaluation reports and may or may not take into consideration MCC’s comments. MCC therefore develops an Official Statement for each evaluation report to confirm the technical and factual accuracy of the report, or clarify any differences on technical or factual issues between MCC and the independent evaluator.

o Lead initial public dissemination efforts. MCC is responsible for leading initial public dissemination of results, in particular ensuring a feedback loop internally in order to inform future decision making on similar program investment, design, and implementation. This requires ensuring the evaluation results are disseminated on the MCC website, and results are disseminated to appropriate stakeholders in a timely fashion.

MCA. MCA is responsible for building local ownership and commitment to the evaluation, oversight of the data collection firm, and quality control of evaluation activities, including the following specific responsibilities:

o Build buy-in and ownership of the evaluation. MCA will work with the independent evaluator and MCC to ensure the evaluation design follows the rules of program design and implementation. When possible, MCA will work with the independent evaluator and MCC to develop a program implementation roll-out plan that enables a rigorous evaluation design. MCA will work with MCC to ensure commitment to the roll-out plan by local stakeholders, and if it is not possible to adhere to the roll-out plan, will inform MCC in order to request the independent evaluator adapt the evaluation design.

o Manage the data collection firms. This responsibility may differ in the Compact period compared to the Post-Compact period. However, as appropriate, the evaluator may develop Terms of Reference for the data collection firm, with technical support from MCC and the MCA. The independent evaluator may be responsible for managing the contracting process, including Request for Proposals, Proposal Review, selection of the data collection firm, development of a work plan, and supervision of the data collection firm in line with the work plan and agreed deliverables.

o Quality review of evaluation materials. Using the program logic as a foundation, MCA is responsible for reviewing evaluation materials, including the design report and survey materials, for feasibility, technical and factual accuracy, and adherence with international standards (when applicable).

o Quality review of evaluation reports. MCA, or a designated representative, is responsible for reviewing independent evaluators’ reports to ensure technical and factual accuracy. While MCA will provide feedback, the independent evaluator is ultimately the author of the evaluation reports and may or may not take into consideration MCA’s comments. MCA therefore develops an Official Statement for each evaluation report to confirm the technical and factual accuracy of the report, or clarify any differences on technical or factual issues between MCA and the independent evaluator.

o Facilitate public dissemination efforts. MCA, or a designated representative, is responsible for facilitating public dissemination of results, in particular ensuring a feedback loop in-country in order to inform future decision making on similar program investment, design, and implementation.

1.2 What are the objectives?

The objectives of the formal MCC review process are to convert core lessons learned from the first five impact evaluations into action:

Ensure early integration of evaluation with program design and implementation. This requires close collaboration between DPE and DCO as early as possible in the process, and then continuously throughout Compact development, implementation, and Post Compact.

Ensure clearly defined program logic. This requires clearly defining the inputs, outputs, outcomes, and goals of each intervention in a way that demonstrates how the intervention will lead to expected results. For this, there must be common understanding across stakeholders on the program logic and the assumptions underlying the program logic.

Ensure evaluations are structured for learning and work toward establishing a feedback loop. This requires understanding the assumptions underlying the program logic for how an intervention will lead to improved outcomes, and ultimately an increase in beneficiary household income, and what questions need to be answered in order to inform future decision-making. This also requires identifying the consumers of the evaluation results and ensuring a feedback loop is established so that evaluation results can inform future decision-making.

Ensure appropriate quality control mechanisms are in place. This requires ensuring the appropriate stakeholders are reviewing key evaluation actions and deliverables, and assessing risk to the interventions, and their evaluations, in a timely manner.

Establish transparent and clear decision making-process. Although the guidelines proposed here may evolve over time as we learn what works and what doesn’t, this is the first step to establishing a formal internal review process to ensure that evaluations are designed and implemented to produce high quality, rigorous evidence that can be used by MCC and host country governments for future decision-making.

1.3 How will we meet these objectives?

This Guidance Note outlines the formal internal review process which defines who should be involved in decisions regarding evaluation activities and deliverables, and when.

Who needs to be involved? A formal Evaluation Management Committee will be established for each Compact, chaired by the Monitoring and Evaluation Managing Director, or a designated representative, and consisting of the appropriate representation from DPE and DCO for Compact activities, including senior management (MDs, DVPs, VPs) when appropriate.

When do they need to be involved? Critical milestones have been established for the Evaluation Management Committee to meet and review proposed evaluation activities, as well as core evaluation deliverables. The critical milestones are: (i) Assess Evaluability, (ii) Develop the Evaluation Plan, (iii) Develop the Evaluation Design and subsequent evaluation deliverables (survey instruments, baseline report(s)), and (iv) Develop the Interim/Final Evaluation Report.

2 Evaluation Management Committee In order to clearly define who needs to be involved in critical decisions related to independent evaluations, an Evaluation Management Committee (EMC) will be established for each Compact comprised of one Chair and four-six Committee members.

Chair:

Monitoring and Evaluation Managing Director, or designated representative

Members1:

Monitoring and Evaluation Lead. The Monitoring and Evaluation (M&E) lead who is responsible for overall leadership on the independent evaluation activities and ensuring common understanding of the program logic(s), evaluation questions, contracting Evaluation Firm(s), and quality control for evaluation materials and reports. It is to the discretion of the committee if an additional M&E lead would provide a peer review as well.

Evaluation Support. (1) technical support from evaluation team dependent on sector.

a. Jack Molyneaux: Agriculture & Irrigation; Transportation; Financial, Investment, and

Trade; Capacity Building and Institutional Development

b. Jennifer Sturdy: Health; Education; Energy; Water and Sanitation

c. Jennifer Witriol: Land

Economic Analysis Lead. The Economic Analysis (EA) lead responsible for assessing the technical design elements, including: program logic, assumptions, evaluation questions, identification strategy, analysis plan, and alignment with the ERR(s) and beneficiary analysis.

It is to the discretion of the committee if an additional EA lead would provide a peer review as well.

Sector Lead(s). The Sector Lead(s) responsible for defining program logic, assumptions, evaluation questions, and identifying users of evaluation results. Please note, in the Post- Compact period, the Sector Practice Lead will be requested to identify appropriate Sector Lead(s) for reviewing materials Post-Compact if necessary.

Country Relations. One representative from the Country Team Lead (CTL)/Resident Country Mission (RCM) to assess policy relevance and risk assessment for implementation challenges.

Please note, in the Post-Compact period, the DCO Managing Director will be asked to identify a designated representative, if possible.

Please note, DCO and DPE management should be brought into EMC reviews at two specific milestones: (i) Review of the Evaluation Plan, and (ii) Review of Interim/Final Reports.

1 Given that participation on the Evaluation Management Committee will require staff time and is crucial to the evaluation quality assurance process, MCC should include participation on the EMC as part of Committee members’ performance plans.

It is anticipated that most EMCs will be established at the Compact-level. The M&E lead will collaborate with the Country Team to determine what is appropriate and to identify the appropriate EMC members. Sector Leads are only expected to participate in reviews of the evaluations in their sector.

The EMC should be established as early in Compact development as possible, and members should expect to participate in the EMC for the duration of the evaluation period. In the event that an EMC member must leave the EMC, the Chair, in collaboration with DPE and DCO management, must identify the appropriate replacement for the EMC member.

External stakeholders, such as the host-country MCA, government, Implementing Entities, and MCA contractors will be included as necessary in each stage of the evaluation process as described in the sections below. In addition, MCAs are encouraged to establish their own, parallel EMC to facilitate appropriate local stakeholder review of the materials.

3 Evaluation Management Process The Evaluation Management Process consists of several key milestones:

1. Assess Evaluability

2. Develop the Evaluation Plan

2.1. Contract the Evaluation Firm(s)

3. Develop the Evaluation Design Report(s)

3.1. Develop the Evaluation Materials (survey instruments, training manuals, etc)

3.2. Develop the Evaluation Baseline Report(s)

4. Develop the Interim and Final Evaluation Report(s)

Subsequent sections provide more detail on the objectives and requirements of each milestone in the Evaluation Management Process. These milestones are intended to serve as critical points in time for the respective EMC members to assess the cost-benefit of continuing, or course correcting, the evaluation activity under review.

At each milestone, the EMC members will be asked to use the Evaluation Risk and Status Assessment Checklist (Annex X) to assess (i) the evaluation activity/deliverable under review, and

(ii) the current risks and cost-benefit of proceeding with the evaluation. For this reason, the MCC M&E lead may propose to convene the EMC (either virtually or in face-to-face meeting) at other points in time in addition to these milestones to discuss any significant changes to the agreed evaluation activities that pose a risk to the evaluation activities and deliverables. Additional actions that may trigger an EMC review and/or meeting to assess the cost-benefit of continued evaluation activities include:

Project re-scoping or re-structuring

Significant deviations from original work plan

Evaluation Firm contract modifications

Loss of a counterfactual

3.1 Evaluability Assessment

The MCC M&E and Sector Leads will initiate discussions around program evaluability at the Concept Note and Concept Paper stages2. Evaluability is defined as the ability of an intervention to demonstrate in measurable terms the results it intends to deliver.

Incorporating Evaluability Assessment into Compact Development and Implementation is still under discussion.

Template 1: Evaluability Assessment Tool

3.2 Evaluation Plan

The MCC M&E lead is responsible for developing the Compact Evaluation Plan, a section in the overall Compact M&E Plan. The Evaluation Plan should identify priority learning opportunities, propose impact or performance evaluations for each Compact intervention, and prepare estimated budgets (MCC and MCA) for each evaluation activity with a corresponding procurement plan for consultants/IDIQ firms and work plan. The Evaluation Plan should also clearly indicate if any intervention will not be evaluated, and the justification for not evaluating.

The following must be developed prior to developing the Evaluation Plan:

Diagnostic and problem identification

Clearly defined program logic

Literature review of evidence on proposed intervention(s)

Initial economic rate of return models and beneficiary analysis (this is optional as it may be developed in parallel with the evaluation plan.)

For the Evaluation Plan, the M&E lead will indicate whether or not the Project/Activity/Sub- Activity will be covered by (i) an impact evaluation, (ii) a performance evaluation, (iii) self-evaluation only, (iv) no evaluation, or (v) to be determined.

Impact Evaluation. According to MCC M&E Policy, a study that measures the changes in income and/or other aspects of well-being that are attributable to a defined intervention. Impact

2 DPE currently discussing how best to accomplish this, potentially through check-lists for Concept Note and Concept Paper review in order to provide feedback on evaluability early in the process.

evaluations require a credible and rigorously defined counterfactual, which estimates what would have happened to the beneficiaries absent the project. Estimated impacts, when contrasted with total related costs, provide an assessment of the intervention’s cost-effectiveness. In should be noted, both Impact and Performance Evaluations are expected to document any difference between what was planned and what actually happened as well as understanding why, if any, difference occurred between the two.

Performance Evaluation. According to MCC M&E Policy, A study that seeks to answer descriptive questions, such as: what were the objectives of a particular project or program, what the project or program has achieved; how it has been implemented; how it is pEMCeived and valued; whether expected results are occurring and are sustainable; and other questions that are pertinent to program design, management and operational decision making. MCC’s performance evaluations also address questions of program impact and cost-effectiveness. In should be noted, both Impact and Performance Evaluations are expected to document any difference between what was planned and what actually happened as well as understanding why, if any, difference occurred between the two.

Self-Evaluation only. Upon completion of each Compact program, both MCC and MCA may comprehensively assess three fundamental questions: (i) Did the MCA program meet Compact objectives; (ii) Why did the Compact program meet or not meet these objectives; and (iii) What lessons can be learned from the implementation experience (both procedural and substantive).

The MCA staff may draft the Compact Completion Report (CCR) in the last year of compact implementation to evaluate these fundamental questions and other aspects of Compact program performance. It should be noted that each department will be responsible for drafting its own section to the report for its own activities, subject to cross-departmental review. The MCC staff, specifically the Country Team, may draft the Compact Process Evaluation Report (CPER) within 6 months after the compact ends to evaluate these same fundamental questions and other aspects of Compact program performance. Similar to the CCR, each division will be responsible for drafting its own section of the report, subject to cross-department review.

No Evaluation. The EMC may determine that no evaluation is feasible or necessary depending on the program design and rules of implementation. Examples of reasons why MCC would not invest in an independent evaluation include: (i) the program is Not Evaluable, or (ii) the cost of conducting an independent evaluation outweighs the benefits anticipated from the results of an independent evaluation.

To Be Determined (TBD). In some contexts, it may not be clear whether or not a counterfactual can be defined. In these specific situations, MCC may proceed with hiring an independent evaluator to guide development of the evaluation design, and assess the feasibility of an impact evaluation.

Although the MCC M&E lead must complete a Compact-level Evaluation Plan, the Evaluation Plan can be developed in stages for each Project (or other level as appropriate) as the Project is designed. Given that Projects are developed at different rates within a Compact, this provides flexibility for the MCC M&E lead to complete the sections of the Compact Evaluation Plan as needed. The content of the Evaluation Plan will vary depending on the status of the more comprehensive Compact M&E Plan, however it will be assessed for completeness by the EMC using the Checklist.

Collaboration. The MCC M&E lead will collaborate with the EMC members, MCA (if established), and local stakeholders on development of the Evaluation Plan, including building local buy-in on the proposed evaluation activities and identifying potential consumers of the evaluation results.

Timeline. The Evaluation Plan should be drafted and submitted by the MCC M&E lead to the EMC as soon as possible upon completion of the necessary triggers for developing the Evaluation Plan. This is required to enable time for subsequent procurement of Evaluation Firms, development of Evaluation Design Reports and coordination with program design and implementation in an adequate timeframe.

Compact Evaluation Plan Review Process. Prior to submission to the EMC, the M&E lead will share the proposed Evaluation Plan with the MCA (if established) and local stakeholders in the Compact country to build local buy-in for the proposed evaluation activities and assess demand for the evaluation results.

The MCC M&E lead is responsible for collating all comments into one comprehensive summary.

The EMC will provide either Clearance or No Clearance on the Evaluation Plan using the Checklist. In the event of disagreement between EMC Members, the Chair will confer with relevant DCO management to reach agreement on next steps.

In the event that an Evaluation Plan receives a No Clearance rating, the MCC M&E lead will need to revise the Evaluation Plan based on the EMC feedback and submit a revised Evaluation Plan until it receives Clearance.

It should be noted that the Evaluation Plan is a living document, and the MCC M&E lead is responsible for revising it as necessary. At a minimum, the MCC M&E lead should review the Evaluation Plan annually. It is highly recommended that it is reviewed prior to budget proposals for the following fiscal year in order to ensure alignment with budget requests.

Management Approval. Prior to final approval by the EMC, the Evaluation Plan must be presented at the respective DPE and DCO managing director and deputy vice president levels in order to obtain management approval on the Evaluation Plan.

Template 2: Evaluation Plan and Budget

3.2.1 Contract Evaluation Firm(s)

Based on the approved Evaluation Plan, the MCC M&E lead will manage the contracting process for the Evaluation Firm(s) as appropriate (i.e. by Compact, by sector or by Project, by Activity, etc.). EMC Members may serve on the Technical Evaluation Panel (TEP) as appropriate.

Collaboration. The MCC M&E lead will develop the TOR and circulate to EMC members, and if an MCA is established, to the appropriate MCA M&E and Sector counterparts, for review and comment. The TOR will be based on the program logic, evaluation questions, and proposed methodology (when appropriate) developed in the Evaluation Plan, as well as standard Evaluation Firm contract language.

Timeline. The Evaluation Firm(s) should be hired per the proposed timeline in the approved Evaluation Plan. The timeline must allow sufficient time for development of the Evaluation Design Report(s) and any required primary baseline data collection prior to project start date to enable coordination with project design and implementation in an adequate timeframe.

Template 3: Evaluation Firm TOR

3.3 Evaluation Design Report(s)

The Evaluation Firm, managed by the MCC M&E lead, will lead development of the Evaluation Design Report(s) in consultation with MCC EMC members, MCA M&E and Sector leads, Implementing Entities, and other relevant stakeholders.

Prior to developing the Evaluation Design Report(s), the Evaluation Firm will review and assess the Evaluation Plan for completeness and consistency. The Evaluation Firm will document any gaps or reasons for deviation from the Evaluation Plan in the Evaluation Design Report(s). This ensures the final Evaluation Design produces an independent, unbiased evaluation of the MCA intervention(s). In addition, the MCC M&E lead will use the assessment to determine if any revisions are required to the Evaluation Plan.

The content of the Evaluation Design Report will vary depending on the Evaluation Firm and program, however it should reference the MCC Evaluation Design Report template.

Collaboration. The Evaluation Firm(s) will lead the development of the Evaluation Design Report(s), in consultation with the MCC EMC, MCA M&E and Sector leads, Implementing Entities, and any other relevant stakeholders. The Evaluation Firm should expect at least one-two trips in-country to develop the design with local stakeholders in order to ensure the evaluation methodology is feasible and there is firm commitment by the MCA and other necessary local stakeholders to employ the proposed evaluation methodology.

Timeline. The Evaluation Design Report is expected to be submitted to the EMC as per the agreed timeline set in the approved Work Plan. This is required to enable time for procurement of any necessary data collection firm in coordination with program design and implementation.

Evaluation Design Report Review Process. Prior to submission to MCC, the Evaluation Firm will share the proposed Evaluation Design Report (at minimum, the research questions, evaluation methodology, sample size requirements, and data requirements) with the MCA M&E and Sector counterparts, as well as any Implementing Entity and other relevant stakeholders. All comments and responses will be documented by the Evaluation Firm (Annex 1 of the Report). The goal is to ensure local commitment and understanding to the proposed evaluation design prior to official submission to the MCC EMC.

The Evaluation Firm will present the Evaluation Design Report to the EMC to assess for (i) technical rigor, (ii) agency and policy relevance, (iii) operational risks and risk mitigation strategies, including necessary coordination between DCO and DPE, and (iv) local stakeholder commitment. The MCC M&E lead is responsible for collating all MCC comments into one comprehensive summary. The Evaluation Firm will document all responses to MCC comments (Annex 1 of the Report).

The EMC will provide either Clearance or No Clearance to proceed with the evaluation using the Checklist. In the event of disagreement between EMC Members, the Chair will confer with relevant DCO management to reach agreement on next steps.

Following the formal clearance of the Evaluation Design Report, MCC will circulate the Final Evaluation Design Report and the Evaluation Firm will present the Evaluation Design to stakeholders. The Final Evaluation Design Report, with Annex 1, will also be publicly posted on the MCC website.

In the event that an Evaluation Design Report receives a No Clearance rating, the Evaluation Firm will need to revise the Evaluation Design Report based on the EMC feedback and submit a revised Evaluation Design Report.

The EMC may also determine the evaluation is Not Cleared, either because of the proposed evaluation design or because of other risks identified during the assessment.