Annex_H_-_MCC_Evaluation_Management_and_Review_Process_DRAFT.pdf

MCC Independent Evaluations Evaluation Management and Review Process

DRAFT: 8/5/2016

Contents 1 Introduction

1.1 What are the different roles and responsibilities?

1.2 What are the objectives?

1.3 How will we meet these objectives?

2 Evaluation Management Committee

3 Evaluation Management Process

3.1 Evaluability Assessment

3.2 Evaluation Plan

3.2.1 Contract Evaluation Firm(s)

3.3 Evaluation Design Report(s)

3.3.1 Survey Firm TOR and Evaluation Materials

3.3.2 Baseline Report

3.4 Final (and Interim) Evaluation Report

1 Introduction The Millennium Challenge Corporation (MCC) invests in independent evaluations to measure results of MCC-financed and Millennium Challenge Account (MCA)-executed interventions in order to hold MCC and MCAs accountable for results and to produce continuous learning. The independent evaluations are conducted by third-party independent experts to ensure that the evaluations represent an unbiased assessment of the interventions. Ideally, evaluations should be designed to answer three fundamental questions related to accountability and transparency:

• Was MCC’s investment implemented according to plan?

• Were there any changes in key outcomes, such as income, for program participants that are attributable to MCC’s investment?

• Why did or didn’t the planned investments lead to changes in key outcomes, such as income?

However, it is widely recognized that these three separate objectives cannot always be met through one evaluation, and there are necessary trade-offs depending on priorities. Prioritizing the objectives of each evaluation is something that MCC, and many other organizations, continue to struggle with when deciding how to guide independent evaluators during evaluation design. MCC is currently planning or actively managing over 150 independent evaluations, and this issue is expected to be raised for each evaluation continuously throughout the evaluation review process.

This Guidance Note outlines how to strengthen MCC’s independent evaluations by summarizing the (i) roles and responsibilities of the independent evaluators, MCC, and MCAs, (ii) current challenges MCC is facing to ensure high quality, independent evaluations, (iii) objectives of the formal MCC review process to address these challenges, and (iv) its structure and the critical milestones that will be established to review evaluation activities and deliverables.

1.1 What are the different roles and responsibilities?

Each institution, the MCC, MCA, and independent evaluator, as well as other key stakeholders, play critical roles in designing, implementing, and disseminating the independent evaluations.

Their roles are as follows:

• Independent evaluator. The independent evaluator is responsible for the overall design, implementation, and dissemination of the evaluation, including the following specific responsibilities:

o Develop a rigorous evaluation design given rules of implementation and feasibility of options. Using the program logic as the foundation, independent evaluators must assess the program design and implementation to develop the most rigorous evaluation design feasible, whether it is a performance or impact evaluation, and identify the most appropriate evaluation methodology feasible given the context. The independent evaluator is responsible for developing an evaluation design report, in consultation with

MCC and MCA, which summarizes the evaluation design, sampling, analysis plan, and other critical design elements of the evaluation.

o Support MCC and MCA to build buy-in and ownership of evaluation. The independent evaluator will meet with MCC and local stakeholders to develop the evaluation design based on program design and implementation. When possible, the independent evaluator will work with MCC and MCA to advise on a program implementation roll-out plan that enables a rigorous evaluation design. In addition, the independent evaluator will continuously present the evaluation objectives, materials, and interim and final results for MCC and local stakeholders to maintain commitment to the evaluation.

o Develop evaluation materials that are held to international standards. Using the program logic as a foundation, the independent evaluators define how key outcomes will be measured and develop the survey instruments or other data collection tools in order to answer critical evaluation questions.

o Ensure appropriate review of evaluation materials and research protocols.

Independent evaluators are responsible for ensuring all evaluation materials, including the evaluation design, survey instruments, sampling strategy, data collection and entry protocols are reviewed by appropriate local stakeholders prior to submission to MCC.

The goal of this review is to ensure that local stakeholders are able to comment on the feasibility of the proposed methodology, as well as the technical and factual accuracy of the materials.

o Ensure Institutional Review Board clearance. The Independent Evaluator is responsible for submitting evaluation materials to an institutional review board (IRB).

IRB review and approval is required before any data collection commences. The independent evaluators will document all approvals and informed consent procedures throughout the evaluation time period.

o Manage the data collection firms. This responsibility may differ in the Compact period compared to the Post-Compact period. However, as appropriate, the evaluator may develop Terms of Reference for the data collection firm, with technical support from MCC and the MCA. The independent evaluator may be responsible for managing the contracting process, including Request for Proposals, Proposal Review, selection of the data collection firm, development of a work plan, and supervision of the data collection firm in line with the work plan and agreed deliverables.

o Supervise data collection. Whether the MCA, MCC or independent evaluator contracts the data collection firm, the independent evaluator is responsible for quality control over data collection activities, including interviewer training, pre-testing, and implementing data quality review checks during data collection.

o Lead data cleaning, analysis, interpretation of results. The independent evaluator is responsible for leading any necessary data cleaning and consultation with the data collection firm to produce final analysis files. The independent evaluator is responsible for leading all analysis in line with the agreed analysis plan developed with MCC, and consulting with MCC should the analysis plan need to be revised or adapted.

o Produce evaluation reports. The independent evaluator is responsible for sharing initial evaluation reports with local stakeholders and MCC for review and feedback. However, given the independent evaluator is the author of the evaluation reports, local stakeholder and MCC feedback may or may not be taken into consideration for the final report. The independent evaluator is responsible for documenting all feedback and evaluator responses.

o Lead public dissemination efforts. The independent evaluator is responsible for leading public dissemination efforts facilitated by MCA and MCC (such as local workshops and conferences), but also additional conferences or other opportunities to publicly disseminate the results of the evaluation. The evaluator will advise MCC on any public dissemination opportunities and collaborate as appropriate.

• MCC. MCC is responsible for oversight of the independent evaluator and quality control of evaluation activities, including the following specific responsibilities:

o Set the evaluation priorities and evaluation questions. Using the program logic and priorities of key users of the evaluation results as the foundation, MCC will develop the evaluation priorities and evaluation questions based on existing evidence, the evidence gap, and the learning priorities for the sector and agency. These will be developed in consultation with the MCA and local stakeholders to maximize use of the evaluation results and their contribution to future decision making.

o Build buy-in and ownership of the evaluation. MCC will work with the independent evaluator and MCA to ensure the evaluation design follows the rules of program design and implementation. When possible, MCC will work with the independent evaluator and MCA to develop a program implementation roll-out plan that enables a rigorous evaluation design. When it is not possible to develop a coordinated roll-out plan, the evaluator will ensure the evaluation design is based on feasible methodologies given program design and implementation. MCC will work with MCA to ensure commitment to the roll-out plan by local stakeholders, and if is not possible to adhere to the roll-out plan, will inform the independent evaluator in order to adapt the evaluation design.

o Determine the evaluation budgets. MCC will develop an evaluation budget, including independent evaluators, expected data collection requirements, and any other activities related to the evaluation.

o Manage the independent evaluators. MCC will develop Terms of Reference for the independent evaluators and manage the contracting process, including Request for Proposals, Proposal Review, selection of the independent evaluator, development of a work plan, and supervision of the independent evaluator in line with the work plan and agreed deliverables.

o Quality review of evaluation materials. Using the program logic and evaluation priorities as a foundation, MCC is responsible for reviewing evaluation materials, including the design report and survey materials, for feasibility, technical and factual accuracy, and adherence with international standards (when applicable).

o Quality review of evaluation reports. MCC is responsible for reviewing independent evaluators’ reports to ensure technical and factual accuracy. While MCC will provide feedback, the independent evaluator is ultimately the author of the evaluation reports and may or may not take into consideration MCC’s comments. MCC therefore develops an Official Statement for each evaluation report to confirm the technical and factual accuracy of the report, or clarify any differences on technical or factual issues between MCC and the independent evaluator.

o Lead initial public dissemination efforts. MCC is responsible for leading initial public dissemination of results, in particular ensuring a feedback loop internally in order to inform future decision making on similar program investment, design, and implementation. This requires ensuring the evaluation results are disseminated on the MCC website, and results are disseminated to appropriate stakeholders in a timely fashion.

• MCA. MCA is responsible for building local ownership and commitment to the evaluation, oversight of the data collection firm, and quality control of evaluation activities, including the following specific responsibilities:

o Build buy-in and ownership of the evaluation. MCA will work with the independent evaluator and MCC to ensure the evaluation design follows the rules of program design and implementation. When possible, MCA will work with the independent evaluator and MCC to develop a program implementation roll-out plan that enables a rigorous evaluation design. MCA will work with MCC to ensure commitment to the roll-out plan by local stakeholders, and if it is not possible to adhere to the roll-out plan, will inform MCC in order to request the independent evaluator adapt the evaluation design.

o Manage the data collection firms. This responsibility may differ in the Compact period compared to the Post-Compact period. However, as appropriate, the evaluator may develop Terms of Reference for the data collection firm, with technical support from MCC and the MCA. The independent evaluator may be responsible for managing the contracting process, including Request for Proposals, Proposal Review, selection of the data collection firm, development of a work plan, and supervision of the data collection firm in line with the work plan and agreed deliverables.

o Quality review of evaluation materials. Using the program logic as a foundation, MCA is responsible for reviewing evaluation materials, including the design report and survey materials, for feasibility, technical and factual accuracy, and adherence with international standards (when applicable).

o Quality review of evaluation reports. MCA, or a designated representative, is responsible for reviewing independent evaluators’ reports to ensure technical and factual accuracy. While MCA will provide feedback, the independent evaluator is ultimately the author of the evaluation reports and may or may not take into consideration MCA’s comments. MCA therefore develops an Official Statement for each evaluation report to confirm the technical and factual accuracy of the report, or clarify any differences on technical or factual issues between MCA and the independent evaluator.

o Facilitate public dissemination efforts. MCA, or a designated representative, is responsible for facilitating public dissemination of results, in particular ensuring a feedback loop in-country in order to inform future decision making on similar program investment, design, and implementation.

1.2 What are the objectives?

The objectives of the formal MCC review process are to convert core lessons learned from the first five impact evaluations into action:

• Ensure early integration of evaluation with program design and implementation. This requires close collaboration between DPE and DCO as early as possible in the process, and then continuously throughout Compact development, implementation, and Post Compact.

• Ensure clearly defined program logic. This requires clearly defining the inputs, outputs, outcomes, and goals of each intervention in a way that demonstrates how the intervention will lead to expected results. For this, there must be common understanding across stakeholders on the program logic and the assumptions underlying the program logic.

• Ensure evaluations are structured for learning and work toward establishing a feedback loop. This requires understanding the assumptions underlying the program logic for how an intervention will lead to improved outcomes, and ultimately an increase in beneficiary household income, and what questions need to be answered in order to inform future decision-making. This also requires identifying the consumers of the evaluation results and ensuring a feedback loop is established so that evaluation results can inform future decision-making.

• Ensure appropriate quality control mechanisms are in place. This requires ensuring the appropriate stakeholders are reviewing key evaluation actions and deliverables, and assessing risk to the interventions, and their evaluations, in a timely manner.

• Establish transparent and clear decision making-process. Although the guidelines proposed here may evolve over time as we learn what works and what doesn’t, this is the first step to establishing a formal internal review process to ensure that evaluations are designed and implemented to produce high quality, rigorous evidence that can be used by MCC and host country governments for future decision-making.

1.3 How will we meet these objectives?

This Guidance Note outlines the formal internal review process which defines who should be involved in decisions regarding evaluation activities and deliverables, and when.

• Who needs to be involved? A formal Evaluation Management Committee will be established for each Compact, chaired by the Monitoring and Evaluation Managing Director, or a designated representative, and consisting of the appropriate representation from DPE and DCO for Compact activities, including senior management (MDs, DVPs, VPs) when appropriate.

• When do they need to be involved? Critical milestones have been established for the Evaluation Management Committee to meet and review proposed evaluation activities, as well as core evaluation deliverables. The critical milestones are: (i) Assess Evaluability, (ii) Develop the Evaluation Plan, (iii) Develop the Evaluation Design and subsequent evaluation deliverables (survey instruments, baseline report(s)), and (iv) Develop the Interim/Final Evaluation Report.

2 Evaluation Management Committee In order to clearly define who needs to be involved in critical decisions related to independent evaluations, an Evaluation Management Committee (EMC) will be established for each Compact comprised of one Chair and four-six Committee members.

Chair:

• Monitoring and Evaluation Managing Director, or designated representative

Members1:

• Monitoring and Evaluation Lead. The Monitoring and Evaluation (M&E) lead who is responsible for overall leadership on the independent evaluation activities and ensuring common understanding of the program logic(s), evaluation questions, contracting Evaluation Firm(s), and quality control for evaluation materials and reports. It is to the discretion of the committee if an additional M&E lead would provide a peer review as well.

• Evaluation Support. (1) technical support from evaluation team dependent on sector.

a. Jack Molyneaux: Agriculture & Irrigation; Transportation; Financial, Investment, and

Trade; Capacity Building and Institutional Development

b. Jennifer Sturdy: Health; Education; Energy; Water and Sanitation

c. Jennifer Witriol: Land

• Economic Analysis Lead. The Economic Analysis (EA) lead responsible for assessing the technical design elements, including: program logic, assumptions, evaluation questions, identification strategy, analysis plan, and alignment with the ERR(s) and beneficiary analysis.

It is to the discretion of the committee if an additional EA lead would provide a peer review as well.

• Sector Lead(s). The Sector Lead(s) responsible for defining program logic, assumptions, evaluation questions, and identifying users of evaluation results. Please note, in the Post- Compact period, the Sector Practice Lead will be requested to identify appropriate Sector Lead(s) for reviewing materials Post-Compact if necessary.

• Country Relations. One representative from the Country Team Lead (CTL)/Resident Country Mission (RCM) to assess policy relevance and risk assessment for implementation challenges.

Please note, in the Post-Compact period, the DCO Managing Director will be asked to identify a designated representative, if possible.

Please note, DCO and DPE management should be brought into EMC reviews at two specific milestones: (i) Review of the Evaluation Plan, and (ii) Review of Interim/Final Reports.

1 Given that participation on the Evaluation Management Committee will require staff time and is crucial to the evaluation quality assurance process, MCC should include participation on the EMC as part of Committee members’ performance plans.

It is anticipated that most EMCs will be established at the Compact-level. The M&E lead will collaborate with the Country Team to determine what is appropriate and to identify the appropriate EMC members. Sector Leads are only expected to participate in reviews of the evaluations in their sector.

The EMC should be established as early in Compact development as possible, and members should expect to participate in the EMC for the duration of the evaluation period. In the event that an EMC member must leave the EMC, the Chair, in collaboration with DPE and DCO management, must identify the appropriate replacement for the EMC member.

External stakeholders, such as the host-country MCA, government, Implementing Entities, and MCA contractors will be included as necessary in each stage of the evaluation process as described in the sections below. In addition, MCAs are encouraged to establish their own, parallel EMC to facilitate appropriate local stakeholder review of the materials.

3 Evaluation Management Process The Evaluation Management Process consists of several key milestones:

1. Assess Evaluability

2. Develop the Evaluation Plan

2.1. Contract the Evaluation Firm(s)

3. Develop the Evaluation Design Report(s)

3.1. Develop the Evaluation Materials (survey instruments, training manuals, etc)

3.2. Develop the Evaluation Baseline Report(s)

4. Develop the Interim and Final Evaluation Report(s)

Subsequent sections provide more detail on the objectives and requirements of each milestone in the Evaluation Management Process. These milestones are intended to serve as critical points in time for the respective EMC members to assess the cost-benefit of continuing, or course correcting, the evaluation activity under review.

At each milestone, the EMC members will be asked to use the Evaluation Risk and Status Assessment Checklist (Annex X) to assess (i) the evaluation activity/deliverable under review, and

(ii) the current risks and cost-benefit of proceeding with the evaluation. For this reason, the MCC M&E lead may propose to convene the EMC (either virtually or in face-to-face meeting) at other points in time in addition to these milestones to discuss any significant changes to the agreed evaluation activities that pose a risk to the evaluation activities and deliverables. Additional actions that may trigger an EMC review and/or meeting to assess the cost-benefit of continued evaluation activities include:

• Project re-scoping or re-structuring

• Significant deviations from original work plan

• Evaluation Firm contract modifications

• Loss of a counterfactual

3.1 Evaluability Assessment

The MCC M&E and Sector Leads will initiate discussions around program evaluability at the Concept Note and Concept Paper stages2. Evaluability is defined as the ability of an intervention to demonstrate in measurable terms the results it intends to deliver.

Incorporating Evaluability Assessment into Compact Development and Implementation is still under discussion.

Template 1: Evaluability Assessment Tool

3.2 Evaluation Plan

The MCC M&E lead is responsible for developing the Compact Evaluation Plan, a section in the overall Compact M&E Plan. The Evaluation Plan should identify priority learning opportunities, propose impact or performance evaluations for each Compact intervention, and prepare estimated budgets (MCC and MCA) for each evaluation activity with a corresponding procurement plan for consultants/IDIQ firms and work plan. The Evaluation Plan should also clearly indicate if any intervention will not be evaluated, and the justification for not evaluating.

The following must be developed prior to developing the Evaluation Plan:

• Diagnostic and problem identification

• Clearly defined program logic

• Literature review of evidence on proposed intervention(s)

• Initial economic rate of return models and beneficiary analysis (this is optional as it may be developed in parallel with the evaluation plan.)

For the Evaluation Plan, the M&E lead will indicate whether or not the Project/Activity/Sub- Activity will be covered by (i) an impact evaluation, (ii) a performance evaluation, (iii) self-evaluation only, (iv) no evaluation, or (v) to be determined.

• Impact Evaluation. According to MCC M&E Policy, a study that measures the changes in income and/or other aspects of well-being that are attributable to a defined intervention. Impact

2 DPE currently discussing how best to accomplish this, potentially through check-lists for Concept Note and Concept Paper review in order to provide feedback on evaluability early in the process.

evaluations require a credible and rigorously defined counterfactual, which estimates what would have happened to the beneficiaries absent the project. Estimated impacts, when contrasted with total related costs, provide an assessment of the intervention’s cost-effectiveness. In should be noted, both Impact and Performance Evaluations are expected to document any difference between what was planned and what actually happened as well as understanding why, if any, difference occurred between the two.

• Performance Evaluation. According to MCC M&E Policy, A study that seeks to answer descriptive questions, such as: what were the objectives of a particular project or program, what the project or program has achieved; how it has been implemented; how it is pEMCeived and valued; whether expected results are occurring and are sustainable; and other questions that are pertinent to program design, management and operational decision making. MCC’s performance evaluations also address questions of program impact and cost-effectiveness. In should be noted, both Impact and Performance Evaluations are expected to document any difference between what was planned and what actually happened as well as understanding why, if any, difference occurred between the two.

• Self-Evaluation only. Upon completion of each Compact program, both MCC and MCA may comprehensively assess three fundamental questions: (i) Did the MCA program meet Compact objectives; (ii) Why did the Compact program meet or not meet these objectives; and (iii) What lessons can be learned from the implementation experience (both procedural and substantive).

The MCA staff may draft the Compact Completion Report (CCR) in the last year of compact implementation to evaluate these fundamental questions and other aspects of Compact program performance. It should be noted that each department will be responsible for drafting its own section to the report for its own activities, subject to cross-departmental review. The MCC staff, specifically the Country Team, may draft the Compact Process Evaluation Report (CPER) within 6 months after the compact ends to evaluate these same fundamental questions and other aspects of Compact program performance. Similar to the CCR, each division will be responsible for drafting its own section of the report, subject to cross-department review.

• No Evaluation. The EMC may determine that no evaluation is feasible or necessary depending on the program design and rules of implementation. Examples of reasons why MCC would not invest in an independent evaluation include: (i) the program is Not Evaluable, or (ii) the cost of conducting an independent evaluation outweighs the benefits anticipated from the results of an independent evaluation.

• To Be Determined (TBD). In some contexts, it may not be clear whether or not a counterfactual can be defined. In these specific situations, MCC may proceed with hiring an independent evaluator to guide development of the evaluation design, and assess the feasibility of an impact evaluation.

Although the MCC M&E lead must complete a Compact-level Evaluation Plan, the Evaluation Plan can be developed in stages for each Project (or other level as appropriate) as the Project is designed. Given that Projects are developed at different rates within a Compact, this provides flexibility for the MCC M&E lead to complete the sections of the Compact Evaluation Plan as needed. The content of the Evaluation Plan will vary depending on the status of the more comprehensive Compact M&E Plan, however it will be assessed for completeness by the EMC using the Checklist.

Collaboration. The MCC M&E lead will collaborate with the EMC members, MCA (if established), and local stakeholders on development of the Evaluation Plan, including building local buy-in on the proposed evaluation activities and identifying potential consumers of the evaluation results.

Timeline. The Evaluation Plan should be drafted and submitted by the MCC M&E lead to the EMC as soon as possible upon completion of the necessary triggers for developing the Evaluation Plan. This is required to enable time for subsequent procurement of Evaluation Firms, development of Evaluation Design Reports and coordination with program design and implementation in an adequate timeframe.

Compact Evaluation Plan Review Process. Prior to submission to the EMC, the M&E lead will share the proposed Evaluation Plan with the MCA (if established) and local stakeholders in the Compact country to build local buy-in for the proposed evaluation activities and assess demand for the evaluation results.

The MCC M&E lead is responsible for collating all comments into one comprehensive summary.

The EMC will provide either Clearance or No Clearance on the Evaluation Plan using the Checklist. In the event of disagreement between EMC Members, the Chair will confer with relevant DCO management to reach agreement on next steps.

In the event that an Evaluation Plan receives a No Clearance rating, the MCC M&E lead will need to revise the Evaluation Plan based on the EMC feedback and submit a revised Evaluation Plan until it receives Clearance.

It should be noted that the Evaluation Plan is a living document, and the MCC M&E lead is responsible for revising it as necessary. At a minimum, the MCC M&E lead should review the Evaluation Plan annually. It is highly recommended that it is reviewed prior to budget proposals for the following fiscal year in order to ensure alignment with budget requests.

Management Approval. Prior to final approval by the EMC, the Evaluation Plan must be presented at the respective DPE and DCO managing director and deputy vice president levels in order to obtain management approval on the Evaluation Plan.

Template 2: Evaluation Plan and Budget

3.2.1 Contract Evaluation Firm(s)

Based on the approved Evaluation Plan, the MCC M&E lead will manage the contracting process for the Evaluation Firm(s) as appropriate (i.e. by Compact, by sector or by Project, by Activity, etc.). EMC Members may serve on the Technical Evaluation Panel (TEP) as appropriate.

Collaboration. The MCC M&E lead will develop the TOR and circulate to EMC members, and if an MCA is established, to the appropriate MCA M&E and Sector counterparts, for review and comment. The TOR will be based on the program logic, evaluation questions, and proposed methodology (when appropriate) developed in the Evaluation Plan, as well as standard Evaluation Firm contract language.

Timeline. The Evaluation Firm(s) should be hired per the proposed timeline in the approved Evaluation Plan. The timeline must allow sufficient time for development of the Evaluation Design Report(s) and any required primary baseline data collection prior to project start date to enable coordination with project design and implementation in an adequate timeframe.

Template 3: Evaluation Firm TOR

3.3 Evaluation Design Report(s)

The Evaluation Firm, managed by the MCC M&E lead, will lead development of the Evaluation Design Report(s) in consultation with MCC EMC members, MCA M&E and Sector leads, Implementing Entities, and other relevant stakeholders.

Prior to developing the Evaluation Design Report(s), the Evaluation Firm will review and assess the Evaluation Plan for completeness and consistency. The Evaluation Firm will document any gaps or reasons for deviation from the Evaluation Plan in the Evaluation Design Report(s). This ensures the final Evaluation Design produces an independent, unbiased evaluation of the MCA intervention(s). In addition, the MCC M&E lead will use the assessment to determine if any revisions are required to the Evaluation Plan.

The content of the Evaluation Design Report will vary depending on the Evaluation Firm and program, however it should reference the MCC Evaluation Design Report template.

Collaboration. The Evaluation Firm(s) will lead the development of the Evaluation Design Report(s), in consultation with the MCC EMC, MCA M&E and Sector leads, Implementing Entities, and any other relevant stakeholders. The Evaluation Firm should expect at least one-two trips in-country to develop the design with local stakeholders in order to ensure the evaluation methodology is feasible and there is firm commitment by the MCA and other necessary local stakeholders to employ the proposed evaluation methodology.

Timeline. The Evaluation Design Report is expected to be submitted to the EMC as per the agreed timeline set in the approved Work Plan. This is required to enable time for procurement of any necessary data collection firm in coordination with program design and implementation.

Evaluation Design Report Review Process. Prior to submission to MCC, the Evaluation Firm will share the proposed Evaluation Design Report (at minimum, the research questions, evaluation methodology, sample size requirements, and data requirements) with the MCA M&E and Sector counterparts, as well as any Implementing Entity and other relevant stakeholders. All comments and responses will be documented by the Evaluation Firm (Annex 1 of the Report). The goal is to ensure local commitment and understanding to the proposed evaluation design prior to official submission to the MCC EMC.

The Evaluation Firm will present the Evaluation Design Report to the EMC to assess for (i) technical rigor, (ii) agency and policy relevance, (iii) operational risks and risk mitigation strategies, including necessary coordination between DCO and DPE, and (iv) local stakeholder commitment. The MCC M&E lead is responsible for collating all MCC comments into one comprehensive summary. The Evaluation Firm will document all responses to MCC comments (Annex 1 of the Report).

The EMC will provide either Clearance or No Clearance to proceed with the evaluation using the Checklist. In the event of disagreement between EMC Members, the Chair will confer with relevant DCO management to reach agreement on next steps.

Following the formal clearance of the Evaluation Design Report, MCC will circulate the Final Evaluation Design Report and the Evaluation Firm will present the Evaluation Design to stakeholders. The Final Evaluation Design Report, with Annex 1, will also be publicly posted on the MCC website.

In the event that an Evaluation Design Report receives a No Clearance rating, the Evaluation Firm will need to revise the Evaluation Design Report based on the EMC feedback and submit a revised Evaluation Design Report.

The EMC may also determine the evaluation is Not Cleared, either because of the proposed evaluation design or because of other risks identified during the assessment. In the event the EMC determines the evaluation is Not Cleared, MCC will not invest in the evaluation activity and the No Clearance rating will be documented to justify why the program does not have an evaluation.

Please note, the Evaluation Design Report is a living document. Even after EMC review, the Evaluation Firm is required to maintain the Evaluation Design Report and revise it when necessary to reflect the actual evaluation design. Any significant changes to the Evaluation Design Report will be shared virtually and will be presented to the EMC if necessary. At a minimum, the Evaluation Firm is expected to review the Evaluation Design Report annually to ensure it is up to date.

Template 4: Evaluation Design Report

3.3.1 Survey Firm TOR and Evaluation Materials

For survey firm procurement, it is highly recommended that the Evaluation Firm provides technical support on the selection of the firm, whether through an MCA, MCC or third party contracting mechanism. For MCA Procurements, it is highly recommended that the MCC M&E lead and Evaluation Firm provide support to the MCA on the development and/or review of the final survey firm TOR, as allowed and discussed with each respective procurement office. For MCA procurements, the MCC will maintain its role of objection/no objection for procurement actions. For MCC procurement, it is highly recommended that MCC considers including a representative from the Evaluation Firm as an observer of the selection committee in order to provide direct support on the selection of the survey firm based on technical capabilities.

Unless otherwise agreed by the EMC, the Evaluation Firm will develop materials (survey instruments, training manuals, etc.) for implementing the evaluation in consultation with the MCC EMC, MCA M&E and Sector Leads (Implementing Entities as necessary), and survey firm. For questionnaire development, it is highly recommended that the Evaluation Firm collaborate with stakeholders prior to and after pilot testing the questionnaire(s). This process will be replicated for all data collection rounds (baseline, interim, final).

Collaboration. The Evaluation Firm(s) will lead the development of the Evaluation Materials in consultation with the MCC EMC, MCA M&E and Sector leads, and any necessary local stakeholders.

Timeline. The Evaluation Materials should be reviewed and cleared as per the agreed timeline set in the approved Evaluation Plan.

Evaluation Materials Review Process. Prior to submission to MCC, the Evaluation Firm will share the proposed Evaluation Materials with the MCA M&E and Sector counterparts, as well as any Implementing Entity and other relevant stakeholders. All comments and responses will be documented by the Evaluation Firm (Annex 1 of the Report). The goal is to ensure local commitment and understanding to the proposed evaluation materials prior to official submission to the MCC EMC.

The EMC will review the Evaluation Materials for (i) linkage with the program logic, (ii) linkage with the M&E Plan, and specifically the Evaluation Plan, (iii) adherence to international standards, and (iv) policy relevance.

The EMC will provide either Clearance or No Clearance to proceed with the evaluation. In the event of disagreement between EMC Members, the Chair will confer with relevant DCO management to reach agreement on next steps.

In the event that the Evaluation Materials receive a No Clearance rating, the Evaluation Firm will need to revise the Evaluation Materials based on the EMC feedback and submit revised Evaluation Materials until it receives Clearance.

The EMC may also determine the evaluation is Not Cleared because of other risks identified during the assessment. In the event the EMC determines the evaluation is Not Cleared, MCC will not invest in the evaluation activity and the No Clearance rating will be documented to justify why the program does not have an evaluation.

Template 5: Survey Firm TOR(s)

3.3.2 Baseline Report

Within 12 months of the completion of baseline data collection3, the Evaluation Firm will produce a Baseline Report. The content of the Baseline Report will vary depending on the Evaluation Firm and Evaluation Design Report.

Collaboration. The Evaluation Firm(s) are encouraged to consult with the MCC EMC, MCA M&E and Sector Leads, and any necessary local stakeholders on the development of the Baseline Report.

3 Some evaluations do not collect baseline data. For these evaluations, the evaluation firm should ensure the Evaluation Design Report validates the evaluation design using available trend data and other sources to justify the decision to use end-line data only.

Timeline. The Baseline Report, including all relevant documentation, must be submitted within 12 months after baseline data collection.

In addition to the Baseline Report, Evaluation Firms and M&E teams are strongly encouraged to produce policy relevant papers for publication in peer review journals when possible.

Baseline Report Review Process. Prior to submission to MCC, the Evaluation Firm will share the Baseline Report with the MCA M&E and Sector counterparts, as well as any Implementing Entity and other relevant stakeholders. All comments and responses will be documented by the Evaluation Firm (Annex 1 of the Report). The goal is to ensure local commitment and understanding to the Baseline Report prior to official submission to the MCC EMC.

Once the Baseline Report is submitted to MCC, the EMC will review the Baseline Report for (i) technical rigor, (ii) policy relevance, (iii) operational risks and risk mitigation strategies, and (iv) appropriate documentation and storage of deliverables. The MCC M&E lead is responsible for collating all comments into one comprehensive summary. All MCC feedback and Evaluation Firm response must be documented by the Evaluator (Annex 1 of the Report).

Based on this review, the EMC will either provide Clearance or No Clearance to move forward.

In the event of disagreement between EMC Members, the Chair will confer with relevant DCO management to reach agreement on next steps.

Following the formal clearance of the Baseline Report, the Evaluation Firm will present the Final Baseline Report to stakeholders. The Final Baseline Report, with Annex 1, will be publicly posted on the MCC website.

In the event that an Evaluation Baseline Report receives a No Clearance rating, the Evaluation Firm will need to revise the Evaluation Baseline Report based on the EMC feedback and submit a revised Evaluation Baseline Report.

The EMC may also determine the evaluation is Not Cleared because of other risks identified during the assessment. In the event the EMC determines the evaluation is Not Cleared, MCC will halt investment in the evaluation activity and the No Clearance rating will be documented to justify why the program does not have an evaluation.

Public Release of Baseline Data. The public release of baseline data will follow the public release of the cleared Final Baseline Report. However, public release of baseline data must adhere to the MCC Data Documentation and Anonymization Guidelines and receive official clearance by the MCC Disclosure Review Board prior to public release.

Template 6: Evaluation Baseline Report

3.4 Final (and Interim) Evaluation Report

The Evaluation Firm will produce a Final4 Evaluation Report for the project. The content of the Evaluation Report will vary depending on the Evaluation Firm and program, however the Evaluation Firm should reference the Final Evaluation Report template.

Collaboration. The Evaluation Firm is encouraged to consult with the MCC EMC, MCA M&E and Sector Leads, and any necessary local stakeholders on the development of the Final Evaluation Report.

Timeline. The Final Evaluation Report, including all relevant documentation, should be submitted within 12 months after follow-up data collection.

In addition to the Final Evaluation Report, Evaluation Firms, MCC, MCA and government stakeholders are strongly encouraged to produce policy relevant papers for publication in peer review journals when possible.

Final Evaluation Report Review Process. For Interim and Final Evaluation Reports, the Review Process will be five steps:

Step 1 – Evaluation Firm Consults with Local Stakeholders. As with all previous key evaluation deliverables, the Evaluation Firm is responsible for sharing the Evaluation (Interim and/or Final) Report(s) with the MCA and any local stakeholders for review and comment prior to formal submission to MCC. This may be done virtually or through a local stakeholder workshop.

The purpose of this review is to ensure that the Evaluation Firm has addressed any technical or factual errors from the report. For MCA stakeholders, during the Compact period this should include the MCA CEO, MCA Sector Lead and MCA M&E lead. In Post Compact period, the MCA should designate appropriate representatives to review the report. Special consideration should be made on who these designated representatives are in order to review the report for technical and factual accuracy and they should be identified in the Post-Compact M&E Plan. The MCA and any other necessary local stakeholders will provide written feedback to the Evaluation Firm. Given this is an independent evaluation, the Evaluation Firm will provide written response to any comments and may elect not to take comments into consideration for the Final Evaluation Report. Any key stakeholder will then be given the opportunity to provide a public “Statement of

4 In the event that the evaluation calls for an Interim Report, the M&E lead will follow the same process for an Interim Evaluation Report.

Difference or Support” based on the evaluation firm’s report and response to feedback. The submission of the Evaluation Report(s) to MCC will include the following:

• Annex 1: Documentation of the comments/feedback from stakeholders.

• Annex 2: Any “Statement of Difference or Support” from the key local stakeholders.

Step 2 – Evaluation Firm submits for MCC review. As with all previous key evaluation deliverables, the Evaluation Firm will then submit the Evaluation Report to the EMC to review for

(i) completeness, (ii) technical rigor, (iii) policy relevance, (iv) appropriate dissemination activities and (v) appropriate documentation and storage activities. The EMC will ensure that DPE and DCO management is invited to participate in the review. If any factual or technical errors are found in the report, written comments will be submitted to the Evaluation Firm for consideration. In addition, the EMC may elect to request an external peer review of the Evaluation Report, in which case peer reviewer comments will also be provided to the Evaluation Firm. The MCC M&E lead is responsible for collating all comments into one comprehensive summary. Given this is an independent evaluation, the Evaluation Firm will provide written response to any comments and may elect not to take comments into consideration for the Final Evaluation Report. These comments and responses must be documented by the Evaluation Firm and submitted with the final Evaluation Report as part of the deliverables package. MCC will draft the MCC Management Response5 and Summary of Findings6 for the evaluation to accompany the public dissemination of the Evaluation Report.

• Annex 1: Documentation of the comments/feedback from stakeholders.

Step 3 – MCC submits package to MCA/designated representative for formal review. Third, the EMC will send the following to the MCA, or designated representative: (i) Final Evaluation Report with Annex I, (ii) MCC Summary of Findings, and (iii) Draft MCC Response. The MCA will review and have the option to provide an Official Country Response7 on the Evaluation Report, if necessary. If the Evaluation Report must be returned to the Evaluation Firm for factual or technical errors, the time will reset at Step 2. However, this should be highly unlikely given MCA’s participation in Step 1.

Step 4 – Final submission to MCC VP Review. The M&E Managing Director, or designated representative, will send the following to MCC VPs for DCO, DPE, OGC, CPA, as well as the

5 The MCC Response is an official MCC management statement that confirms MCC acceptance of the final report and documents any outstanding differences between MCC and the Evaluation Firm as it relates to (i) factual and/or

(ii) technical issues.

6 The MCC Summary of Findings places the evaluated program in context of the overall Compact, summarizes the program logic, assumptions, monitoring indicators and results, as well as evaluation questions and key results. The Summary of Findings also summarizes key lessons learned by MCC resulting from the evaluation findings.

7 The Official Country Response is an official host-country statement that either confirms their acceptance of the evaluation findings or documents any outstanding differences between the host country and the Evaluation firm as it relates to (i) factual and/or (ii) technical issues.

Chief of Staff: (i) Final Evaluation Report with Annex I, (ii) MCC Summary of Findings, (iii) Draft MCC Response Statement, and (iv) Official Country Response (if provided). MCC VPs will finalize the Official MCC Response Statement to be posted with the final, public Evaluation Report.

Step 5 – Public Dissemination. Following the formal review of the Final Evaluation Report with Annex I, MCC Summary of Findings, Country Response and MCC Response, the documents will be sent to CPA for branding. Once branded, the documents will be publicly posted on the MCC Evaluation Catalog. In addition, the Evaluation Firm is expected to present the Evaluation results in MCC headquarters, as well as an in-country workshop to appropriate stakeholders.

Public Release of Interim/Final Data. The public release of interim/final data will follow the public release of the cleared Interim/Final Evaluation Report. However, public release of interim/final data must adhere to the MCC Data Documentation and Anonymization Guidelines and receive official clearance by the MCC Disclosure Review Board prior to public release.

Template 7: Final Evaluation Report http://data.mcc.gov/evaluations/index.php/catalog http://data.mcc.gov/evaluations/index.php/catalog

3.4 Final (and Interim) Evaluation Report