J.1._MCC_Evaluation_Management_and_Review_Process.docx

Contents

Overview The Millennium Challenge Corporation (MCC) invests in independent evaluations to measure results of MCC-financed and Millennium Challenge Account (MCA)-executed programs. The independent evaluations are conducted by third-party experts to produce high quality, credible analysis that hold MCC and country partners accountable for whether or not intended results of the program occurred and contribute directly to learning why and why not for future decision-making. Therefore, each evaluation is guided by three questions:

· Was the program implemented according to plan (in terms of quantity and quality of outputs)?

· Were there any changes in key outcomes for program participants that may be attributable to MCC’s investment?

· Why did or didn’t the investments lead to changes in key outcomes, such as income?

These guidelines outline MCC’s Evaluation Management Process (EMP) to define the (i) roles and responsibilities of the independent evaluators, MCC, and MCAs, (ii) objectives of the formal MCC review process, and (iii) the milestones and risk factors that inform how MCC manages the evaluation life cycle with these objectives in mind.

Roles and Responsibilities MCC, MCAs, and independent evaluator - as well as other key stakeholders - play critical roles in designing, implementing, and disseminating the independent evaluations. Their roles and responsibilities are defined in the Evaluator Scope of Work (SOW). Generally, they are:

Millennium Challenge Corporation (MCC): MCC is responsible for oversight of the Independent Evaluator and quality control of evaluation activities, including the following specific responsibilities:

1. Assess when a program is ready for evaluation planning through evaluability assessment;

2. Determine what program components (Project(s), Activities, Sub-Activities) will be covered by the evaluation;

3. Set the evaluation questions to achieve accountability and learning objectives;

4. Build buy-in and ownership of the evaluation;

5. Contract and Supervise the Independent Evaluator;

6. Conduct quality reviews of all evaluation products (reports, questionnaires, etc);

7. Facilitate public dissemination efforts to inform decision-makers on learning generated by the evaluation;

8. Oversee preparation of public-use data that ensures appropriate balance of enabling verification of analysis and broad use of the data with adherence to promises of confidentiality to survey respondents; and

9. Determine the evaluation budgets.

MCC M&E ensures appropriate documentation and management of its Evaluation Portfolio in the Evaluation Pipeline. Although historically excel-based, MCC M&E is working on migrating to a web-based platform with final data migration expected by December 2017:

· Evaluation Pipeline (excel database)

· Evaluation Pipeline (web-based application – December 2017 launch)

Millennium Challenge Account (MCA) and country partners: MCA and country partners are responsible for identifying users of the evaluation, building local ownership and commitment to the evaluation, oversight of the data collection firm, and quality control of evaluation activities, including the following specific responsibilities:

1. Assess when a program is ready for evaluation planning through evaluability assessment;

2. Set the evaluation questions to achieve accountability and learning objectives;

3. Build buy-in and ownership of the evaluation;

4. Manage the data collection firms (as applicable during Compact);

5. Conduct quality reviews of all evaluation products (reports, questionnaires, etc); and

6. Facilitate public dissemination efforts.

Independent Evaluator: The Independent Evaluator is responsible for the overall design, implementation, and dissemination of the evaluation, including the following specific responsibilities:

1. Finalize evaluation questions to meet needs and demand from MCC and country partners;

2. Develop the most rigorous evaluation design feasible given rules of program implementation;

3. Support MCC and MCA to build buy-in and ownership of evaluation;

4. Develop evaluation materials that are held to international standards;

5. Ensure Institutional Review Board clearance of survey protocols;

7. Manage the data collection firms (as applicable);

6. Ensure data quality during collection and entry through supervisions and management;

7. Lead data cleaning, analysis, and interpretation of results;

8. Produce high quality, credible, transparent evaluation reports;

10. Lead preparation of public-use data that ensures appropriate balance of enabling verification of analysis and broad use of the data with adherence to promises of confidentiality to survey respondents; and

9. Lead public dissemination efforts.

Evaluation Management Process (EMP) Objectives The objectives of the EMP stem from the core lessons learned from the “First Five Ag” impact evaluations (please also reference Sturdy, Aquino, and Molyneaux 2014):

Ensure clearly defined program. This requires clearly defining the inputs, outputs, outcomes, and goals of each intervention in a way that demonstrates how the intervention will lead to expected results. For this, there must be common understanding across stakeholders on the problem diagnostic, program logic, targeted populations and their geographic locations, metrics for measuring results, and risks and assumptions.

Ensure early integration of evaluation with program design and implementation. The evaluation methodology and timeline is closely linked to program design and implementation. This requires close collaboration between M&E and Operations (both in MCC and MCA) as early as possible in the process once the essential elements of the project are defined (who will be targeted, where, how, and what are the essential project design elements), and then continuously throughout Compact development, implementation, and Post-Compact.

Ensure evaluations are structured for learning and work toward establishing a feedback loop. This requires identifying the consumers of evaluation results and ensuring a feedback loop is established so that evaluation results can inform future decision-making.

Ensure appropriate quality control mechanisms are in place. This requires making sure that the appropriate stakeholders are reviewing key evaluation deliverables, such as evaluation design reports, and assessing risks to the interventions and their evaluations, in a timely manner.

To meet these objectives, the EMP defines:

Who is involved? The MCC EMP defines who from Management, Operations, M&E, and Economic Analysis needs to review critical evaluation products/deliverables to ensure accountability and learning objectives are met.

When do they need to be involved? The MCC EMP defines the critical milestones in every evaluation life cycle when the EMC reviews and provides feedback on evaluation products/deliverables.

What are the risk factors? The MCC EMP defines the risk factors facing each evaluation at each milestone to facilitate MCC assessment as to whether the expected benefits of the evaluation are worth the costs. These risk factors often accumulate over time.

Evaluation Management Committee (EMC) An EMC is established for each independent evaluation comprised of one Chair and five or more Committee members. The EMC should be established as early in Compact Development as necessary for each Evaluation, and members should expect to participate in the EMC for the duration of the evaluation life time (5+ years). If an EMC member must leave the EMC, the Chair, in collaboration with DPE and DCO management, must identify the appropriate replacement.

Chair. The MCC M&E Managing Director (or designated representative) is responsible for decisions related to investments in the independent evaluations. Using input from EMC members, the Chair decides if the expected benefits of the evaluation outweigh the costs.

Members[footnoteRef:1] [1: Given that participation on the EMC will require staff time and is crucial to the evaluation quality assurance process, MCC should include participation on the EMC as part of Committee members’ performance plans.]

Monitoring and Evaluation (M&E) Lead. The M&E lead is responsible for overall management of the Compact’s independent evaluation activities and ensuring common understanding of the program logic(s) and evaluation questions, contracting Evaluation Firm(s), and ensuring quality control for evaluation materials and reports.

Evaluation Technical Lead. MCC has designated the following Technical Leads in each of the sectors for the evaluation portfolio. These Leads are responsible for sharing best practices in their sector(s) across different country evaluations:

· Jack Molyneaux: Agriculture & Irrigation; Transportation; Financial, Investment, and Trade; Capacity Building and Institutional Development;

· Shreena Patel: Health, Energy, and Water and Sanitation;

· Ryan Moore: Education; and

· Jennifer Lisher: Land

Economic Analysis (EA) Lead. The EA lead is responsible for assessing technical evaluation design elements, including: understanding of program logic, assumptions, evaluation questions, identification strategy, analysis plan, and alignment with the cost-benefit and beneficiary analysis.

Sector/Technical Operations Lead(s). The Sector/Technical Operations Lead(s) is responsible for defining program logic, assumptions, evaluation questions, and identifying users of evaluation results. Please note, in the Post-Compact period, the DCO Sector Practice Lead will be requested to identify appropriate Sector Lead(s) for reviewing materials Post-Compact.

Country Relations. One representative from the Country Team Lead (CTL)/Resident Country Mission (RCM) is responsible for assessing policy relevance, understanding country context, and contributing to risk assessments. In the Post-Compact period, the DCO Regional Managing Director will be asked to identify a designated representative.

In addition to above, MCC staff must always reference the Evaluation Review Matrix to confirm who should see what at each milestone. MCC M&E is responsible for documenting review dates, decisions, and feedback using tools like the Evaluation Master Tracker.

As noted below, MCA and other country partners (policymakers, Implementing Entities, implementers, other) should be given the opportunity to review and comment on key evaluation products/deliverables. Ideally, the MCC EMC will be able to consider any remaining issues and/or concerns raised by the MCA and country partners during its review of the evaluation product/deliverable. To facilitate this, MCAs are encouraged to establish their own, parallel EMCs to ensure appropriate local stakeholder review of the materials.

As per the Evaluation Firm SOW, the Independent Evaluator is responsible for documenting all feedback and their response to that feedback in an Annex of the evaluation deliverable (Evaluation Design Report, Baseline Report, Interim/Final Report).

Evaluation Milestones and Risk Factors The EMP defines several key milestones in the evaluation life cycle when the EMC meets to review and assess the independent evaluator and evaluation products/deliverables. These milestones are critical points in the evaluation life cycle for the EMC members to assess the cost-benefit of continuing, or course correcting, the independent evaluation under review. The milestones[footnoteRef:2] are: [2: Some evaluations may include one or more midline/interim studies as well. Any midline/interim reports should follow the same review and clearance procedures as endline/final reports. ]

1. Evaluability Assessment

2. Evaluation SOW

3. Evaluation Design Report

4. Baseline Materials and Data Collection

5. Baseline Report + Data

6. Endline Materials and Data Collection

7. Final Report + Data

At each milestone, it is preferred that the country partners/local stakeholders have an opportunity to review and comment prior to submission to the MCC EMC. This is because there may be remaining issues/challenges between the local stakeholders and evaluator that the MCC EMC should be aware of in its review and feedback. Parallel MCC EMC and local stakeholder review is possible, it is just not preferred as it may result in multiple MCC EMC reviews to address issues raised by local stakeholders.

The MCC M&E lead and/or Independent Evaluator will distribute or present the evaluation product/deliverable to the MCC EMC for review and comment (see standard email templates here). The MCC M&E lead is responsible for collating all comments into one comprehensive summary (see Master Tracking tool here).

Following any review/feedback with the Independent Evaluator, the MCC EMC will provide either Clearance or No Clearance on the evaluation product/deliverable. In the event of disagreement on how to proceed - as documented in the comments by MCC EMC Members and/or any remaining disagreement with local stakeholders - the Chair is the final decision maker and will confer with any relevant management/staff to reach agreement on next steps. The MCC M&E lead is responsible for documenting all related discussions and decisions. The material evidence, such as emails, memos, other documents, should be saved appropriately in S drive or Sharepoint with a clear link/reference.

In addition to the milestones defined here, the MCC M&E lead and M&E MD will also lead frequent Risk Assessments of the independent evaluation portfolio. MCC M&E Management will trigger a Risk Review round 1-2 times per year, however it is the responsibility of the MCC M&E lead to continuously monitor and assess risks to raise with Management and the EMC as necessary. These Risk Assessments may identify additional points that require an MCC EMC decision meeting, such as:

· Program re-scoping or re-structuring;

· Significant deviations from original work plan;

· Independent Evaluator contract modifications; and

· Loss of a counterfactual/Treatment group non-compliance/threats to internal validity.

The following sections detail the milestones, as well as the risk factors and objectives the EMC should consider at each milestone.

Evaluability Assessment MCC uses an Evaluability Assessment tool to determine whether or not the Project is ready for evaluation planning based on available documentation on the five core areas of any project design:

· Problem diagnostic with sufficient supporting data/evidence;

· Project objectives and theory of change/logic with sufficient supporting data/evidence;

· Project participants clearly defined and justified in terms of geographic scope and eligibility criteria;

· Metrics for measuring results for both accountability and learning clearly defined; and

· Risks and assumptions clearly defined with potential risk mitigation strategies.

The main risks facing the independent evaluation at this stage are lack of understanding of or lack of agreement on the program logic and unclear definition of project participants resulting in:

1. Incorrect definition of the Project components that should be covered by the evaluation (for example, given program logic and common target populations, El Salvador I Formal Technical Education may have been one evaluation instead of three);

2. Incorrect identification of type of evaluation that should be used, and

3. Hiring the Independent Evaluator too early resulting in unnecessary costs.

There are TWO different points for Evaluability Assessment. First, MCC staff (M&E, DCO, EA) will use the Evaluability Assessment tool to guide input into the Investment Memo for the Investment Management Committee (IMC) during Compact Development. In addition to the IM, the tool will be used by M&E staff and management to inform when the Project(s) is ready for evaluation.

Second, the Evaluability Assessment is also the first deliverable of the Independent Evaluator. At that stage, the Independent Evaluator’s Evaluability Assessment is sent to the EMC for review to ensure the Independent Evaluator has the necessary information regarding Project design and implementation to design the evaluation.

Evaluability Assessments

Evaluation SOW + IGCE Once the project is determined to be “ready” for evaluation planning, MCC M&E will take the lead on developing the Independent Evaluator SOW (see standard template here). The SOW will be developed with the following objectives in mind:

· Define which Project components (Project(s), Activities, Sub-Activities, etc) will be covered by the evaluation;

· Define Evaluation Type (Impact or Performance) if possible given understanding of Project design and implementation rules;

· Identify consumers of the evaluation results to set priority research questions and outcomes;

· Define research team qualification, with particular attention to possible evaluation methodology and sector expertise; and

· Define expected costs for evaluation through the Independent Evaluator Independent Government Cost Estimate (IGCE) and the data collection budget for MCA (example of IGCEs here).

In addition to the risks listed above, the risks facing the independent evaluation at this stage include a lack of understanding of the program logic resulting in:

1. Incomplete/inaccurate prioritization of research questions and outcomes,

2. Incomplete/inaccurate definition of research team qualifications, and

3. Underestimating the costs of the evaluation.

The EMC should review and clear on the Independent Evaluator SOW, ensuring that the above risks have been addressed and/or have a mitigation strategy, prior to submission to MCC CGM to issue the Task Order. At this stage, the EMC should also determine who will serve on the Technical Evaluation Panel (TEP) to facilitate selection of a high quality independent evaluator.[footnoteRef:3] [3: A TEP is a group that reviews proposals from bidders and rates each one against the requirements in the SOW in order to identify a preferred bidder. It is helpful to represent a range of perspectives on a TEP. At a minimum, the TEP should include the PM, COR, and Sector Lead. It can also be helpful to include the EA Lead. ]

Evaluation Design Report At this milestone, MCC M&E transfers ownership and leadership of the independent evaluation to the Independent Evaluator. The role of MCC and MCA/country partners is to provide timely feedback and comments to the Independent Evaluator to design the most rigorous, high quality, credible evaluation given project design and implementation rules. The MCC EMC will review the Evaluation Design Report (see template here) with the following objectives in mind:

· Prioritize research questions and outcomes that meet demand from key decision-makers;

· Ensure that the program Objective and all key accountability metrics modeled in the cost-benefit analysis are measured or justification is provided as to why not;

· Apply the most rigorous evaluation methodology feasible given project design and implementation rules;

· Clearly define the analysis plan to ensure consensus on outcomes – their definitions and measurement;

· Clearly define sample population and sampling strategy that aligns with project target populations;

· Clearly define exposure period that maps data collection timelines with project start date timelines; and

· Update costs as necessary.

The risks facing the independent evaluation at this stage are a lack of understanding of the program logic and program implementation resulting in:

1. Infeasible evaluation methodology (for example, proposing a randomized construction schedule for health centers to create an RCT for the evaluation, when it is not possible to randomize construction schedules given other field realities – see Lesotho Health; or randomly selecting farmers based on selection criteria that does not align with how the farmers are selected for treatment – see Honduras Farmer Training);

2. Lack of alignment between the target population and the evaluation sample population (for example, treatment groups not being treated and/or control groups receiving the program – see Namibia CBRLM);

3. Lack of agreement or understanding on required exposure period (see several examples of randomized roll-outs that did not allow for sufficient exposure period in the First Five Ag such as Ghana Farmer Training), and

4. Sample sizes that are not large enough to detect impact.

Once the Independent Evaluator finalizes the Evaluation Design Report based on MCA, country partners, and MCC EMC feedback (ensuring that the above risks are addressed and documenting the comments and responses to comments) the Independent Evaluator must prepare documentation for an entry in the MCC Evaluation Catalog to post the Evaluation Design Report, and all subsequent evaluation products/deliverables.

EMC Quality Control – Independent Evaluator Following the review and feedback round for the Evaluation Design Report or later deliverables, the MCC EMC can determine the Independent Evaluator is unable to produce a high quality, credible evaluation that meets MCC’s objectives for accountability and learning. In these cases, the MCC M&E lead must work in close coordination with CGM to cancel the contract and begin a new contract process. M&E will also produce a Cancellation Memo that is posted on the Evaluation Catalog for transparency and accountability.

Baseline Materials and Data Collection The EMC will provide review and feedback on the Baseline Materials (questionnaires, informed consent, survey protocols) with the following objectives in mind:

· Monitor and assess if project is being implemented according to plan (quantity, quality, and timing of outputs) to ensure evaluation methodology still follows project design and implementation;

· Ensure that baseline data collection begins and ends as close as possible to the start date of the Project;

· Review questionnaires to ensure there is agreement on how outcomes of interest will be measured and who will be surveyed/interviewed to report on these outcomes;

· This is informed by monitoring best practices in questionnaire design across MCC to facilitate standardization in outcome measurement across evaluations in a sector.

· Review data quality plan for training, field work management, and data entry;

· Ensure Independent Evaluator obtains all appropriate and required clearances (both local clearances and ethical review clearances);

· Review informed consent statement(s) to ensure appropriate promises of confidentiality (see Microdata Guidelines for more details)

In addition to risks raised above, the risks facing the independent evaluation at this stage are:

1. Survey instruments that do not capture the data required to answer the research questions;

2. Excessively long surveys that affect data quality;

3. Missed opportunities for standardization in outcome measurement for MCC meta-analysis in a sector; and

4. Incomplete/inaccurate definition of survey team quantity and/or quality.

Once the Baseline Materials are cleared by the MCC EMC, MCC M&E will post the Questionnaires on the MCC Evaluation Catalog.

Baseline Report + Data Within 12 months of the completion of baseline data collection[footnoteRef:4], the Independent Evaluator will produce a Baseline Report (see template here). The MCC EMC will review the Baseline Report with the following objectives in mind: [4: Some evaluations do not collect baseline data. For these evaluations, the evaluation firm should ensure the Evaluation Design Report validates the evaluation design using available trend data and other sources to justify the decision to use end-line data only.]

· Assess if evaluation is being implemented according to plan (threats to internal validity if an impact evaluation; appropriate sample sizes and response rates);

· Monitor and assess if project is being implemented according to plan (quantity, quality, and timing of outputs);

· Monitor and assess if project risks have materialized and if assumptions are holding;

· Ensure appropriate management of data (see Microdata Guidelines for more details)

In addition to risks raised above, the risks facing the independent evaluation at this stage are:

1. Treatment and control groups that are not appropriately balanced (for example consumption, income, etc.);

2. Significantly different context that can affect expected results (for example, if an outcome of interest in the cost-benefit analysis (CBA) for the ERR is incidence of child diarrhea, but the baseline data shows low to no incidence of diarrhea, this should be raised as a results risk for management and trigger a new CBA, particularly if this outcome is being used to drive evaluation sample sizes – see El Salvador I WASH);

3. High non-response rates for a particular survey instrument (more than typical levels); and

4. Problems with data quality (missing values and outliers).

Once the Baseline Report is cleared by the MCC EMC, MCC M&E will post on the MCC Evaluation Catalog.

Public Release of Baseline Data Only Following the Microdata Management Guidelines, de-identification and preparation of baseline data only should be based on specific demands for the data. Unless there is explicit demand for the public-use baseline data, the Independent Evaluator should prepare the data package with the Data De-Identification Worksheet for documentation purposes, but will NOT conduct the de-identification and DRB review process until the Final Data Package.

Endline Materials and Data Collection The EMC will provide review and feedback on the Endline Materials (questionnaires, informed consent, survey protocols) with the following objectives in mind:

· Assess if evaluation is being implemented according to plan (threats to internal validity if an impact evaluation; appropriate sample sizes and response rates);

· Monitor and assess if project is being implemented according to plan (quantity, quality, and timing of outputs);

· Ensure that endline data collection begins AFTER the minimum agreed exposure period – i.e. the time required for the target population to be exposed to the program to produce the expected improvements/impact on outcomes.

· Review questionnaires/survey instruments to ensure there is agreement on how outcomes of interest will be measured and who will be surveyed/interviewed to report on these outcomes;

· Review data quality plan for training, field work management, and data entry;

· Ensure Independent Evaluator obtains all appropriate and required clearances (both local clearances and ethical review clearances); and

· Review informed consent statement(s) to ensure appropriate promises of confidentiality (see Microdata Guidelines for more details)

At this point, the evaluation needs to be monitored for all risks listed.

Once final Endline Questionnaires are cleared by the MCC EMC, MCC M&E will post on the MCC Evaluation Catalog.

Final Evaluation Report + Data Within 12 months of the completion of endline data collection, the Independent Evaluator will produce a Final Report. MCC has an expanded five-step review process to manage the public release of Independent Evaluator reports that present results, whether Interim and/or Final.

In addition to all previously listed risks, an additional risk to a high quality, credible evaluation to be considered when reviewing the Final Report:

1. Error or bias in analysis and/or interpretation of results. (Please reference Honduras Farmer Training Management Response and other documentation related to concerns with evaluation quality).

Step 1 – Country Partner(s) Review. As with all previous key evaluation products/deliverables, the Independent Evaluator is responsible for sharing the Evaluation (Interim and/or Final) Report(s) with the MCA and country partners for review and comment prior to formal submission to the MCC EMC. This may be done virtually or through a local stakeholder workshop. The purpose of this review is to ensure the Independent Evaluator has addressed any technical or factual errors in the report. For country stakeholders, during the Compact period this should include the MCA CEO, MCA Sector Lead and MCA M&E lead, implementers, and Implementing Entities. In Post Compact period, the MCA should designate appropriate representatives to review the report. Special consideration should be made on who these designated representatives are in order to review the report for technical and factual accuracy and they should be identified in the Post-Compact M&E Plan. The MCA and any other necessary local stakeholders will provide written feedback that will be publicly documented in an Annex of the Report. The Independent Evaluator will provide written response to any comments and may elect not to take comments into consideration for the Final Report. Any key stakeholder will then be given the opportunity to provide a public “Statement of Difference or Support” based on the evaluation firm’s report and response to feedback. The submission of the Evaluation Report(s) to MCC will include the following:

· Annex 1: Documentation of the comments/feedback from country stakeholders.

· Annex 2: Any “Statement of Difference or Support” from country stakeholders.

Step 2 – MCC EMC Review. As with all previous key evaluation deliverables, the Independent Evaluator will submit the Final Report to the EMC to review for (i) completeness, (ii) technical rigor, (iii) policy relevance, and (iv) appropriate dissemination activities.. The EMC will ensure that DPE and DCO management is invited to participate in the review. If any factual or technical errors are found in the report, written comments will be submitted to the Independent Evaluator for consideration. In addition, the EMC may elect to request an external peer review of the Evaluation Report, in which case peer reviewer comments will also be provided to the Evaluation Firm (see template for Peer Reviews here). The MCC M&E lead is responsible for collating all comments into one comprehensive summary. The Independent Evaluator will provide written response to any comments and may elect not to take comments into consideration for the Final Report. These comments and responses must be documented by the Independent Evaluator and submitted with the Final Report as part of the deliverables package.

· Annex 3: Documentation of the comments/feedback from MCC stakeholders.

At this stage, the Final Report is complete and will not be altered. During this stage, MCC M&E will also draft the MCC Management Response[footnoteRef:5] and Summary of Findings[footnoteRef:6] for the evaluation to accompany the public dissemination of the Evaluation Report. These drafts should be shared with the EMC prior to Step 3. [5: The MCC Response is an official MCC management statement that confirms MCC acceptance of the final report and documents any outstanding differences between MCC and the Evaluation Firm as it relates to (i) factual and/or (ii) technical issues.] [6: The MCC Summary of Findings places the evaluated program in context of the overall Compact, summarizes the program logic, assumptions, monitoring indicators and results, as well as evaluation questions and key results. The Summary of Findings also summarizes key lessons learned by MCC resulting from the evaluation findings.]

Step 3 – MCC submits package for final Country Review. Next, MCC M&E will send the following to the MCA, or designated representative: (i) Final Report with Annex 1-3, (ii) MCC Summary of Findings, and (iii) Draft MCC Management Response. The MCA will review and have the option to provide a public Official Country Response[footnoteRef:7] on the Final Report, if necessary. [7: The Official Country Response is an official host-country statement that either confirms their acceptance of the evaluation findings or documents any outstanding differences between the host country and the Evaluation firm as it relates to (i) factual and/or (ii) technical issues.]

Step 4 – Final submission to MCC VP Review. The MCC M&E Managing Director, or designated representative, will send the following to MCC VPs for DCO, DPE, OGC, CPA, as well as the Chief of Staff: (i) Final Report with Annex 1-3, (ii) MCC Summary of Findings, (iii) Draft MCC Management Response, and (iv) Official Country Response (if provided). MCC VPs will finalize the Official MCC Response Statement to be posted with the public Final Report.

Step 5 – Public Dissemination. Following the formal review of the Final Report with Annex 1-3, MCC Summary of Findings, Country Response and MCC Response, the documents will be publicly posted on the MCC Evaluation Catalog.

Public Release of Complete Data Package Following the Microdata Management Guidelines, de-identification and preparation of the Complete Data Package should be developed within 6 months of the completion of the Final Report.

MCC Independent Evaluations Evaluation Management Process (EMP) Version: July 2017

Summary of Risks over Evaluation Life Cycle

Incomplete/inaccurate definition of research team qualifications

Underestimating the costs of the evaluation

Infeasible evaluation methodology

Lack of program implementation fidelity

Lack of alignment between the target population and the evaluation sample population

Lack of agreement or understanding on required exposure period

Sample sizes that are not large enough to detect impact

Survey instruments that do not capture the data required to answer the research questions

Missed opportunities for standardization in outcome measurement for MCC meta-analysis in a sector

Incomplete/inaccurate definition of survey team quantity and/or quality

Treatment and control groups that are not appropriately balanced

Significantly different context that can affect expected results

High non-response rates for a particular survey instrument

Problems with data quality

Error or bias in analysis and/or interpretation of results

X