J.5._Evaluation_Design_Report_Template.docx

MCC Independent Evaluation Design Report[footnoteRef:1] Outline Template (July 2017) [1: As noted in the evaluator SOW, the EDR should be updated to reflect significant changes in design or to add more detailed design information. There is no set frequency for these updates, but they should be discussed with the PM.]

LIST OF ACRONYMS

TABLE OF CONTENTS

1. INTRODUCTION & BACKGROUND

A. Country context – No more than a paragraph or two, to set the stage B. Objectives of this report

2. OVERVIEW OF THE COMPACT AND THE INTERVENTIONS EVALUATED

A. Overview of the project and implementation plan

i. Original project description – Can be pulled from MCC M&E Plan, but should be elaborated or updated as necessary, for example with information from contractors covering implementation. If activities within a project are evaluated separately, or only one activity is evaluated and the others are not, this should be clear.

ii. Project participants - Who is targeted and why, for each activity evaluated

iii. Geographic coverage - Where activities are/were implemented and why

iv. Description of implementation to date - How was the project implemented so far, any changes/re-scoping. This is only an overview, you are not required to report any monitoring indicators. If implementation has not yet begun, this can be noted here.

B. Theory of Change - This should include the key outputs for all activities in all projects in the program being evaluated, and how they lead to expected outcomes. If relevant, include the full Compact logic (all programs) as well. Only a description of the logic as laid out in the design is needed; an assessment will come in a separate report.

C. Cost Benefit Analysis & Beneficiary Analysis - Briefly describe the analyses conducted by MCC, highlighting the key parameters and linking to the theory of change and key outcomes, as relevant.

D. Literature Review

i. Summary of the existing evidence

ii. Gaps in literature

iii. Policy relevance of the evaluation

3. EVALUATION DESIGN

A. Evaluation questions – Include a justification for why each question is included/important and link to the program logic. These questions should be numbered, and linked clearly in the methodology sections below. Research Question #1 must always address the fidelity of implementation to the planned intervention and the quality of implementation (i.e. process study).

B. Evaluation design overview – This is a description of the overall approach to the evaluation, framing of different approaches that will be employed, and how they complement each other. Mixed methods evaluations should pay particular attention to this section, to ensure adequate explanation of how it all “fits together” before elaborating on the various methods. In addition, note the parameters of the CBA for which data will or will not be collected and why. Include a summary table following this template:

C. Quantitative approach

i. Methodology – State the specific evaluation methodology/approach that will be used (e.g. experimental, quasi-experimental, pre-post, etc.). Include reference to the research questions that will be addressed with each method and why. Specify how MCA monitoring data/implementation information will be used to address Research Question #1 (implementation), and subsequently inform the evaluation (if at all). Be sure to specify the strategy for identifying treatment and comparison groups, where applicable.

ii. Timeframe of exposure - Include justification for survey timing relating back to program logic and when impacts are anticipated to take effect.

iii. Study Sample

1. Sample unit(s) – Specify the unit at which the study will target (e.g. households, firms, schools etc). Include any level of clustering if applicable.

2. Sample size and associated assumptions, including power calculations

3. Sample frame – Describe whether sufficient data for the frame exists from another source, or will need to be collected as part of the evaluation. If a listing is required, detail the methodology for doing so, including: key questions, approximate listing size, required respondent, surveyors, data processing and data quality.

4. Sampling strategy – Include any stratification decisions and the justification, and specify the level of representation of the sample. Specify whether sampling will be done in real time in the field, or in advance of data collection.

iv. Primary data collection

1. Instruments – Specify key modules for each round, measurement of variables of interest, and whether the collection will be paper or electronic.

2. Rounds and timing – Describe how the timing links to the timeframe of exposure and the expected outcomes to be explored at each round, based on the program logic.

3. Respondent(s) within the sample unit – Specify the target respondent, and strategy for missing/absent respondents both initially and at subsequent data collection rounds.

4. Staff – Note key qualifications specific to this evaluation that will be required of survey firm, supervisors, and enumerators (e.g. language/cultural issues or sector-specific knowledge).

5. Data processing – For electronic data collection, describe the devices, application, data flow and management. For paper data collection, describe the entry template and entry protocol.

6. Data quality – Describe the processes that will be used to ensure quality data is collected, and the protocol for fixing any errors caught.

7. Summary table – Summarize the rounds of data collection, timing, respondents, sample size, sample units and relevant instruments/modules in a table.

Respondent

v. Secondary data– please list the sources of secondary data that will be used to support the evaluation and describe their relevance for the analysis. Note any requirements related to data capture, i.e. Will the data be transferred electronically, or will it require data entry? If so, detail the methodology, including a description of the entry template and protocol. In addition, please describe how the quality of the data will be assessed prior to using.

vi. Analysis plan – Include model specifications and possible variations that will be explored.

D. Qualitative approach

i. Methodology (e.g. case studies, ethnography, series of interviews, observations, etc.) - Include reference to the research questions that will be addressed with each method, and why.

ii. Timeframe of exposure - Include justification for survey timing relating back to program logic and when impacts are anticipated to take effect.

iii. Study Sample

1. Sample unit(s) – Specify the unit at which the study will target (e.g. households, firms, schools etc). Include any level of clustering if applicable.

2. Sample size and associated assumptions and calculations

3. Sample frame – Describe whether the data for the frame exists from another source, or will need to be collected as part of the evaluation.

4. Sampling strategy – Specify how you find people and why. Include any stratification decisions and the justification, and specify the level of representation of the sample.

iv. Primary data collection

1. Instruments/protocol/interview guide – Specify key modules or questions for each round

2. Rounds and timing – Describe how the timing links to the timeframe of exposure and the expected outcomes to be explored at each round, based on the program logic.

3. Respondent(s) within the sample unit – Specify strategy for missing/absent respondents both initially and at subsequent data collection rounds.

4. Staff – Note key qualifications specific to this evaluation that will be required of survey firm, supervisors, and enumerators (e.g. language/cultural issues or sector-specific knowledge).

5. Data processing – Explain how the data will be transcribed and translated.

6. Data quality – Describe the processes that will be used to ensure quality data is collected, translated and transcribed, and the protocol for fixing any errors caught

7. Summary table – Summarize the rounds of data collection, timing, respondents, sample size, sample units and relevant instruments/modules in a table.

Respondent

v. Analysis plan

1. Coding – Describe the approach and the initial scheme, and triangulation strategies

2. Analysis method/framework E. Challenges

i. Limitations of interpretations of the results

ii. Risks to the study design – For all evaluations, include strategies to monitor key risks/assumptions in the program logic and evaluation design, or other risks related to implementation. For RCTs in particular, include strategies to minimize problems with randomization compliance and strategies to deal with deviation from assignment status F. Summary of Methodology to Estimate Post-compact Economic Rate of Return (ERR)

i. Specify which key parameters in the MCC CBA are measured as part of the proposed evaluation design

ii. Proposed CBA methodology to estimate the project’s post-compact ERR

4. ADMINISTRATIVE

A. Summary of IRB requirements and clearances B. Data Protection – Summarize processes for respecting privacy of respondents during data collection, transfer, storage, analysis, disposal and dissemination, including informed consent, password protection, encryption/access protocols, and any other boundaries deemed necessary to protect the data based on an assessment of its sensitivity.

C. Preparing data files for access, privacy and documentation – Address the viability of producing public use data for replicability and/or the exploration of new research questions. Discuss any strategies that will be employed during data collection and data cleaning to achieve these results. Include an estimate of LOE for this task.

D. Dissemination Plan E. Evaluation Team Roles and Responsibilities F. Table: Evaluation timeline & Reporting schedule – Specify the timing in a table, as per the template below. The naming of the data collection rounds can be customized to the evaluation. However, please notate (*) the round of data collection that is timed/expected to measure key impacts on the project objectives stated in the compact.

5. REFERENCES

6. ANNEXES

A. Stakeholder Comments and Evaluator Responses B. Evaluation Budget - Given sensitivities around future procurements, the budget should not be a public document. However, the evaluator and MCC M&E should have a clearly defined budget for the evaluation, including planned evaluator LOE/ODC and direct data collection costs (i.e. survey firm sub-contract or individual data collection contractors,) which should be tracked over time with actual evaluator and data collection costs. The evaluator must provide an estimate for data collection costs for all rounds of data proposed, and should note and justify if these costs differ from what was originally proposed in the budget at contract award.