Annex_B_Evaluability_Assessment_draft.pdf

Project Evaluability Assessment (PEA)

Background As a results-based institution, MCC is dedicated to using rigorous methods to identify the core constraints to growth and design and implement evidence-based Projects in every Compact and Threshold country to address those core constraints.

For this reason, the concept of evaluability is central to MCC’s results agenda, as it is impossible to determine the results achieved by MCC projects unless they are designed up-front to be evaluable downstream. MCC defines evaluability as the ability of an intervention to demonstrate in measurable terms the results it intends to deliver. An evaluable intervention uses data to identify and verify the problem(s) it intends to address and underlying causes, has an evidence-based design to address the problem(s) identified, clearly identifies assumptions and risks associated with the intervention to address the problem(s), including identifying means for verifying and mitigating risks, and has clear and time-bound metrics for the results of the intervention in terms of outputs and outcomes.

Objective The objective of an evaluability assessment is to use specific, transparent standards and best practices for assessing the following five dimensions of a project: (i) Problem Diagnostic, (ii) Project Objectives and Logic, (ii) Risks and Assumptions, (iii) Project Participants/Beneficiaries, and (v) Accountability and Learning Metrics.

Evaluability in Practice A tool has been developed to help guide the iterative process of evaluability assessment.1 The Project Evaluability Assessment tool draws on and expands on the standards and best practices identified in the MCC Due Diligence Book in order to assess the five dimensions of a project through Compact development and into Compact implementation. The tool is intended to help guide teams to answer five big questions:

• Is the problem clearly defined and is there sufficient evidence to support the problem diagnostic?

• Are the project objectives and theory of change/logic clearly defined?

• Are the risks and assumptions clearly defined with potential risk mitigation strategies?

• Are project participants clearly defined and justified in terms of geographic scope and eligibility criteria?

• Are the metrics for measuring results for both accountability and learning clearly defined?

In each of the five dimensions, the evaluability assessment tool lists a series of questions based on standards and best practices. While a Project may not meet all of the standards and best practices, and it is possible that the list of questions does not capture all potential factors that should be flagged, the purpose is to highlight the strengths and weaknesses of each intervention so that they can be addressed prior to Compact signature, in the development of the supplemental agreements, and during implementation as appropriate.

The tool may be applied throughout Compact development and implementation, and it is important to note that the assessment is intended to clarify and improve project documentation. For this reason, the assessment is only conducted on specific project documents as they are developed: (i) Concept Notes, (ii) Concept Papers, (iii) Investment Memo, (iv) Other Project and Evaluation Design Document (s).

1 The evaluability assessment tool was developed through a yearlong pilot from 2012-2013, tested by the MCC M&E Team on the following 7 Compact and Threshold programs: Threshold (Honduras); Implementation (Cape Verde II and Lesotho); Compact Development (El Salvador II, Georgia II, Ghana II, Benin II).

The current proposed process for assessment of project evaluability is intended to (i) maximize integration of the evaluability assessment within the Country Team, and (ii) ensure quasi-independent assessment for quality assurance. The proposed process is:

1. Concept Note: Country team members submit Project(s) Evaluability Memo to Country Team Lead with a cc to respective MDs.

2. Concept Paper: Country team members submit Project(s) Evaluability Memo to Country Team Lead with a cc to respective MDs.

3. Investment Memo: Country M&E lead works with Country Team to produce Investment Memo, reflective of current state of Project(s) Evaluability. Once submitted, an independent peer review of the IM, using the evaluability assessment tool, will be conducted. The composition and timing of this peer review is TBD.

At each stage, the intention is to provide feedback to the Country Team and Management on the current status of the Project(s) which is one of several inputs into project and evaluation selection, design, and implementation decisions.

Project2 – Evaluability Assessment Assessment Categories Strengths, Weaknesses, and Next Steps

Dimension 1: Is the problem clearly defined and is there sufficient evidence to support the problem diagnostic?

Standards and best practices

1. Is there quantitative evidence regarding constraints to and sources of economic growth?

2. Is the problem(s) clearly defined and understood with sufficient evidence/quantitative (baseline) data available to support claims?

3. Is there evidence to support root causes identified?

4. Will all root causes be addressed by the proposed intervention or other, complementary intervention(s)?

5. Is there a public good rationale and/or market failure that necessitates government intervention and funding? This is linked to sustainability – how will recurrent costs be covered in the future, how will private sector investment be triggered, etc?

6. Is the institutional context understood, the political economy understood, and does the team clearly understand how the proposed intervention(s) link to other initiatives by the gov’t and/or other partners?

7. Is there a clear understanding on how different social and cultural dynamics (gender, poverty, race, ethnicity, etc.)

may be influenced by or influence the problem identified?

Strengths:

Weaknesses:

Next steps:

Dimension 2: Are the project objectives and theory of change/logic clearly defined?

Standards and best practices

1. Is the objective of the Project clearly stated with a link from the problem diagnostic?

2. Is the project logic and economic model clear, plausible and based on existing evidence and literature? If there is limited evidence, is there commitment to generating evidence via an impact evaluation (linked to Section 5)?

3. Is there a clear logic that links different projects within the Compact program that is then linked to accelerating economic growth?

4. Is it clear which component(s) of the problem diagnostic will be addressed by the proposed intervention, which will be addressed by complementary activities and which will remain risks for the MCC intervention to achieve proposed results?

5. Are the inputs, outputs, outcomes clearly defined and linked to the economic analysis (ERRs)?

6. Is the timeline for expected results clear and based on evidence?

7. Is it clear whether or not benefits are expected to be

2 It is anticipated that an evaluability assessment will be conducted for each Project; however this may vary depending on Compact composition. For example, the country team may prefer to conduct the evaluability assessment for individual Activities rather than Projects.

sustained beyond the life of the compact?

Dimension 3: Are the risks and assumptions clearly defined with potential risk mitigation strategies?

Standards and best practices

1. Are the risks to achieving expected results clear, with clearly defined risk mitigation strategies?

2. Does the ERR reflect these assumptions and risks? Has sensitivity analysis been used to select key risks and assumptions?

3. Is it clear how risks will be monitored?

4. Is it clear how design and implementation may be altered as information on new risks/realization of risks occurs?

5. Does the project team make a critical assessment of the degree to which there may be blind spots or unknown unknowns in a project of this nature (e.g. how foreseeable are the potential risks that may arise in new sectors)?

Dimension 4: Are project participants clearly defined and justified in terms of geographic scope and eligibility criteria?

Standards and best practices

1. Is the selection criteria for project participants clearly defined and based on the problem and evidence in the program logic?

2. Is program participants’ selection based on credible, quantifiable selection criteria?

3. Are specific demographics (age, gender, poverty status) defined where necessary?

4. Are the geographic location(s) for the Project defined and based on the problem listed above and evidence in the program logic?

5. Will the Project design and implementation plan vary by different sub-groups and/or geographic locations based on the problem listed above and evidence in the program logic?

6. Can the selection be replicated for the purposes of an impact evaluation (linked with Section 5)?

Dimension 5: Are the metrics for measuring results for both accountability and learning clearly defined?

Standards and best practices Are there clearly defined indicators and data sources identified for monitoring project implementation?

1. Are there clearly defined indicators for measuring expected performance (processes, outputs)?

2. Are the indicators linked to the ERR?

3. Is it clear which indicators will be disaggregated by gender, age, income as appropriate?

4. Is there a clear understanding of the time frame for expected results of each indicator (if varies)?

5. Is there sufficient information to set appropriate and feasible baseline and annual/quarterly targets?

6. Are there sufficient human and financial resources in the MCA and IEs to conduct the necessary data collection/reporting during the life of the intervention?

Are data collection costs known and budgeted for?

7. Is it clear who will use the data and for what purpose(s)?

Are there clearly defined indicators and data sources identified for monitoring project results?

1. Are there clearly defined indicators for measuring expected performance (processes, outputs)?

2. Are the indicators linked to the ERR?

3. Is it clear which indicators will be disaggregated by gender, age, income as appropriate?

4. Is there a clear understanding of the time frame for expected results of each indicator (if varies)?

5. Is there sufficient information to set appropriate and feasible baseline and annual/quarterly targets?

6. Are there sufficient human and financial resources in the

MCA and IEs to conduct necessary data collection/reporting during the life of the intervention?

Are data collection costs known and budgeted for?

7. Is it clear who will use the data and for what purpose(s)?

Is the evaluation clearly defined for maximizing learning and accountability?

1. Is there commitment by all key stakeholders to implement the independent evaluation?

2. Are evaluation questions and outcomes clearly defined and prioritized?

3. Is it clear which outcomes will be disaggregated by gender, age, income as appropriate?

4. Is it clear who will use the evaluation results and for what purpose(s)?

5. Is the evaluation methodology the most rigorous and feasible possible?

6. Is it clear how an evaluation (performance or impact) will contribute to the evidence base in the sector?

7. Are there interim/continuous evaluation results which could help inform decisions during the compact life? If so, is such an evaluation built into the evaluation plan?

8. Do the potential benefits and learning from an evaluation of the program outweigh the costs?

9. Are there sufficient human and financial resources in the MCC, MCA and IEs to conduct necessary data collection/reporting during the life of the evaluation? Are data collection costs known and budgeted for?