ATTACHMENT_J-4.pdf

RFP 294-17-00000 ATTACHMENT J-4

ATTACHMENT J-4

Evaluation Statement of Work (SOW) Requirements

A. An evaluation Statement of Work (SOW) provides the framework for an evaluation and communicates the evaluation questions. An evaluation SOW is needed to implement evaluations through external entities and also serves to guide internal evaluations with modifications as appropriate.

A well-written evaluation SOW must:

1. Identify the evaluation as an impact or performance evaluation per the definitions in ADS 201 (https://www.usaid.gov/ads/policy/200/201)

2. Describe the specific strategy, project, activity, or intervention to be evaluated, including appropriate award numbers, award dates, funding levels, and implementing partners.

3. Provide a brief background on the country and/or sector context; specific problem or opportunity the intervention addresses; and the development hypothesis, theory of change, or simply how the intervention addresses the problem.

4. Identify existing and relevant strategy, project, or activity documents or performance information sources that are available, with special attention to monitoring data.

5. State the purpose of, audience for, and anticipated use(s) of the evaluation.

6. Identify a small number of proposed evaluation questions (1–5 questions are recommended) that are answerable with empirical evidence and relevant to future programmatic decisions or learning.

7. Identify all evaluation questions requiring sex-disaggregated data, the use of gender-sensitive data collection methods, and analysis of differential impacts on males and females.

8. Specify proposed data collection and analysis method(s) or request that prospective evaluators propose data collection and analysis method(s) that will generate the highest quality and most credible evidence on each evaluation question—taking time, budget, and other practical considerations into account.

9. In an impact evaluation, identify questions about measuring the change in specific outcomes attributable to a specific USAID intervention.

10. In an impact evaluation, require specific experimental or quasi-experimental methods or request the prospective evaluators to propose experimental or quasi-experimental methods.

11. Specify the evaluation deliverable(s) and their timelines and logistics including (at minimum):

• A written design that includes key questions, methods, main features of data collection instruments, and data analysis plan;

• A draft report;

• A final report that includes all elements described in ADS 201mah, USAID Evaluation

Report Requirements (https://www.usaid.gov/ads/policy/200/201mah); and

• Datasets—and supporting documentation such as code books, data dictionaries, scope, and methodology used to collect and analyze the data—to be submitted to the USAID Development Data Library.

12. Clarify requirements for reporting and dissemination.

13. Include the Criteria to Ensure the Quality of the Evaluation Report

(https://www.usaid.gov/ads/policy/200/201maa) to communicate USAID’s quality criteria to evaluators.

14. Clarify expectations about the methodological and subject matter expertise and composition of the evaluation team, including expectations concerning the involvement of local evaluation team members and evaluation specialists.

15. Require evaluation team members to provide a written disclosure of conflicts of interest (COI) and require key personnel to submit their COI disclosure with the proposal.

16. Describe intended participation of USAID staff, implementing partners, national counterparts, or beneficiaries in the design or conduct of the evaluation.

17. Specify the expected period of performance.

18. Address scheduling, logistics, security requirements, and other support.

19. Include illustrative information about level of effort expected.

20. Be accompanied by an independent government cost estimate (if applicable).

https://www.usaid.gov/ads/policy/200/201maa

B. EVALUATION STATEMENT OF WORK TEMPLATE

Evaluation SOW must include the following sections:

1. Activity Description

Describe the specific intervention, project/program, or process to be evaluated; include information on the evaluation scope (single project, multi-project, or multi-country), target group(s) and location(s) (urban/rural, countries, districts), project start/end dates, funding levels and sources, implementing partners, sectors/topics, USAID Technical Office(s) and A/COR(s).

2. Development Hypothesis and its Implementation

Provide a brief background on the development hypothesis and its implementation. If the program/project has a Results Framework or Logical Framework, include this framework or a summary of the hypothesis involved using clear if-then language. For impact evaluations be sure to identify the specific intervention(s) to be examined and the effects/outcome indicator changes those interventions are expected to produce.

Include a brief description of the current implementation status and any significant changes (to context or activity) that may have impacted implementation (budget changes, design modifications, results achieved/not achieved).

3. Existing Performance Information Sources

Identify existing performance information sources, with special attention to monitoring data. Explain when and how this information will be made available to the evaluation team. Include both activity and USAID specific documents as well as additional secondary sources that will help the evaluation team understand the context in which the activity functions and any changes to this context since activity start.

4. Evaluation Purpose, Audience and Intended Uses

State the purpose of, audience for, and anticipated use(s) of the evaluation. Focus on the management purpose of the evaluation, such as the project management or forward planning decisions it will support.

USAID Evaluation Policy (pg 3 and 8 at http://www.usaid.gov/evaluation/policy) and ADS 203.3.1.3 (pg 8- 9 at http://www.usaid.gov/ads/policy/200/203) provide reasons or additional triggers for conducting an evaluation. Provide a specific purpose that explicitly links the evaluation to future decisions made by USAID leadership, partner governments, and/or key stakeholders on future activities, policy direction, etc.

5. Evaluation Questions

Identify a small number of evaluation questions that are relevant to future decisions and answerable with empirical evidence. Count your question marks when determining whether you have adhered to the small number guidance. Avoid compound questions. List questions in priority order. USAID’s How-To Note on Evaluation Statements of Work suggests identifying 3-5 questions. For impact evaluations highlight questions that involve causality/attribution, e.g., what effect did the intervention have on specific intended outcomes?

6. Gender Disaggregation and Gender Differential Effects

Identify all evaluation questions for which gender-disaggregated data are expected; also identify questions for which an examination of gender specific or gender differential effects are expected. Consider whether gender differences would be evident in access, participation, benefits to individuals, or other type of results.

Identify expectations involving sex disaggregated data and gender specific effects on a question by question basis.

7. Evaluation Design/Methods

Identify evaluation method(s) that will generate the highest quality and most credible evidence on each evaluation question, taking time, budget, and other practical considerations into account and specify methods with sufficient detail. Be transparent about which methods are associated with each question.

Include any expectations about specific methods or the scale of data collection efforts, including sampling procedures, where appropriate, e.g., random sample, quota or criteria sample. For impact evaluations, indicate what type of design is anticipated, e.g., experimental, quasi- experimental (or a specific quasi-experimental design or technique) and any expectations about treatment/comparison group sizes, power analysis.

8. Data Analysis

Describe how data collected on evaluation questions will be analyzed. Consider what types of descriptive statistics or analysis of narrative data are desired. Be clear about which analysis techniques are associated with each method and question. For impact evaluations, indicate how differences between groups will be determined.

9. Strengths and Limitations

Describe strengths and limitations of the evaluation methods. Be specific about the relationship between known or anticipated limitations of the design/methods and how they might affect the quality/credibility of answers to evaluation questions. Indicate that the evaluation team is expected to identify how they will minimize the impact of, or otherwise address, any limitations.

10. Evaluation Deliverables

Specify the evaluation deliverable(s) and their timelines and logistics, including requirements for the transfer of data to USAID and expectations concerning evaluation team involvement in the dissemination of evaluation results. Notify teams about USAID’s expectations for the delivery of study data, code books, transcripts, and how each should be transmitted, e.g., flash drive, CDs.

11. Team Composition

Clarify expectations about the methodological and subject matter expertise and composition of the evaluation team, including expectations concerning the involvement of local evaluation team members (one team member should be an evaluation specialist). All team members are required to provide a signed statement attesting that they have no conflict of interest, or describing an existing conflict of interest.

12. Participation of USAID Staff and Partners

Describe intended participation of USAID staff, implementing partners, national counterparts or customer/beneficiaries in the design or conduct of the evaluation. Who, how and when?

13. Scheduling and Logistics

Address scheduling, logistics and other support. Include any security issues or procedures that may apply.

Identify local holidays, fixed dates, weather and other factors that have implications for the evaluation plan and schedule.

13. Reporting Requirements

Clarify requirements for reporting and dissemination, including mandatory inclusion of Appendix 1 of the Mandatory Reference on Evaluation. Specify the sections required in the report, expected number of pages, and any formatting and branding requirements (including reference to the Evaluation Report Template -http://usaidlearninglab.org/library/evaluation-report-template and How-To Note Preparing Evaluation Reports - http://usaidlearninglab.org/library/how-note- preparing-evaluation-reports).

[Appendix 1 is provided below and will automatically appear at the end of this section of the Statement of Work you develop using this template.]

Evaluation Policy: Appendix 1

Criteria to Ensure the Quality of the Evaluation Report

• The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.

• Evaluation reports shall address all evaluation questions included in the scope of work.

• The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer.

• Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report.

• Evaluation findings will assess outcomes and impact on males and females.

• Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).

• Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence.

• Sources of information need to be properly identified and listed in an annex.

• Recommendations need to be supported by a specific set of findings.

• Recommendations should be action-oriented, practical and specific, with defined responsibility for the action.

14. Budget

Include a budget. This can be include a dollar value or an indication of the level of effort expected, by task (initial document review, planning/preparation, field work, data analysis, report writing) and/or type of evaluation personnel (team leader/evaluation specialist/technical specialist/statistician).

C. Evaluation Statement of Work ( SOW) Checklist

Keyed to USAID ADS 203.3.1.5 (11/2/12) and USAID How-To Note Evaluation Statements of Work

Checklist Elements Keyed to Suggested SOW Outline Sections Yes No Section 1: Activity Description 2

Describe the specific intervention, project/program, or process to be evaluated

1. Is the SOW clear and specific about what is to be evaluated?

Section 2: Development Hypothesis and Its Implementation Provide a brief background on the development hypothesis and its implementation

2. Does the SOW state the development hypothesis (or theory of change) for the activity/intervention using clear if-then logic/language in a narrative, Results Framework or Logical Framework format?

3. Does the SOW describe the implementation status of the activity?

Section 3: Existing Performance Information Sources Identify existing performance information sources, with special attention to monitoring data

4. Does the SOW identify specific existing performance information sources?

5. Are indicated documents provided with the SOW or does the SOW specify how and when the documents/data will be provided to the evaluation team?

6. Does the SOW describe other secondary sources of information that would be useful to the evaluation team?

Section 4: Evaluation Purpose, Audience and Intended Use State the purpose of, audience for, and anticipated use(s) of the evaluation

7. Is the SOW clear and specific about why, in management terms, the evaluation is being conducted, i.e., what management decisions an evaluation at this time will inform?

8. Does the SOW indicate who makes up the audience(s) for the evaluation and how they plan to use it?

Section 5: Evaluation Questions Identify a small number of evaluation questions that are relevant to future decisions and answerable with empirical evidence

9. Does the SOW identify a small number of specific questions to be answered?

10. Please enter the number of visible question marks you see in the box provided.

11. Does the SOW indicate the relative priority of each evaluation question?

Section 6: Gender Disaggregation and Gender Differential Effects Identify all evaluation questions for which gender-disaggregated data are expected; also identify questions for which an examination of gender specific or gender differential effects are expected

12. Does the SOW identify questions for which data is expected to be disaggregated by sex (M/F)?

13. Does the SOW identify questions for which an examination of gender specific or gender differential effects are expected?

2 The term activity is used in this checklist to refer to an activity, project or program that will be evaluated.

Section 7: Evaluation Design/Methods Identify evaluation method(s) that will generate the highest quality and most credible evidence on each evaluation question, taking time, budget, and other practical considerations into account and specify methods with sufficient detail

14. Does the SOW recommend specific design/methods for conducting the evaluation?

15. Does the SOW recommend data collection methods on a question-by-question basis?

16. Does the SOW state clear requirements for sampling stratification and/or the presence of a sampling plan?

Section 8: Data Analysis Describe how data collected on evaluation questions will be analyzed

17. Does the SOW recommend specific methods for data analysis?

18. Does the SOW recommend an analysis method for each evaluation question or type of information?

Section 9: Strengths and Limitations Describe strengths and limitations of the evaluation methods

19. Does the SOW identify known or anticipated strengths and limitations of methods proposed?

Section 10: Evaluation Deliverables Specify the evaluation deliverable(s) and their timelines and logistics, including requirements for the transfer of data to USAID and expectations concerning evaluation team involvement in the dissemination of evaluation results

20. Does each deliverable listed specify requirements including a timeline?

21. Does the SOW explain how deliverables (including data) will be transferred to USAID?

Section 11: Team Composition Clarify expectations about the methodological and subject matter expertise and composition of the evaluation team, including expectations concerning the involvement of local evaluation team members (one team member should be an evaluation specialist)

22. Does the SOW specify the size of the evaluation team required and qualifications/responsibilities for specific positions?

23. Is the SOW explicit about requiring that one team member be an evaluation specialist?

24. Does the SOW describe expectations concerning the involvement of local evaluation team members?

25. Is the SOW explicit about requiring that team members to provide signed statements about any conflict of interest?

Section 12: Intended Participation of USAID Staff and Other In County Stakeholders Describe intended participation of USAID staff, implementing partners, national counterparts or customer/beneficiaries in the design or conduct of the evaluation

26. Does the SOW describe who beyond the evaluation team will participate in the evaluation?

27. Does this section describe how and when these individuals will participate in the evaluation?

Section 13: Scheduling and Logistics Address scheduling, logistics and other support

28. Is the SOW clear and specific about any dates that need to be reflected in the evaluation team’s plan?

29. Does the SOW indicate whether it will be USAID’s responsibility to provide the team with logistical support or if the team is expected to make its own arrangements?

Section 14: Reporting Requirements Clarify requirements for reporting and dissemination, including mandatory inclusion of the Appendix 1 of the Mandatory Reference on Evaluation

30. Does the SOW describe requirements for the draft and final report?

31. Is the SOW clear about any requirements that will support the dissemination of the evaluation’s results?

32. Does the SOW include a copy of Appendix 1 of the USAID evaluation policy, which describes USAID’s criteria for ensuring the quality of an evaluation report, as required by that policy?

Section 15: Evaluation Budget Include a budget

33. Is the SOW clear about the total budget or at least the LOE available for the evaluation?

[END OF ATTACHMENT J-4]