Attachment_2_-_USAID_Mozambique_Evaluation_Mission_Order_(1).pdf

® ~§.!.~Q MOZAMBIQUE

MISSION ORDER

s u BJ E c T : Evaluation 0 RD ER N 0. MO 203 - 1

FI LI NG : Series 200 - Programming Policy CHRON NO.

DATE ISSUED : DATE EFFECTIVE :

Upon Signature

S U P E RS E D ES : 200 - 2

AUTHORITY: See Section 11 below PAGE 1 OF 11 PAGES

I. PURPOSE

The purpose of this Mission Order is to provide Mission guidelines, procedures and recommendations related to evaluation of development objectives, projects and activities/implementing mechanisms (IMs).

11. AUTHORITY/POLICY

• Government Performance and Results Act (GPRA) of 1993 and

Government Performance and Results Modernization Act (GPRMA) 2010 regarding government-wide requirements regarding performance management.

• ADS 203.3.1 regarding all aspects of evaluation; ADS 203.3.3.1 regarding the Evaluation Plan in the Mission-wide Performance Management Plan.

• Performance Plan and Report (PPR) guidance regarding reporting in the Evaluation Registry of the PPR.

• USAID Evaluation Policy to provide clarity to USAID staff, partners and stakeholders about the purposes of evaluation, the types of evaluations that are required and recommended, and the approach for conducting, disseminating and using evaluations.

Ill. DEFINITIONS Significant terms from the ADS for this Mission Order (MO) are:

• Evaluation (ADS 203.3.1 ): the systematic collection and analysis of information about the characteristics and outcomes of programs and projects as a basis for judgments to improve effectiveness, and/or inform decisions about current and future programming. Evaluation is distinct from assessment, which may be designed to examine country or sector context to inform project design, or an informal review of projects.

• Impact Evaluation {ADS 203.3.1.1): measures change in development outcome that is attributable to a defined intervention; these evaluations are based in models of cause and effect and require a credible and

U.S. AQency for lntemational Devek>pment JATComplex Rua 1231, n.0 41 P.0 .Box783 Maputo, Mozambique

Tel: (258) 21 35 20 00 Fax (258) 21 35 21 00 www.usaid.gcNtrnz

U.SA Postal Address 2330 Maputo Place U.S. Department of Slate W3shington, D.C. 20521-2330 rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change.

• Performance Evaluation (ADS 203.3.1.1): focus on descriptive and normative questions: what a particular program, project, [or activity] has achieved; how it is being implemented; how it is perceived and valued;

whether expected results are occurring and other questions that are pertinent to program design management and operational decision making.

• Mission-wide Performance Management Plan (PMP) (ADS 203.3.3): a tool to plan and manage the process of monitoring, evaluating, and analyzing progress toward achieving results identified in a COGS and project LogFrame in order to inform decision-making, resource allocation, learning, and adapting projects and programs.

• Development Objective (00) (ADS 201.3.3.3): the most ambitious result that a USAID Mission or Bureau/Independent Office (B/10), along with its partners, can materially affect, and for which it is willing to be held accountable.

• Project (ADS 201.3.11 ): A project is a set of executed interventions, over an established timeline and budget intended to achieve a discrete development result (i.e. the project purpose) through resolving an associated problem. It is explicitly linked to the COGS Results Framework.

• Activity/Implementing Mechanism (IM) (ADS 201.3.11): as means of implementing a program or project to achieve identified results, generally through the use of a legally binding relationship established between an executing agency (generally a USG agency like USAID or a host government agency) and an implementing entity (contractor, grantee, host government entity, international organization) to carry out programs with USG funding. Examples of implementing mechanisms include contracts, cooperative agreements, grants, inter-agency agreements, bilateral project agreements, fixed amount reimbursement and performance agreements and cash transfers to host country governments, public-private partnerships, Development Credit Authority (DCA) agreements, Development Innovation Venture (DIV) awards, and policy dialogue carried out by USG officials.

• Program (ADS 200): a program is aligned with a COGS Development Objective and includes all projects and other activities that are associated with a particular DO.

List of Acronyms:

ADS Automated Directives System AM Activity Manager AOR Agreement Officer's Representative COGS Country Development Cooperation Strategy COR Contracting Officer's Representative DEC Development Experience Clearinghouse DO Development Objective DQA Data Quality Assessment EP Evaluation Plan

- 2-

EPOC

IM

IP

M&E

MO

PAD

PD

PDO

PO

POC

PMP

PPL

PPL/LER

PPR

PIR

sow

TEPOC

Evaluation Point of Contact Implementing Mechanism Implementing Partner Monitoring and Evaluation Mission Order Project Appraisal Document Project Design Project Development Officer Program Office Point of Contact Performance Management Plan Bureau of Policy, Planning and Learning PPL Office of Learning, Evaluation and Research Performance Plan and Report Portfolio Implementation Review Statement of Work Technical Evaluation Point of Contact

Other terms include:

• Program Office (PO) Evaluation Point of Contact (EPOC): Per ADS 203.3.1.2, Missions must designate an EPOC within the Mission PO.

This individual plays key roles throughout the processes outlined in this MO, helps ensure Mission compliance with the USAID Evaluation Policy across the Mission's projects, and interact with USAID/W regional or technical bureaus and the Bureau of Policy, Planning, and Learning, Office of Learning, Evaluation, and Research (PPL/LER) .

Their function is both as a value-added subject matter expert on evaluation policies and procedures for the Mission while serving a central quality control and information-gathering function.

• Contracting Officer's Representative (COR)/Activity Manager (AM):: For Mission awards, some evaluation responsibilities fall to the COR; for awards made in Washington (e.g., field support, Food for Peace, etc.), some evaluation responsibility falls to the AM.

• Technical Evaluation Point of Contact (TEPOC): Each DO team must designate at least one technical EPOC (TEPOC) to work with the EPOC(s) on evaluation-related activities. Generally, inhere is one, a DO Team's M&E Specialist will be the TEPOC. This individual is responsible for ensuring team-level compliance with this Evaluation MO and USAID policy. The TEPOC ensures that evaluations are planned at the program and activity design phase, and ensures that evaluations are incorporated into relevant activity's M&E Plans and program PMPs.

The TEPOC also serves as a liaison with the PO to strengthen the Mission's evaluation outcomes. Each DO team is responsible for ensuring that their TEPOC has appropriate training .to manage the assigned duties.

- 3-

IV. ROLES AND RESPONSIBILITIES

All Mission Offices are responsible for the implementation of this Mission Order and adherence to the Agency guidance and principles on evaluation.

However, Program and Technical Offices are central to successful evaluations. Between these offices, there must be a clear delineation of roles and strong sense of cooperation and transparency. See Figure A in ADS

203.3.1.2 for respective roles of these offices across various evaluation tasks.

The Program Office (PO) is responsible for updating this Mission Order.

V. PROCEDURES

A. Planning

The Mission Evaluation Plan As part of the Mission-wide PMP, the PO will work with the DO teams and EPOCs to create and maintain a multi-year Evaluation Plan (EP) that corresponds with the life of the Country Development Cooperation Strategies (CDCS). It is formulated and approved initially as part of the PMP process. The plan will identify evaluations that are expected to be completed during the life of the CDCS, plus any planned for out-years to ensure sufficient planning lead time. The EP includes both evaluations required by the USAID policy (see below) and evaluations that are not required. The EP will include the required information about each planned evaluation as described in ADS 203.3.3.1, including, but not limited to: the program/projecUactivity to be evaluated, technical area, type of evaluation, estimated cost, procurement mechanism, start and finish dates, POCs for that particular evaluation, and any comments (such as possible evaluation questions, etc.), plus any additional information determined by the PO.

Determination of Evaluations to Include in a Mission Evaluation Plan For each project, consideration will be given during design to the performance evaluation(s) or impact evaluation(s) that will be undertaken (see Project Design CPD) MO), including identification of key evaluation questions. The PO will determine if the project under design is subject to the "large projects" evaluation requirement or the "pilot activities" evaluation requirement. Those requirements are stated fully in ADS 203.3.1.3, but are restated briefly here:

• Large projects: Each large project should undergo at least one external evaluation. According to the current ADS definition, a project is not just one implementing mechanism but generally a collection of them (see Definitions). A large project is one that equals or exceeds the mean (average) project size in dollar value for the DO. To calculate the average project size, divide the total program expenditures of the most recent full fiscal year budget available by that year's number of projects for each DO. This average dollar amount per project at the DO level is

-4-the starting point for identifying the "large" projects that will require evaluation in that DO.

• Pilot activities: Any activity/IM within a project involving untested hypotheses or demonstrating new approaches that are anticipated to be expanded in scale or scope through U.S.

Government foreign assistance or other funding sources will, if feasible, undergo an external impact evaluation. DO teams should identify pilot activities at the design stage. If pilot activities are added to a project after the design stage, the EPOC must be notified. If it is not possible to effectively undertake an impact evaluation, the DO team may instead propose a performance evaluation, with PO approval.

• Non-required Evaluations: Projects/programs/activity/IMs that are not subject to the "large project" or "pilot activities requiremenr may still, of course, be evaluated. This may include, for instance, DO level evaluations or evaluations based on the high-priority evaluation questions identified in the COGS.

In such a case, DO teams, the PO, or the front office may propose either external or internal evaluations to the PO for inclusion in the Mission EP. DO leadership, in coordination with the PO, should make strategic choices for what can and should be evaluated based on management and learning priorities, but with the understanding that a majority of resources under USAID management will be subject to evaluation. Some triggers that may indicate that a non-required evaluation is needed include:

o Performance management indicates there are unexpected results (positive or negative) that need to be explained.

o A key management decision must be made and there is inadequate information.

o Periodic PIRs have identified key questions that need to be answered.

o Customer or partner feedback suggests that there are implementation problems or unmet needs.

o The contribution of USAI D activities to results is questioned. Issues of sustainability, cost-effectiveness, or relevance arise. The validity of RF hypotheses and critical assumptions is questioned. Recommendations for actions to improve performance are needed.

o Extracting lessons is important for the benefit of other DOs or for future programming, strategic planning, or project design.

The Mission provides an Evaluation (and Research) Plan during each of its semi-annual PIRs. Therefore, on an annual basis, generally during the month of October, the EPOC leads an evaluation planning meeting with all TEPOCs to determine which evaluations need to take place and the relevant priority of each. In preparation for the Mission's mid-year PIR, generally during the month of May, the EPOC works with

-5-the TEPOCs to follow up on and update the EP. The EP is presented to the Mission and is discussed during the PIR. The EPOC makes any modifications to the EP after each PIR and periodically, if and when needed.

Evaluation Plan Maintenance While the EP is initially drafted and approved with the Mission-wide PMP, it is fully expected that through project design, learning, and unforeseen changes, the EP will change.

• The EP will be updated on an ongoing basis by the EPOC in coordination with DO teams as projects are developed; as new information during program/project/activity implementation "triggers" a need for an evaluation (ADS 203.3.1.3); or details for already planned evaluations are filled-in or changed.

• Generally, major decisions on changes must be made through some other formal decision process, for example, as decided in a PIR (see Portfolio Review MO), a Project Appraisal Document's (PAD) review meeting, or a Quarterly Financial Review meeting. Major decisions would include the addition or subtraction of an evaluation, the re-classification of an evaluation from required to non-required, and a change in evaluation type.

However, there will also be routine maintenance, such as updating budget estimates to actuals and changing names and dates. In both cases, DO teams should share new information with the EPOC, and the EPOC will simply enter the new information into the EP to keep it up to date.

• Updates must be made at a minimum on a semi-annual basis, generally around the months of October and May. The Plan may serve as basis for the Evaluation Registry, submitted as part of the PPR.

• The EPOC will provide a read-only copy of the most recent version of the EP for Mission use in the Mission's Program Information Index (Pll) folder at:

P:\Pll\PR0\07_MnE\02_Mission_Eva/uation_Plan.

Budgeting Per ADS 203.3.3.1, the EP includes the estimated budget for each evaluation. As part of EP maintenance, the EPOC will update those budget estimates (including updating with actual figures) on an ongoing basis. Based on this work, the PO will calculate, on an annual basis, a budget estimate for the external evaluations to be undertaken during the following fiscal year. The purpose of this exercise is to compare Mission EP's budget with the goal that 3% of USAID Mission/Office's total program budget is applied towards conducting external evaluations, as stated in the ADS, and to inform Mission decision making about the extent and cost of an evaluation versus an Agency standard (see ADS 203.3.1.2). Evaluation procurement information, including the related Program, Design and Learning funds, will be kept and managed in the Mission's OPSMaster database. In addition, -6-missions are required to annually update the Evaluation Registry of the PPR with budget data for the current fiscal year plus two out years.

Planning Individual Evaluations In planning an individual evaluation, it is USAID/Mozambique policy to follow these steps and guidelines:

• Evaluations should be considered in the CDCS and in the early stages of project design to ensure adequate planning and implementation (see PD MO).

• Evaluations may also be identified during the course of a portfolio review at any level (see PR MO) or as needs arise during implementation. When this occurs, the EPOC should add the evaluation to the EP to ensure it is adequately resourced.

• Project managers and COR/AOR/AMs are responsible for ensuring that implementing partners (IPs) of the project or activity that will be evaluated are aware of any planned evaluations and the steps IPs need to take to ensure a successful evaluation, such as the collection, maintenance, and documentation of appropriate monitoring data.

• DO Teams, through a designated TEPOC, will initiate and lead the writing of statements of work (SOW) for evaluations in cooperation with the EPOC. The SOW should conform to USAID Evaluation Policy requirements as described in ADS

203.3.1.5. The EPOC, TEPOC and COR/AOR/AM of the project

or activity being evaluated should work closely together in drafting the evaluation SOW.

• It is recommended that before writing the SOW, the TEPOC and other SOW drafters should review USAID guidance documents, such as the "Evaluation Statements of Work" How-To Note, the Checklist for Drafting Evaluation SOWs, and others found on Program Net or from the PO.

• If the activity being evaluated is Field Support (a centrally managed mechanism), then the Washington-based AOR/COR must be involved in the drafting and approval stages of the

SOW.

• Part of the value-added function of the PO is to suggest potential implementing mechanisms for carrying out the evaluation.

• Once drafted, the EPOC (or designee) will lead a peer review of the SOW, including no fewer than two individuals in addition to the EPOC (or the designee). Peer reviewers may include individuals from the DO team and PO, as well as USAID/Washington regional and technical bureaus, PPULER and local partners. Emphasis should be placed on finding at least one peer reviewer with evaluation methods expertise. The peer review team should be given at least four full business days to review the SOW and provide comments and edits back to the

TEPOC.

• After receiving comments on the SOW, the relevant DO team will make any revisions in cooperation with the PO before final

- 7-clearance by the PO. The PO will be responsible for ensuring that the SOW is compliant with USAID Evaluation Policy.

B. Implementation Independence is a critical element to high quality evaluations that is highlighted in USAID's Evaluation Policy and ADS 203. For this reason, it is USAID policy that all required evaluations are to be carried out externally through a third-party contractor, managed directly by USAID, and the contract for the evaluation will be managed by the PO or Technical Team M&E Specialist (i.e., the Evaluation COR/AM is a PO staff member or Technical Team M&E Specialist, and absolutely not the ProjecUActivity COR). In cases where Mission management determines that appropriate expertise exists within the Mission, Washington, or other posts and that engaging USAID staff will facilitate institutional learning, an evaluation team may include USAID staff, but an outside expert with appropriate evaluation skills and experience will lead the team. Non-required evaluation teams can be internal or external as needed.

• During the planning for each individual evaluation, the PO and

TEPOC will discuss the management and implementation responsibilities of the evaluation, including the COR/Manager designation. Because circumstances vary with available resources (staff and time) to manage an evaluation adequately, sometimes the PO will be COR and sometimes the Technical Team M&E Specialist will be COR. Considerations about the roles and responsibilities of the evaluation POC and TEPOC will consider the technical and workload capacity of the individuals.

Ultimately, the COR recommendation should select the person best poised to deliver a high quality, independent evaluation in a timely manner.

• The Supervisory Program Officer and Technical Team Leader provide written clearance (via email is acceptable) on the COR recommendation, and the Contracting Officer makes the final decision, in consultation with the Deputy Mission Director.

• Exceptions to the requirement for an external leader of the evaluation team will be documented in a memo and cleared by the Mission Director for a required evaluation.

• It is highly recommended that USAID/Mozambique staff, especially the evaluation COR, be part of the evaluation team, under the supervision of the T earn Leader. The contractor is responsible for drafting all evaluation deliverables; no USAID staff is expected to provide evaluation deliverables to the contractor.

• The PO will be responsible for collecting signed statements from evaluation team members attesting that they have no conflict of interest or describing an existing conflict of interest relative to the project being evaluated (See recommended form). If a conflict of interest or potential or appearance of a conflict of interest exists, the PO will determine if the evaluation team member

- 8-should be removed from the evaluation team in order to ensure that the conflict does not impair the independence of the evaluation.

• As part of his/her evaluation management responsibilities, the evaluation COR is responsible for sharing the winning contractor's evaluation designs completed by an evaluation team with relevant stakeholders, particularly implementing partners or other funders who contributed to the evaluation or the program/project/activities addressed by the evaluation.

Reports It is required that the draft evaluation reports undergo peer review.

• Draft evaluation reports should be received by the Evaluation

COR/Manager and distributed to relevant individuals for peer review.

• The Evaluation COR/Manager will lead the peer review of the draft report, including no less than two individuals in addition to the Evaluation COR/Manager. Peer reviewers may include individuals from the DO team, the PO, USAID/Washington regional and technical bureaus, and local partners. No more than half of the peer reviewers should be from the DO team that oversees the project to be evaluated. The peer review time is subject to the timing stipulated in the SOW; it is recommended that the peer review be given at least five business days to complete.

• Peer reviewers will be asked to review the draft report according to the USAID Evaluation Policy standards of quality (as described in ADS 203.3.1.8).

• The Evaluation COR/Manager is required to approve the final evaluation report before it can be sent back to the evaluation contractor and before authorization of payment to the contractor can be made.

• The Mission may identify issues, question analysis and provide suggestions to the evaluation team. However the final content of the evaluation is determined by the evaluation team. If necessary, the Mission may submit a statement of differences as part of the final evaluation report (see below).

All draft reports will be shared with implementing partners whose activities are addressed in the evaluation and other organizations who contributed funding to the activity(ies) or the evaluation, as well as other implementers involved in the project.

• The implementing partners, funders, and evaluation team members will be given the opportunity to write an individual statement of difference regarding any significant unresolved differences of opinion to be attached to the final evaluation report.

- 9 -

• If any statements of difference are received, the evaluation team will be given the opportunity to revise and/or respond to the statements in the evaluation report.

In addition:

• The PO will be responsible for ensuring that the final report is compliant with USAID Evaluation Policy and ADS 203 and follow the recommended reporting practice as described in the "Preparing Evaluation Reports" How-To Note.

• All data collected by USAID for purposes of an evaluation will be uploaded and stored in a central and accessible location managed by the PO. The PO will ensure that the evaluation team provides data that are organized and fully documented for use by those not fully familiar with the project or the evaluation.

In accordance with ADS 203.3.1.10, data can be submitted to DevelopmentData@usaid.gov.

C. Learning and Reporting Evaluation is the systematic collection and analysis of information to better understand characteristics and outcomes of a Mission's programs, projects and activities. Given the time and effort that is expended in planning and conducting evaluations, it is essential that the Mission use evaluations to understand performance, test development hypotheses, question assumptions and cause and effect relationships and, ultimately, manage for results and learning. Per ADS 203.3.1 , to ensure that evaluations are used and learning occurs, it is important to respond to and share evaluation findings. These procedures operationalize these principles.

Responding to Evaluation Findings Following the completion of an evaluation, the PO, in coordination with the DO teams, will lead relevant Mission staff through the process of responding to evaluation findings as described in ADS 203.3.1.9. This process will lead to the development of an action plan for addressing the evaluation findings, conclusions, and recommendations the Mission has accepted or any other actions it deems necessary as a result of evaluation findings, conclusions, and recommendations. If necessary, the Mission may submit a statement of differences as an annex to the evaluation report. During Portfolio Implementation Reviews, the status of action plans for evaluation findings and their use in respective decisions will be discussed and documented (see Portfolio Review

MO).

Sharing Evaluations with Stakeholders Findings from evaluations will be shared as widely as possible, with a commitment to full and active disclosure. Following the completion of the evaluation report, the PO will ensure that the final evaluation report is posted on the Development Experience Clearinghouse (DEC) no later than three months after completion. Any exception to this

-10-requirement to make the full evaluation report available publicly on the DEC must be approved by the Director of PPULER, per the "Exceptions to Public Disclosure of USAID-funded Evaluations" guidelines document (see ProgramNet at https://programnet.usaid.gov/library/public-disclosure-waiver guidelines). Exceptions to full public disclosure require a justification in writing from the evaluation TEPOC, addressed to the Mission Director and cleared by the EPOC. Every effort must be made to develop public reports and include classified material in a separate version or annex.

The PO, in cooperation with the relevant DO team, will determine whether to post the evaluation on the Mission website and/or translate the executive summary into the local language.

VI. EFFECTIVE DATE AND CANCELLATION

This Mission Order is effective on the date of signature, and will remain in force until cancelled or amended.

Mission Director

- 11 -

USAID/Mozambique Mission Order on Evaluation

Drafted: Shannon Howard, M&E Specialist, Program Office

Clearances

A/PO:

IHO:

CSmith (' A,.,.Jit

Aleggo-e==~==(Jl==~=======~-

Date \J~~ Zolf j

Date l(t-°1 l 1~ ATB: TBorn Date ----- ----

DG: NMueller 1/" / -2---

Date 2...l r=e..!. ~C>J 't

( ED: Date lCf kJo ~~

I

AAO:

EXO: NKester

~· . )~ ... °"'' ':. -\- \}-\ J,, '2r . II ~\.;I ; . c.-

1) () rn o 1<.. f e_~ ~.J.(i t::. , d 1 ',,..., r( C-·J e; ~ s -~+;\a_ - J I+->'+.·' ~. IJ 0 ~ c l.P~r

I":: t1;\f ( P ."f d '1;.+ tf 1:: ~µ . ·~ J Jv ~ l c \ "'- f' 6(£7 '7 t.J . \ '. bP ... { -~ . .:. .' Jt> (~}\"~-{{ ' < ~..,,. ·~ ~<l A c .~ 1N-? J.

~\f ~ cl- \5 cu.s.:-.. '• \\ (..) ( s I'' ~ ( fr i' f.{J , • .)Ji ~- -·"' u \ {f'V' ~ \ ~ \00ti.e JC

. ff \ \\ J "";)I l~ - TL- ..> '