Attachment_3.pdf

ANNUAL REPORT

FISCAL YEAR 2016

OCTOBER 2015 – SEPTEMBER 2016

USAID/Jordan Monitoring and Evaluation Support Project

October 31, 2016

This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International.

ANNUAL REPORT

FISCAL YEAR 2016

OCTOBER 2015 – SEPTEMBER

USAID/JORDAN MONITORING AND EVALUATION

SUPPORT PROJECT

Contracted under AID-278-C-13-00009

USAID/Jordan Monitoring and Evaluation Support Project

DISCLAIMER

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

USAID/Jordan MESP – Annual Report (FY 2016) Oct 2015 – Sept 2016 ii

LIST OF ACRONYMS

ADS Automated Directives System

AMELP Activity Monitoring, Evaluation and Learning Plan

AMEP Activity Monitoring and Evaluation Plan

AMERP Activity Monitoring, Evaluation, and Research Plan

AOR Agreement Officer’s Representative

CDCS Country Development Cooperation Strategy

CEP Community Engagement Project

CEPPS Consortium for Elections and Political Process Strengthening

CIS Civic Initiatives Support Project

CISLE Cultivating Inclusive and Supportive Learning Environments in Jordan’s Schools

Activity

CoP Community of Practice

COR Contract Officer’s Representative

CSOs Civil Society Organizations

CWG USAID/Jordan’s Communications Working Group

DEC USAID’s Development Experience Clearinghouse

DO Development Objective

DOC Development Outreach Communications

DQA Data Quality Assessment

DRG Democracy, Rights, and Governance

EDE Economic Development and Energy

EDY Education and Youth

EGMA Early-Grade Mathematics Assessment

EGRA Early-Grade Reading Assessment

EMIS Educational Management Information System

GIS Geographical Information System

GoJ Government of Jordan

HO Home Office

IP Implementing Partner

IRI International Republican Institute

KaMP Knowledge Management Portal

KPI Key Performance Indicator

M&E Monitoring and Evaluation

MESC Management Engineering Services Contract Activity

MESP Monitoring and Evaluation Support Project

MIS Management Information Systems

MoE Ministry of Education

MoL Ministry of Labor

M-PMP Mission Performance Management Plan

MSI Management Systems International

OCI Organization Conflict of Interest

R Result

ToT Training of Trainers

RCT Randomized Controlled Trial

P-MEP Project M&E Plan

PIRS Performance Indicator Reference Sheets

PMP Performance Management Plan

PRO Program Office

USAID/Jordan MESP – Annual Report (FY 2016) Oct 2015 – Sept 2016 iii

PPR Performance Plan and Report

QR/AR Quarterly Report/Annual Report

RF Results Framework

SMEs Small and Medium Enterprises

SOW Statement of Work

TNA Training Needs Assessment

ToC Theory of Change

TVET Technical and Vocational Education and Training

WRE Water Resources and Environment

LIST OF ACTIVITY ACRONYMS

BEST Building Economic Sustainability through Tourism

CBIWDM Community-Based Initiative for Water Demand Management

CEP Community Engagement Project

CEPPS The Consortium for Elections & Political Process Strengthening

CIS Civic Initiatives Support Program

CISLE Cultivating Inclusive and Supportive Learning Environments

CITIES Jordan Cities Implementing Transparent, Innovative, and Effective Solutions

ESCB Energy Sector Capacity Building

FRP II Fiscal Reform Project II

FRPFM Jordan Fiscal Reform and Public Financial Management Project

HRH 2030 Human Resources for Health 2030

HSD Health Services Delivery

IFES International Foundation for Electoral Systems

JCAP Jordan Communication, Advocacy and Policy

JCP Jordan Competitiveness Project

JLGF Jordan Loan Guarantee Fund

JSEP Jordan School Expansion Project

LEIIP Learning Environment: Improved Infrastructure Program

LENS Local Enterprise Support

LETS Learning Environment Technical Support

MESC Non- Revenue Water Project

NFE Non-Formal Education

PAP Public Action for Water, Energy, and Environment Project

RAMP Reading and Math Project

ROLPAS Rule of Law and Public Accountability Strengthening

SCHEP Sustainability Cultural Heritage Engagement Project

SHOPS Strengthening Health Outcomes in the Private Sector

SKEP Schools for a Knowledge Economy Project - Architect and Engineering Services

UCYI UNICEF Child, Adolescents and Youth Initiatives in Jordan

WFD Jordan Workforce Development

WREC Water Reuse and Environmental Conservation

WMI Water Management Initiative

USAID/Jordan MESP – Annual Report (FY 2016) Oct 2015 – Sept 2016 iv

TABLE OF CONTENTS

Introduction

Key Accomplishments

Detailed MESP Results

MESP Goal: Enhanced utilization of quality data to effectively assess progress towards the achievement of the CDCS (by Mission staff)

R1. Strengthened PMP Implementation and Utility A. M&E Mission Training and Technical Assistance B. M&E Support for IPs C. Review or Establish Baselines and Targets, including Surveys and Studies D. Support Implementation of the Mission PMP E. Collect, Analyze and Report PMP Data F. Data Quality Assessment G. Establish a Mission M&E MIS/Geographical Information System (GIS)

R2. Project/Activity Evaluation Strengthened A. Evaluation Assistance B. M&E Mission Training and Technical Assistance C. M&E Support for IPs D. Assessments, Special Studies and Surveys E. Evaluation Utilization

R3. Effective Communication of Selected Mission Results Implemented A. Strategic Communication and Strengthened Collaboration

MESP Collaboration, Learning, and Adaptation (CLA) Implementation

Lessons Learned

Management and Administration

Constraints and Limitations

ANNEX I: Cumulative List of Major Project Deliverables

ANNEX II: MESP M&E Technical Support to Activities to Date

ANNEX III: MESP M&E Technical Support to Mission to Date

ANNEX IV: MESP Performance Data Table (PDT)

ANNEX V: AMEP Change Log

ANNEX VI: Workshops, Programs, and Events Schedule

USAID/Jordan MESP – Annual Report (FY

2016)

INTRODUCTION

Management Systems International (MSI) began the United States Agency for International Development

(USAID) Jordan Monitoring and Evaluation Support Project (MESP) on Dec. 1, 2013, under contract

AID-278-C-13-00009. The principal purpose of the work is to provide technical and advisory services to the USAID/Jordan Program Office (PRO), USAID/Jordan Development Objective (DO) teams, and implementing partners (IPs) in the areas of project monitoring, evaluation, research, organizational learning, and knowledge management to enable USAID/Jordan to fulfill its performance and evaluation requirements.

The third year of MESP implementation was a great success in all areas of technical and systems support.

Among a number of achievements, the successful deployment, population and use of DevResults to enhance Mission data-driven decision-making stands out as perhaps the best example of what can be accomplished when a USAID mission such as Jordan’s exhibits deep commitment to the core principles of collaboration, learning and adaptation when rolling out fundamental changes to critical systems and processes. The patient, deliberate, and flexible approach that PRO adopted in partnership with MESP and the larger

IP community culminated in the milestone of being able to rely on a unified up-to-date web-based information system as the primary source of performance indicator data for the first time in this mission’s history during the spring portfolio review. In continuation of this successful precedent, MESP has focused on further expansion and institutionalization of IP and

Mission collaboration at every phase and opportunity in shaping MESP interventions to maximize their utility and effectiveness.

MESP has set a standard to incorporate stakeholders at the earliest opportunity available in order to more effectively inform design and implementation of activities. Through previous experience, this process has proved to be the most effective means of developing support that meets their interests and needs. In addition to the DevResults success, the value of this proactive collaborative approach was clearly demonstrated in the successful implementation of USAID/Jordan’s first Annual M&E Conference in May 2016, which welcomed Chiefs of Party and Monitoring and Evaluation (M&E) staff from IPs across

USAID’s portfolio. Designed, planned, and implemented through a steering committee of MESP, IP and

USAID M&E staff, the conference embodied the fullest possible integration of MESP stakeholders in its activities. With key support from both Mission and IP staff, the conference provided an effective platform for showcasing new and innovative approaches to M&E. Based on the responses from those in attendance, the conference succeeded in meeting its core objectives in promoting best practices, facilitating networking, positioning M&E, and brainstorming opportunities for future collaboration.

Encouraged by the Mission and PRO, MESP has routinely facilitated the participation of IPs in the

Activity’s decision-making processes at both the planning and learning phases in order to maximize the

87 93 99

11 11

Q2 2015 Q3 2015 Q4 2015 Q1 2016 Q2 2016 Q3 2016 Q4 2016

Graph A - Number of indicators reported in

DevResults by type

Activity Custom USAID Office PMP USAID Mission PMP effectiveness of MESP technical support. This collaborative approach has helped identify solutions to needs and challenges often identified by the IPs themselves. The MESP-designed and implemented M&E

Apprenticeship Program, which is intended to rapidly transform participants into effective entry-level

Activity M&E officers, represents a key example of the importance of MESP and IP collaboration in identifying and addressing needs and challenges IPs face in recruiting qualified M&E staff, with the appropriate skills to quickly succeed in an M&E role. Furthermore, this year’s Community of Practice

(CoP) sessions served to further reflect participatory learning and responsive adaptation, as the session topics were selected by the IPs and were met with overwhelming interest and participation.

MESP continued to provide a variety of different types of technical support to the Mission, including intensive support during a period of ongoing Data Quality Assessments (DQA). Due to the unique nature of MESP’s role, the technical team simultaneously supported Mission staff during the process as well as provided ‘well-timed’ technical assistance to IPs including workshops to support monitoring and evaluation staff in conducting their DQAs most effectively. Furthermore, in continuation of the support provided to the Mission by PRO and MESP during the process of conducting DQAs, PRO and MESP implemented a learning exercise to review and evaluate the effectiveness of the support provided.

Results from this assessment confirmed that DQA practices have been strengthened in targeted areas and that Mission staff perceive PRO and

MESP support to their DQA work as effective. The success of PRO/MESP work in this area is further supported by responses to MESP annual Mission M&E survey where Mission staff selected DQA as the top M&E task where they observed the most improvement over the year. The findings and recommendations generated from the DQA assessment also provided guidance for further strengthening future DQA support.

Directly contributing to enhancing the quality of IP reported MEL data, MESP

M&E Specialists provided feedback on a total of 15 AMEPs/AMELPs developed and/or revised by IPs. The majority of MESP’s efforts focused on facilitating Mission and IP understanding of the Activities’ detailed underlying causal logic as part of the logic model, key results, performance indicators, and learning questions development process. In nearly all instances, AMELP reviews and feedback was clarified and honed in a team with MESP and PRO staff members, a collaborative approach to learning that provides the most opportunities for joint understanding and capacity building. The value of MESP’s AMELP support was clearly recognized by

Mission staff in the annual Mission M&E survey where more than 90% of respondents whose Activities had received AMELP support from MESP reported that this support had produced a better product.

Providing further evidence that MESP’s support to the Mission and PRO is valuable in improving performance, AMELP development and revision was tied for second place as the Mission M&E task that was identified as most improved for the year in the same survey. As a result of MESP/PRO work in these areas, a need for additional capacity building on these topics for Mission and IP staff was identified and addressed through the development of a new two to three-day workshop focused on causal logic and adaptive management. Workshop materials were completed and reviewed by PRO and USAID/Washington with piloting scheduled for early in FY 2017.

90.9%

9.1%

Graph B - Do you think that MESP technical assistance helped produce a better product

(AMELPs)? (N=11)

Yes No

MESP continued to expand its Monitoring, Evaluation, and Learning (MEL) capacity building and technical assistance activities in Year 3, conducting revised and newly developed workshops that incorporated the lessons learned and feedback received during previous sessions and responded to emerging Mission and

IP priorities. A total of 14 workshops were delivered to a total of 136 attendees representing 29

Activities. MESP also revisited its core training approach of conducting standardized workshops across

IPs. For the first time, a tailored version of the ‘Activity M&E Plan (AMEP)/Performance Indicator

Reference Sheet (PIRS)’ workshop was delivered to technical staff of Building Economic Sustainability through Tourism (BEST) Activity at their offices. Similarly, the ‘Preparing for Data Quality Assessment

(DQA)’ workshop was offered to technical staff of HRH2030 Activity. Offering these workshops in IP offices offered several advantages including increasing MEL knowledge with a larger group of technical staff and allowing for additional emphasis on specific priority elements of the training requested by IP MEL managers.

MESP also advanced other key components of Mission and IP M&E systems with the development and revision of several templates such as the Quarterly Report and Performance Indicator Reference Sheets

(PIRS) template. Feedback and suggestions from Mission and IP staff were continually utilized during the development/revision process. In addition, MESP finalized and submitted a draft of a tailored Evaluation

Report checklist and an Evaluation Statement of Work (SOW) checklist. At the request of PRO, several changes have been made in the terminology used throughout MESP’s capacity building portfolio to reflect revisions made to USAID’s Automated Directives System (ADS) related to MEL.

Achieving a key milestone in improving knowledge sharing and collaboration within the USAID community, MESP was granted USAID/Washington approval to launch the Mission’s Knowledge

Management Portal (KaMP). MESP, IPs and Mission staff populated the portal with key sets of Jordan development-related resources in thematic collections, including USAID/Jordan M&E templates and tools, essential documents for understanding the Syrian crisis, and the latest research on how to empower Jordanian at-risk youth, as well as Government of Jordan (GoJ) reports, statistics, Geographic

Information System (GIS) files, and other resources that can sometimes be difficult to find. MESP promoted the KaMP among its primary target audience: IP and USAID staff. As part of MESP’s commitment to improving outreach between itself, IPs and the Mission, MESP developed and launched the ‘M&E Matters’ Newsletter, dedicated to sharing the latest from the world of M&E in Jordan among

USAID IPs.

As part of its continued commitment to ensuring the availability of required M&E data for Mission decision-making, MESP completed, conducted, and/or initiated a total of six evaluations and one assessment. Evaluations included the Civic Initiative Support (CIS) mid-term evaluation and the final performance evaluations for SHOPS and the Public Action for the Water, Energy, and Environment

(PAP) project, all of which were approved by the Mission and shared on the DEC. The Jordan Loan

Guarantee Fund (JLGF) performance evaluation, supported by local subcontractors Integrated Services and Mindset, was finalized and submitted, the JLGF evaluation is notable as it was the first such exercise where the team co-developed recommendations in a collaborative process with the Mission. Working in partnership with MESP sub-contractors Mathematica and Integrated, MESP completed the design and began implementation of an impact evaluation focused on USAID interventions designed to improve learning outcomes in primary schools. Working with a team of around 30 data collectors, the MESP team conducted a four-day training and pre-test to finalize the instrument and data collection approach for the initial matching/verification exercise required to select intervention and control schools for the

RAMP impact evaluation. The MESP team conducted the data collection for this exercise based on a sample of 576 schools (160 treatment schools and 416 potential comparison schools) generated by the

Mathematica team. The MESP team also conducted a workshop to develop the EGRA and EMGA tools required for this study. MESP also launched the midterm performance evaluation of CEP late in Q4.

MESP completed one assessment, the Civil Society Sector Assessment.

As detailed earlier, the Mission and MESP reached a critical milestone in the development and deployment of DevResults as, for the first time, the new Management Information System (MIS) was used to generate indicator data and basic visualizations for use in the Portfolio Review. Reflecting achievement of a similar milestone for geospatial data availability and utility, MESP’s new GIS specialist worked with the MIS/GIS team in expanding and enhancing the Mission’s IP GIS data library. This resulted in the Mission achieving the sustainable capability to produce maps illustrating the distribution of USAID interventions based on three main categories: grants, infrastructure and trainings. MESP will continue to provide support to Mission M&E processes and systems in order to help ensure the availability and accessibility of M&E data required for effective Mission learning and adaptive management.

KEY ACCOMPLISHMENTS

Training and Technical Assistance for the Mission

Developed training material for a new multi-day training workshop for both Mission and IP staff on ‘Causal Logic/Adaptive Management’.

Conducted second annual Mission Survey of perceptions of MESP technical assistance and

Mission M&E practices.

Technical Assistance for USAID PMP

Assisted the Mission in the DQA process.

Finalized learning report focused on Mission DQA improvements.

Provided support to the Mission in developing a tracker to follow up with IPs on the implementation of recommendations from the DQA process.

Completed initial reviews of IP QRs and Mission site visit reports.

Provided support to IPs in changing their M&E terminology in response to updated ADS.

Submitted revised AMELP template and received comments from PRO.

Revised the Activity Monitoring and Evaluation Plan (AMEP) checklist and shared it with PRO for review.

Finalized and shared with IPs the revised Performance Indicator Reference Sheet (PIRS) template and guidance for IPs.

Prepared a short “how-to” guide for planning and conducting site visits based on the completed site visit trip report template for Mission staff.

Supported PRO in finalizing a Quarterly Report template and guide for IPs based on interviews and discussions with IP staff (Q2).

Reviewed JCAP’s Knowledge, Attitude, and Practices survey (KAP) and provided feedback.

Training and Technical Assistance for IPs

Conducted 14 training workshops for IPs, provided 476 person-hours training for IP staff.

Conducted a new workshop for IPs focusing on “Selecting Performance Indicators.”

Developed and conducted a tailored version of the AMEP/PIRS workshop for technical staff at

BEST.

Conducted ‘Preparing for Data Quality Assessment (DQA)’ workshop for technical staff at

HRH2030.

Revised and updated all training materials to reflect guidance from USAID as per the new ADS.

Conducted three Community of Practice sessions.

Planned, organized, and conducted the first USAID/Jordan M&E Conference for Mission and IP staff across USAID/Jordan portfolio. The Conference attracted 114 individuals and Mission

Director Dr. Jim Barnhart provided opening remarks.

Shared M&E Conference brainstorming session review report.

Developed the Apprenticeship modules, promoted and recruited 13 apprentices, delivered the first phase of the program, and started the second phase that will cover two months of on-the-job training with the IPs.

Shared the final report from the FY 2015 IP Assessment with PRO for review, from which key data were integrated into MESP learning in previous quarters as soon as they were available.

Began conducting in-depth interviews with IPs as part of the annual IP assessment of MESP technical assistance.

Technical Assistance for IP AMEPs/AMELPs

Reviewed, discussed, and/or provided feedback to the following 15 Activities in support of developing and enhancing AMELPs:

o Building Economic Sustainability through Tourism (BEST) o Civic Initiatives Support (CIS) o Community Engagement Program (CEP) o Energy Sector Capacity Building (ESCB) o Human Resources for Health 2030 (HRH 2030) o Jordan Communication, Advocacy and Policy (JCAP) o The Consortium for Elections & Political Process Strengthening (CEPPS)/IRIJordan

School Expansion Project (JSEP) o Non-Formal Education (NFE) o Schools for a Knowledge Economy Project - Architect and Engineering Services (SKEP) o Sustainable Cultural Heritage through Engagement of Local Communities Project

(SCHEP)

o Workforce Development (WFD) o Water Management Initiative (WMI) o Fiscal Reform and Public Financial Management (FRPFM) o Health Services Delivery (HSD)

Evaluations

The mid-term performance evaluation of the Civic Initiatives Support (CIS) Program was approved; the report is available on DEC and KaMP:

(https://jordankmportal.com/resources/civic-initiatives-support-cis-program-mid-term-performance-evaluation).

The Civil Society Assessment report was approved; the report is available on DEC and KaMP:

(https://jordankmportal.com/resources/usaid-civil-society-assessment-final-report).

The final performance evaluation of Public Action for Water, Energy, and Environment Project

(PAP) was approved; the report is available on DEC and KaMP:

(https://jordankmportal.com/resources/final-performance-evaluation-of-the-public-action-for-water-energy-and-environment-project-pap).

As part of the RAMP impact evaluation, three rounds of data collection will be conducted to assess changes in student learning outcomes and teachers’ instructional practices, including baseline data collection in November 2016.

Submitted the final report of Jordan Loan Guarantee Fund (JLGF) Performance Evaluation.

Finalized the SOW for the Community Engagement Program (CEP) mid-term performance evaluation. The evaluation team arrived in late September and in-brief meeting was held with

USAID.

Submitted the Evaluation Utilization report of the Fiscal Reform Project II (FRP II) as well as the

Jordan National Youth Assessment (YA), Youth for the Future (Y4F) Project, School

Construction Stakeholder Assessment (SCSA), and Learning Environment Technical Support

(LETS) Project evaluations to the Mission, detailing the Mission’s plans for addressing the evaluations’ recommendations.

Conducted an evaluation utilization assessment of the SHOPS evaluation in partnership with

PRO.

Assessments, Studies and Surveys

The Civil Society Sector Assessment was approved and the report is available on DEC and

KaMP: (https://jordankmportal.com/resources/usaid-civil-society-assessment-final-report).

Completed and submitted assessment reports for EDE that provide guidance on the current and future potential for employment-related growth in USAID’s targeted sector:

o Job Creation Assessment, o Cost-per Job Assessment, and the o TVET Assessment.

https://jordankmportal.com/resources/civic-initiatives-support-cis-program-mid-term-performance-evaluation) https://jordankmportal.com/resources/civic-initiatives-support-cis-program-mid-term-performance-evaluation) https://jordankmportal.com/resources/usaid-civil-society-assessment-final-report) http://pdf.usaid.gov/pdf_docs/pa00ktm1.pdf) http://pdf.usaid.gov/pdf_docs/pa00ktm1.pdf) https://jordankmportal.com/resources/usaid-civil-society-assessment-final-report

Establishment of Mission MIS/GIS and KaMP

Conducted training for second cohort of Activities/new Activities on DevResults.

Trained three newly recruited M&E specialists from WFD, Takamol and CEP on using

DevResults.

Provided key support to BEST in aligning their data reporting utilizing ChemResults with the

Mission’s DevResults.

DevResults was used to generate indicator data and basic visualizations for use in Mission

Portfolio Review.

MESP MIS/GIS Specialist attended GIS training in preparation for providing support and training to IPs on GIS requirements and data collections from the field.

Developed a work plan for GIS data collection to support IPs in retroactively reporting their

GIS data.

Developed and shared with IPs the GIS template and guidelines for reporting in DevResults and

IP QRs.

Produced maps with existing IPs’ GIS data, showing distribution of USAID interventions based on three main categories: grants, infrastructure and trainings.

Developed maps for the JLGF evaluation presentation.

KaMP received USAID/Washington approval and was launched with IPs and Mission staff in Q2

FY2016.

Completed the initial population of KaMP with a host of development related and Jordan-specific resources and collections.

Delivered a presentation of KaMP and its functionalities during the Communications Working

Group in April.

Held several orientation sessions for IP and Mission staff wishing to learn more about KaMP.

Prepared and submitted the KaMP management strategy, which was approved by PRO.

Developed a KaMP status report to be shared with PRO on a regular basis.

Communications

In support of the Civil Society Sector Assessment report, two infographics developed by MESP to provide a snapshot of the complex landscape depicted in the report were approved; the infographics are available on KaMP: (https://jordankmportal.com/resources/jordans-civil-society-at-a-glance-infographic and https://jordankmportal.com/resources/the-civil-society-sector-assessment-infographic).

Disseminated KaMP supporting material, including the how-to guide and business cards.

Developed, edited and branded the Apprenticeship Program Training Modules.

Developed and disseminated the Apprenticeship Program Flyer for candidate recruitment among IPs.

https://jordankmportal.com/resources/jordans-civil-society-at-a-glance-infographic https://jordankmportal.com/resources/jordans-civil-society-at-a-glance-infographic https://jordankmportal.com/resources/the-civil-society-sector-assessment-infographic https://jordankmportal.com/resources/the-civil-society-sector-assessment-infographic

Utilized a new method of data collection during the M&E conference through live polling of attendees (PollEverywhere) to gather opinions, feedback, and questions instantly.

Developed promotional materials for the M&E conference.

Disseminated two editions of MESP’s M&E Matters newsletter to Mission and IP staff.

Developed and disseminated the MESP Info Sheet, an infographic outlining the role of MESP and the technical assistance available to new and existing IPs.

Collaboration, Learning and Adaptation

Revised the annual Mission survey based on lessons learned in FY 2015, such as review of the data collection methodology and amended questions to elicit more useful data.

Incorporated lessons learned from the FY 2015 IP Assessment into MESP’s training plan such as scheduling the DQA workshop to coincide with the Mission’s DQA season as well as incorporating more practical exercises in the newly developed ‘Selecting Performance

Indicators’ workshop.

Conducted a survey to identify which of the seven technical sessions delivered during the M&E conference IP staff would be most interested in repeating in greater depth during the upcoming

Community of Practice (CoP) session. The decision to repeat one of the conference technical sessions on Data Visualization was based on attendee feedback and requests.

Collaborated with PRO and IP staff to address recruitment challenges within M&E in Jordan and as a result planned, developed and implemented the Apprenticeship Program.

Established and coordinated an organizational committee of PRO, MESP, and IP M&E staff in the planning and development of the first annual M&E Conference.

MESP requested and agreed with PRO that all IPs should include any reported data modified in

DevResults within their Quarterly Reports and amended the template accordingly.

MESP’s AMEP was revised and approved in order to improve the utility of collected M&E data.

As part of the process to develop new training materials for the upcoming “Selecting

Performance Indicators” workshop, MESP interviewed five IP M&E staff to obtain more information on the practice of selecting indicators at the IP level, the challenges faced, and learning expectations for the workshop.

DETAILED MESP RESULTS

MESP Goal: Enhanced utilization of quality data to effectively assess progress towards the achievement of the CDCS (by Mission staff)

G L

-0

Percent of surveyed Mission technical management staff reporting agreement that their utilization of M&E data to effectively assess progress towards achievement of the CDCS has improved since the last survey

Baseline:

88.2%

FY 2015 Actual:

88.2%

Q4 FY2016:

N/A

FY 2016 Actual*:

75% FY 2016 Target:

80%

*Indicator collected on an annual basis

MESP conducted the second round of the Annual Mission staff survey, the objective of which is to assess and document changes, potentially resulting from MESP interventions, in the utilization and availability of required performance data for management to achieve the Mission’s DOs. While the reported results for this indicator did not quite reach the estimated target, the large percentage of respondents that reported improvement in this area, 75%, is a positive indication that the Mission continues to make progress in this critical area of practice.

The primary method of data collection is an online survey using 18 questions to elicit the feedback of

Mission staff who oversee Activities as part of their job. In order to maximize Mission ownership of and participation in the survey, PRO sent the online survey link via email to 37 Mission staff including PRO, AOR/CORs, and M&E focal points. Twenty Mission staff responded to the survey (54.1 percent response rate). Responses reflect all six of the Mission’s technical offices (Water Resources and

Environment (WRE), Population and Family Health (PFH), Economic Development and Energy (EDE), Education and Youth (EDY), Democracy Rights and Governance (DRG), and Gender).

Responses to the survey provide data on a variety of performance indicators tracked by MESP. This year, the survey tool was modified to reflect changes made to performance indicators as part of the

MESP’s Activity M&E Plan (AMEP) review as well as various aspects of MESP’s revised work plan. Such changes include indicators relating to the deployment of DevResults and its use in generating data for decision-making. Three indicators were added as follows:

Percent of surveyed Mission technical management staff responding agree/strongly agree (on a five point Likert scale) to “My ability to use indicator data in decision-making has improved as a result of DevResults.”

Percent of surveyed Mission technical management staff who agree that IP indicator data is more available as a result of DevResults.

Percent of Mission technical management staff who report that MESP technical reviews of

AMEPs helped produce a better product.

In addition, during the analysis of the responses to the survey last year, MESP identified a potential limitation in the survey scale and the calculation used for an indicator at the goal level of the Activity:

Percent of surveyed Mission technical management staff reporting agreement that their utilization of M&E data to assess progress towards achievement of the Country Development Cooperation Strategy (CDCS) has improved since the last survey.

In the previous survey and resulting data analysis, the indicator value was calculated by adding the percent of respondents who indicated they “strongly agree,” “agree,” or “somewhat agree”, with the statement: ‘My utilization of M&E data to assess progress towards achievement of the CDCS goal has improved in the past 12 months.’ As demonstrated in Graph C, it may be concluded that 89 percent of

Mission staff respondents of the 2015 survey partially or fully agreed that this performance area had improved using these three response categories. The survey question in 2015 however, did not contain a neutral response (Neither agree or disagree). The absence of a neutral response can be limiting as it forces respondents to only choose between a positive or negative option. Moreover, when respondents do not have the ability to answer the question, whether as a result of the question not being relevant to them in their current capacity or as a result of their lack of familiarity, the neutral option is an appropriate response. In addition to adding the neutral response, the MESP team also simplified the response categories for this question to “strongly agree,” “agree,” “neither agree nor disagree,”

“disagree,” and “strongly disagree.” The positive and negative scale was reduced from three potential answers (Strongly Agree, Agree, Somewhat Agree) to two (Strongly Agree, Agree).

As demonstrated in Graph D, 75 percent of respondents indicated that they ‘agree,’ or ‘strongly agree,’ that the performance area had improved over the past 12 months, or since starting work with USAID for those respondents who had worked less than 12 months in the Mission. This high percentage of reported improvement is a general indication that performance in this critical area of Mission staff technical work is moving in a positive direction. While the 2016 reported results are a 14 percentage point reduction from reported 2015 results (if

Somewhat Agree, Agree and Strongly Agree are added together), this may be accounted for by the 15 percent who opted for the neutral response in 2016. There was also a considerable increase (+9 percentage points) in those respondents who Strongly Agree with the statement in 2016 (15 percent versus 6 percent). In addition to improvements to the response scales, several open-ended questions were added to the survey to provide more insight and depth into the availability and utilization of performance data. Most of the responses to the open ended questions were around the limitations in indicators data availability which is partly due to limitations in DevResults and the continuous changing of indicators.

Lessons Learned:

In the previous Mission Survey, the responses provided an overview of whether utilization of data had improved within the Mission; however, it provided a limited insight into how or why.

As a result, this year MESP included more questions that are open-ended in order to solicit a greater understanding of the changes throughout the year.

6%

22%

61%

11%

0%

0%

0% 20% 40% 60% 80%

Strongly agree

Agree

Somewhat agree

Somewhat disagree

Disagree

Strongly disagree

Graph C - Mission Survey 2015: "My utilization of M&E data to assess progress towards achievement of the CDCS goal has imporved in the last 12 months" (N=18)

15%

60%

15%

10%

0%

0% 20% 40% 60% 80%

Strongly agree

Agree

Neither agree or disgaree

Disagree

Strongly disagree

Graph D - Mission Survey 2016: "My utilization of M&E data to assess progress towards achievement of the CDCS goal has imporved in the last 12 months" (N=20)

R1. Strengthened PMP Implementation and Utility

R

.1

Average rating of the availability of performance monitoring data that the surveyed mission staff need to do their job (on a scale from 1 to 10, one being no available data and ten being data is readily available)

Baseline*:

5.2

FY 2015 Actual:

6.2

Q4 FY 2016:

N/A FY 2016 Actual†:

6.8

FY 2016 Target:

* Baseline was established at the same time as the 2015 data collection, asking respondents to retroactively rate the availability of performance monitoring data.

†Indicator collected on an annual basis.

Availability of Mission M&E Data for Decision-making

As described above, the M&E survey of Mission staff is designed to provide performance data in a variety of MESP support areas. Working in close partnership with PRO, MESP technical assistance is intended to support the Mission’s utilization of M&E data for portfolio management in multiple ways, including improving the availability of required performance monitoring data through the population of

DevResults. Last quarter, the Mission and MESP reached a critical milestone in the development and deployment of DevResults as, for the first time, the new management information system (MIS) was used to generate indicator data and basic visualizations for use in the Portfolio Review.

In order to document changes in data availability potentially resulting from MESP interventions, MESP’s revised AMEP includes the following performance indicator:

R 1.1 Average rating of the availability of performance monitoring data that the surveyed mission staff need to do their job (on a scale from one to 10, one being no available data and ten being data is readily available).

The continued positive trend for this indicator and other data collected during this year’s survey lends support to the belief that interventions in this area are having positive results.

Respondents of the Mission Survey were asked to rate the availability of performance data required to conduct a portfolio review or other M&E-related task data at the present time. The rating scale ranged from one (not available) to 10 (readily available). Last year, the average response to this question was 6.2 with a baseline of 5.2 (estimated retrospectively by the same respondent), while in response to the survey this year, the average was calculated as 6.8 (mode of seven with five responses, and a range from three to 10). This may demonstrate that positive change is taking place in targeted areas.

Two additional questions were included in the survey to give more depth and understanding to the changes in the availability of performance monitoring data. When asked “Since you started at the

Mission, has the availability of performance monitoring data that you need to do your job (for example, if you were performing a portfolio review or other M&E tasks) improved, worsened or stayed the same?” Sixty-five percent of respondents reported that the availability of performance monitoring data had improved since they started at the Mission (Graph F). Analysis of responses to qualitative follow-up questions asking why respondents felt data availability had improved highlights the positive influence of

DevResults in this area as five of the seven respondents mentioned that data is more readily available as a result of the system.

65% 25% 5

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Graph F - Mission Survey 2016 "Since you started at the Mission, has the availability of performance monitoring data that you need to do your job (for example, if you were performing a portfolio review or other M&E tasks) improved, worsened or stayed the

Improved Stayed the same Worsened Don’t Know

6.8

6.2

5.2

0 1 2 3 4 5 6 7 8

Graph E - Mission Survey 2015-2016

Average respondent rating out of 10 of the availability of "performance monitoring data that you will need to do your job." (N=20)

Baseline 2015 2016Not

Available Readily

Available

While the results for the question about changes in the availability of performance data presented above may reflect positive change, responses to the subsequent question on the actual availability of such data when needed illustrate the need for continued improvement. When asked, “Over the last six months, did you have the performance monitoring data that you need to do your job effectively?” only 25% of respondents indicated that such data were available “always” with 75 percent of respondents reporting availability as “sometimes” (Graph G).

20% 75% 5

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Graph G - Mission Survey 2016

"Over the last six months, did you have the performance monitoring data that you need to do your job effectively?" (N=20)

Always Sometimes Don't know

A. M&E Mission Training and Technical Assistance R

.2

.5

Number of person-hours of training in Monitoring and/or Evaluation conducted by MESP

Baseline:

FY 2015 Actual:

Q4 FY 2016*:

FY 2016 Actual†

FY 2016 Target:

* Mission Staff: 0 hours IP Staff: 56 hours †Mission Staff: 3.5 hours IP Staff: 409.5 hours

Mission Staff Perceptions of MESP Technical Support

Over the course of the first two and a half years of implementation, MESP has provided support to

Mission staff through a combination of technical assistance, tools, guidelines, and workshops that are designed to improve Mission and IP performance in MEL practices. MESP’s annual Mission survey includes several questions intended to help MESP and PRO understand the perceived utility of MESP support. As with last year’s assessment, the results of this year’s data collection to date are very positive and indicate a high level of perceived utility by Mission staff of MESP’s technical support.

Mission staff were asked to state their level of agreement with the following statement: “MESP support helped improve my ability to implement M&E tasks.” Three-quarters of the 20 respondents indicated either agreement or strong agreement in this area (Graph H). While the previous years’ survey used a different scale for coding of responses (“Strongly agree,” “Agree,” “Somewhat agree,” “Disagree”), the results of 94% of respondents falling into the three possible categories of positive agreement with 70% selecting the two highest levels of agreement are quite similar to reported data for FY 2016.

The survey also included questions intended to assess perceptions related to specific areas of technical support provided by MESP. One area of intensive work by MESP staff is in the area of AMEP/AMELP reviews. In order to limit responses to staff that had actually worked with MESP in this area, a filter question was first used to determine which respondents had overseen Activities that received MESP support with their AMEPs, of which slightly more than half had done (11 of 20). Of these 11 respondents, all but one or 91% responded ‘yes’ when asked ‘Do you think that MESP technical assistance helped produce a better product?’ This high level of perceived utility is a strong endorsement

15%

60%

20%

5%

GRAPH H - Mission Survey 2016

To what extent do you agree with the following statement:

"MESP support helped improve my ability to implement

M&E tasks" (N=20)

Strongly agree Agree Neither agree or disagree Disagree of MESP’s value in helping Mission managers work with IPs to produce higher-quality AMEPs. While there is no way to directly attribute it to MESP support based on the survey results, it is also positive to note that respondents highlighted AMEP development/revision as one of the top three areas of M&E task improvement along with portfolio reviews (tied with five responses) and DQAs (six responses).

Mission Staff Training/Workshops in FY 2016

Causal Logic Training

Based on a number of different observed challenges in the articulation of clear cause and effect logic as reflected in Activity-level logical frameworks and indicators, MESP and PRO decided to develop a new workshop focused on building the capacity of Mission and IP staff in causal logic and adaptive management. During Q2, MESP and PRO held several meetings to work on the details of this training including; the learning objectives, target audience, methodology, duration, and proposed dates. MESP also worked closely with HO to design the training. Based on these meetings, MESP submitted in Q2 a draft concept note and course outline for PRO approval.

MESP continued to work closely with PRO and HO to refine and improve the proposed training. PRO sought guidance from USAID in Washington for further enhancements. In Q4, MESP submitted the draft training materials, and proposed methodology for a multi-day training, which were reviewed by PRO and returned with further guidance.

MESP and PRO agreed to organize a pilot training that will include participants from MESP and PRO to further improve the training and tailor it to USAID/Jordan needs and context. Betsy Bury, MESP’s Technical Director, will deliver the training which is expected to serve two purposes:

Being a pilot training, it will be an opportunity to review the training materials, methodology, exercises, and other aspects of the training and provide feedback to improve, enhance, and make them more Jordan context specific.

It will also serve as a ToT training for MESP staff in order to be able to organize future similar trainings for Mission and IP staff locally.

The training will help participants to:

Have very strong skills in articulating the underlying Activity theory of change and can understand and troubleshoot various types of project logic models, including results frameworks, log-frames and other types of logic models;

Understand principles of “cause and effect” and “sufficient and necessary” in program logic results development, including the need to create plausible initial approaches in situations where the underlying cause and effect relationships are not well understood;

Understand principles for selecting useful performance and contextual indicators for strategies, projects and Activities; and

Have experience conducting data driven performance reviews/portfolio reviews and will be able to analyze and discuss monitoring data and use it to inform adaptive management.

Once the training has been tested and finalized, MESP will organize and conduct the training for Mission and IP staff as needed utilizing its local team. The pilot training is planned to be held during Q1 of FY

2017.

Mission Staff Technical Assistance

Review of Mission Site Visit Reports

At the request of PRO, MESP reviewed the site visit reports completed by Mission Staff as part of their

Activity management and data quality assurance responsibilities. The review is intended to provide feedback to PRO on the quality of information included in the reports. MESP’s M&E specialists are comparing the information in completed field visit reports with the template and guidance. The process will help to identify potential methods of improving what and how information is reported.

In addition, PRO requested that the review be conducted in conjunction with the review of IP quarterly and annual reports. This is intended to establish whether IPs are reporting on the visits or any follow-up as needed to respond to issues identified through the site visits.

MESP’s review has identified several areas of potential action in order to strengthen the effectiveness, efficiency, and utility of these visits. MESP will meet with PRO and share the results of the reviews in order to plan potential technical support in early FY 2017.

Review of IP Quarterly and Annual Reports

As mentioned above, PRO requested MESP support in reviewing the IPs’ quarterly and annual reports.

In particular, PRO requested that MESP review the reported performance indicator data to identify any issues or discrepancies compared with data in DevResults. MESP was also requested to review the different types of issues addressed as challenges, successes, and best practices. As with the site visit review, the purpose of MESP support in this area is to help identify potential capacity building and guidance needs for future MESP/PRO support in order to improve the quality of IP reporting.

In order to systemize the reviewing process, MESP developed and began testing a tracker, which captures and summarizes the content within key fields of the reports i.e. submission, performance data, challenges, success stories, and general/specific comments in the report. MESP will meet with PRO to review the tracker and finalize procedures for subsequent reviews in early FY 2017.

MESP identified one issue of note that revisions to the indicators and data reported on DevResults were not currently reported by the Activity, including changes made by MESP to its own data in the system.

As a result, there was no up to date log of revisions made to the current indicators and data readily available to explain potential discrepancies between data reported in previously approved quarterly reports and data that were subsequently revised at a later date. In order to ensure that all stakeholders, including potential future evaluators, have sufficient documentation of such changes, MESP recommended that IPs include details about such revisions within their quarterly/annual reports that cover the period of time when the change was actually made. Having a log of revisions is of particular importance during the process of portfolio reviews and evaluations in order to identify and explain why there may be observed discrepancies in data. While these revisions may be covered in the AMELP change logs, these mostly focus on changes to actual indicators and other components of MEL systems rather than specific revisions to previously reported data.

Once reviewed and approved by PRO, MESP intends to use the tracker in a reviewing process where each member of the technical staff will be assigned a number of IPs to review their respective reports.

Support to DQAs

In continuation of the support provided during Year 2 where MESP provided technical assistance, tools, and guidelines targeted to improve the preparation and performance of IP and Mission staff in conducting DQAs, MESP staff attended most DQA sessions conducted in FY 2016 and provided support to Mission and IP staff as needed.

In order to establish whether Mission staff perceived improvements in areas of the DQA process targeted by PRO and MESP support during…