Attachment_4.pdf

ANNUAL REPORT

FISCAL YEAR 2017

OCTOBER 2016 – SEPTEMBER 2017

USAID/Jordan Monitoring and Evaluation Support Project

October 31, 2017

This publication was produced for review by the United States Agency for International

Development. It was prepared by Management Systems International.

ANNUAL REPORT

FISCAL YEAR 2017

OCTOBER 2016 – SEPTEMBER

USAID/JORDAN MONITORING AND EVALUATION

SUPPORT PROJECT

Contracted under AID-278-C-13-00009

USAID/Jordan Monitoring and Evaluation Support Project

DISCLAIMER

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

USAID/Jordan MESP – Annual Report (FY 2017) Oct 2016 – Sept 2017 ii

LIST OF ACRONYMS

ADS Automated Directives System

AMELP Activity Monitoring, Evaluation and Learning Plan

AMELP Activity Monitoring and Evaluation Plan

AMERP Activity Monitoring, Evaluation, and Research Plan

AOR Agreement Officer’s Representative

BEST Building Economic Sustainability through Tourism

CBIWDM Community-Based Initiative for Water Demand Management

CDCS Country Development Cooperation Strategy

CEP Community Engagement Program

CEPPS Consortium for Elections and Political Process Strengthening

CIS Civic Initiatives Support Program

CISLE Cultivating Inclusive and Supportive Learning Environments

CITIES Jordan Cities Implementing Transparent, Innovative, and Effective Solutions

Activity

CLA Collaboration, Learning and Adaptation

CoP Community of Practice

COR Contract Officer’s Representative

CSS Center of Strategic studies

DEC USAID’s Development Experience Clearinghouse

DO Development Objective

DOC Development Outreach Communications

DQA Data Quality Assessment

DRG Democracy, Rights, and Governance

EDE Economic Development and Energy

EDY Education and Youth

EGMA Early-Grade Mathematics Assessment

EGRA Early-Grade Reading Assessment

EMIS Educational Management Information System

ESCB Energy Sector Capacity Building

FAITAS Federal Acquisition Institute Training Application System

FRP II Fiscal Reform Project II

FRPFM Jordan Fiscal Reform and Public Financial Management

GIS Geographical Information System

GoJ Government of Jordan

HFG Jordan Health, Finance and Governance

HO Home Office

HRH 2030 Human Resources for Health 2030 Activity

HSD Health Services Delivery

IFES International Foundation for Electoral Systems

IP Implementing Partner

IRI International Republican Institute

JCAP Jordan Communication, Advocacy and Policy Activity

JCP Jordan Competitiveness Program

JCD ` Jordan Customs Department

JLGF Jordan Loan Guarantee Facility

JSEP Jordan School Expansion Project

KaMP Knowledge Management Portal

KPI Key Performance Indicator

USAID/Jordan MESP – Annual Report (FY 2017) Oct 2016 – Sept 2017 iii

LEIIP Learning Environment: Improved Infrastructure Program

LENS Local Enterprise Support Activity

LETS Learning Environment Technical Support

M&E Monitoring and Evaluation

MESC Management Engineering Services Contract Activity

MESC Non-Revenue Water Project (for NRW)

MESP Monitoring and Evaluation Support Project

MIS Management Information Systems

MoE Ministry of Education

MoL Ministry of Labor

MoPIC Ministry of Planning and International Cooperation

M-PMP Mission Performance Management Plan

MSI Management Systems International

NFE Non-Formal Education

NRW Non-Revenue Water

OCI Organization Conflict of Interest

P-MEP Project M&E Plan

PAP Public Action for Water, Energy, and Environment Project

PDT Performance Data Table

PIRS Performance Indicator Reference Sheets

PITT Performance Indicators Tracking Table

PMP Performance Management Plan

PRO Program Office

PPR Performance Plan and Report

QR/AR Quarterly Report/Annual Report

R Result

RAMP Early Grade Reading and Math Project

RCT Randomized Controlled Trial

RF Results Framework

ROLPAS Rule of Law and Public Accountability Strengthening

SOW Statement of Work

SMEs Small and Medium Enterprises

TNA Training Needs Assessment

ToC Theory of Change

ToT Training of Trainers

TVET Technical and Vocational Education and Training

UCYI UNICEF Child, Adolescents and Youth Initiatives in Jordan

WFD Jordan Workforce Development and Enterprise Support Project

WMI Water Management Initiative

WRE Water Resources and Environment

WREC Water Reuse and Environmental Conservation

YWP Youth with Potential

USAID/Jordan MESP – Annual Report (FY 2017) Oct 2016 – Sept 2017 iv

TABLE OF CONTENTS

List of Acronyms ................................................................................................................................................................ ii

Introduction

Key Accomplishments

Detailed MESP Results

MESP Goal: Enhanced utilization of quality data to effectively assess progress towards the achievement of the CDCS (by Mission staff)

R1. Strengthened PMP Implementation and Utility

R2. Project/Activity Evaluation Strengthened

R3. Effective Communication of Selected Mission Results Implemented

Option 2

MESP Collaboration, Learning, and Adaptation (CLA) Implementation

Management and Administration

Organizational Conflicts of Interest (OCI)

End Use Audit

Subcontractors

Administration and Procurement

Personnel

Constraints and Limitations

Key Planned Activities for the Coming Quarter

ANNEX I: Cumulative List of Major Project Deliverables

ANNEX II: MESP M&E Technical Support to Activities to Date

ANNEX III: MESP M&E Technical Support to Mission to Date

ANNEX IV: MESP Performance Data Table (PDT)

ANNEX V: DevResults Modifications

USAID/Jordan MESP – Annual Report (FY 2017) 1

INTRODUCTION

Management Systems International (MSI) began the United States Agency for International Development

(USAID) Jordan Monitoring and Evaluation Support Project (MESP) on Dec. 1, 2013, under contract

AID-278-C-13-00009. The principal purpose of the work is to provide technical and advisory services to the USAID/Jordan Program Office (PRO), USAID/Jordan Development Objective (DO) teams, and implementing partners (IPs) in the areas of project monitoring, evaluation, research, organizational learning, and knowledge management to enable USAID/Jordan to fulfill its performance and evaluation requirements.

As MESP entered its fourth year, it focused on delivering higher-level, more intensive support designed to enhance the availability, accessibility, and utilization of quality M&E data by Mission and IP managers. A key theme embraced this year was Collaborating, Learning, and Adapting (CLA). In Year 4, MESP reviewed its work and practices through a CLA self-assessment and as a result, integrated CLA practices, concepts and approaches throughout its work plan to further strengthen data utilization and management of CDCS.

Demonstrating its commitment to effective learning and adaptive management, MESP updated its annual capacity building plan and technical assistance activities, conducting newly developed workshops that incorporated the lessons learned and feedback received during previous sessions and responded to emerging Mission and IP priorities. This year MESP launched a three-day workshop on Causal Logic/

Adaptive Management for Mission and M&E IPs staff, focusing on facilitating understanding of the

Activities’ detailed underlying causal logic as part of the logic model, key results, performance indicator, and learning questions development process. MESP also introduced a Data Visualization workshop which was internally piloted and the first session conducted in Q4. Additionally, responding to MESP’s CLA self-assessment, MESP launched regular drop-in sessions that are designed for IP M&E staff in certain topics, and also aimed to strengthen relationships with IPs to further the reach and effectiveness of

MESP support. Moreover, MESP received approval and began work on additionally workshops to be offered in FY2018, including Most Significant Change (MSC), Theory of Change (ToC), and Data

Management (including the use of dashboards).

Directly contributing to enhancing the quality of IP reported MEL data, MESP M&E Specialists provided technical assistance to the Mission and IPs, including feedback on 16 AMELPs developed and/or revised by IPs. The majority of MESP’s efforts focused on facilitating Mission and IP understanding of the

Activities’ detailed underlying causal logic as part of the logic model, key results, performance indicators, and learning questions development process. In nearly all instances, AMELP reviews and feedback was clarified and honed in a team with MESP and PRO staff members, a collaborative approach to learning that provides the most opportunities for joint understanding and capacity building.

Collaboration and collective learning also continued to be a prominent focus of MESP’s work in Year 4.

Through experience in previous years, collaborative design and implementation processes have proven to be the most effective means of developing support that meets IP and Mission interests and needs.

The value of MESP’s proactive approach was demonstrated again this year in the successful implementation of the USAID/Jordan 2017 MEL Conference from May 8-9, 2017. The conference involved over 120 MEL practitioners from IPs across USAID’s portfolio. With key support from the

Mission, including Mission Director Dr. Jim Barnhart providing opening remarks, the conference brought together international and local subject matter experts including several high-level representatives of

USAID/Washington Offices and global support activities. The 2017 conference benefitted from a number of adaptations to the original design informed by participant feedback. Most notably of these was the inclusion of a dedicated collaborative learning space, “MEL Expo,” within the larger conference hall where 15 IPs prepared and hosted booths to showcasing their activities’ results and MEL work.

USAID/Jordan MESP – Annual Report (FY 2017) 2

Encouraged by the Mission and PRO, MESP began the implementation of the second round of the M&E

Apprenticeship Program, which is intended to rapidly transform participants into effective entry-level

Activity M&E officers, represents a key example of the importance of MESP and IP collaboration in identifying and addressing needs and challenges IPs face in recruiting qualified M&E staff, with the appropriate skills to quickly succeed in an M&E role. Furthermore, this year’s Community of Practice

(CoP) sessions served to further reflect participatory learning and responsive adaptation, as the session topics were selected by the IPs and were met with overwhelming interest and participation.

As part of its continued commitment to ensuring the availability of required M&E data for Mission decision-making, MESP completed, conducted, and/or initiated a total of seven evaluations and two assessments. Progress this year included the final approval of the SDO4 and JLGF evaluations, and the launch of designs for a number of evaluations and assessments including a performance evaluation of

BEST, the CITIES impact evaluation, General Population Survey, and the Economic Growth Assessment.

Data collection activities for the RAMP impact evaluation midline also successfully concluded.

Additionally, to more effectively measure its progress against MESP’s highest level goal, MESP team completed the Evaluation Utilization Assessment of the CIS Midterm Evaluation, and also began the utilization assessment of the JLGF evaluation by sharing the evaluation utilization matrix and questions with USAID and the JLGF team. Both assessments found a high rate of adoption and consideration of evaluation recommendations for current and future programming at the USAID and IP levels.

Further support to utilization of data at the Mission level was also accomplished through continued implementation of DevResults, the Mission’s Management Information System (MIS), as well as

Geographic Information Systems (GIS) support. As a result of DevResults, 60 percent of those interviewed at the Mission reported that IP indicator data is more available. MESP continued to support the Mission’s expanded GIS capabilities, including generating maps required for different Mission technical offices. PRO and MESP conducted three GIS training workshops for IPs to highlight the importance of collecting up-to-date GIS data, and show IPs how to best report GIS coordinates and corresponding activity-level details to ensure standardized, comprehensive and timely availability of geospatial data.

Additionally, USAID/Jordan, with the support of MESP, achieved a key milestone this year in supporting increased data availability, accessibility and utilization. On March 15, 2017, USAID/Jordan officially launched KaMP, the Jordan Development Knowledge Management Portal, to further expand the reach of KaMP. Overall, KaMP met its objectives this year by increasing the number of USAID active Activities to 29 (75% of active activities) and non-USAID partners by 24 organizations. As of the end of

September, there are now 147 registered users, 2,974 resources on KaMP, with visitors holding an average of 1,158 sessions per month, with 66% of sessions, on average, coming from new visitors.

KEY ACCOMPLISHMENTS

Training for the Mission and IPs

• Conducted five rounds of a three-day workshop focused on Causal Logic and Adaptive

Management with 4 Mission staff and 38 IPs staff with a total of 1008 training hours.

• Piloted and conducted the Data Visualization Workshop with 12 IP staff with a total of 42 training hours.

• MESP and PRO staff participated in and were trained as facilitators on the CLA self-assessment process, and co-facilitated the process for the Youth Task Force.

USAID/Jordan MESP – Annual Report (FY 2017) 3

• Hosted a complimentary post-conference workshop delivered by MECap The road of better fit:

Skills building series (Identifying Opportunities for Complementary M&E of USAID programming with a total of 16 attendees/128 training hours.

• Conducted 15 workshops with a total of 176 participants and 1549 training hours.

• Developed and received approval from PRO on the concept notes for MSC, TOC training workshops.

Technical Assistance for USAID PMP

• Supported PRO in DQA visits.

• Generated reports from DevResults for PPR and Portfolio review.

• Uploaded the revised Activity MEL plan template on KaMP.

MEL Technical Assistance for IPs

• Held 2017 USAID/Jordan MEL Conference.

• Concluded Round One of the Apprenticeship Program, resulting in 12 out of 13 apprentices securing jobs. All 8 seconded apprentices were transferred to an M&E title/position, and three out of five non-seconded apprentices were hired at the IP that hosted them.

• Initiated Round Two of the Apprenticeship Program with 15 apprentices

• Conducted two CoP sessions attended by 82 IPs for Activities under EDY, EDE, WRE, DRG, GNDR and PFH.

• Reviewed, discussed, and/or provided feedback to the following IPs in support of developing and enhancing their MEL plans:

o Youth with Potential (YWP) o Youth Power (YP) o Health, Finance and Governance (HFG) o Sustainability Cultural Heritage Engagement Project (SCHEP) o Workforce Development (WFD) o Jordan Competitiveness Program (JCP) o Jordan Loan Guarantee Facility (JLGF) Technical Assistance Activity o Rule of Law and Public Accountability Strengthening Program (ROLPAS) o Cities Implementing Transparent Innovative, and Effective Solutions (CITIES) o Human Resources for Health 2030 (HRH2030) o Health Service Delivery (HSD) o Jordan Communication Advocacy Project (JCAP)

• Assisted JCAP in their KAP survey.

Establishment of Mission MIS/GIS

• Launched the three DevResults major visualization overhauls: the new Indicator Targets vs

Actuals graph

USAID/Jordan MESP – Annual Report (FY 2017) 4

• Conducted three DevResults IP training sessions: two introductory sessions, and one advanced session

• Introduced IPs to the new dashboards in DevResults

• Reflected all Mission PMP revisions/changes on DevResults, including all Mission indicator targets

• Trained 9 activities on DevResults: NFE, ROLPAS, WMI, HSD, WSIP, SEED, FRPFM, and YWP

• Supported Mission in the portfolio review by generating Mission indicator reports from

DevResults for the WRE portfolio

• Produced maps for the Mission’s fall 2016 Portfolio Review, and to support PRO by producing maps with existing IPs’ GIS data

• Conducted three GIS IP training sessions with 23 participants

Evaluations

• Finalized JLGF evaluation report which was approved by the Mission and uploaded on the DEC and KaMP

• Responded to Mission comments, and shared the second draft of the CEP evaluation

• Received Mission approval on SDO4 evaluation report

• Submitted the draft report of Takamol evaluation

• Finalized with PRO the BEST evaluation SOW for comments and feedback

• Submitted the RAMP IE draft baseline report to USAID

• Rolled-out and concluded EGRA/ EGMA mid-line data collection in eight governorates across

Jordan. As part of the RAMP midline data collection, teacher observation data was collected for a descriptive and impact study.

• Conducted two drop-in sessions on “Preparing for Evaluations” attended by 37 IPs for Activities under EDY, EDE, WRE, DRG, GNDR and PFH

• Conducted utilization review of CIS with the AOR and CIS team, and submitted the CIS evaluation utilization assessment report.

• Received Mission and JLGF responses to the JLGF evaluation utilization matrix

Assessments, Studies and Surveys

• Job Creation Assessment approved and uploaded to DEC and KaMP

• Responded to Mission comments, and shared the second draft of the JCD assessment report

• Submitted an updated draft of the assessment SOW of the Economic Growth and

Competitiveness Assessment

USAID/Jordan MESP – Annual Report (FY 2017) 5

Communications

• Supported PRO in launching Jordan Development Knowledge Management Portal (KaMP) to different stakeholders

• Produced and disseminated a short video on KaMP

• Launched KaMP first competition, and shared with IPs the KaMP Competition brochure

• Announced the first round of KaMP competition winners during USAID/Jordan 2017 MEL

Conference

• To date, 2974 useful resources uploaded to KaMP

• Produced the MIS/GIS one-pager, as an introductory document for new Activities

• Produced and uploaded to KaMP the two-pager for the CIS project mid-term evaluation in

English and Arabic

• Translated and uploaded to the DEC the Arabic version of the Civil Society Assessment report

• Disseminated the third and fourth editions of MESP ‘MEL Matters’ newsletter to Mission and IP staff

• Finalized and disseminated the 2016 M&E Apprenticeship program video

Collaboration, Learning and Adaptation

• Submitted three case studies for the CLA competition, one of which, developed on behalf of

PRO, was selected as a finalist in the global competition

• Developed a CLA action plan to strengthen CLA practices within the USAID/Jordan community and further strengthen data utilization and management of CDCS

• Submitted and received approval on the concept note for “EDE pause and reflect” session

• Collaborated with PRO and IP staff to address recruitment challenges within M&E in Jordan and as a result planned, developed and implemented the Apprenticeship Program

• Met with IP M&E staff in the planning and development of the USAID/ Jordan 2017 MEL

Conference

• Introduced focused “Drop-in Sessions” on particular topics to strengthen M&E practices within

IPs

USAID/Jordan MESP – Annual Report (FY 2017) 6

DETAILED MESP RESULTS

MESP Goal: Enhanced utilization of quality data to effectively assess progress towards the achievement of the CDCS (by Mission staff)

G L

-0

Percent of surveyed Mission technical management staff reporting agreement that their utilization of M&E data to effectively assess progress towards achievement of the CDCS has improved since the last survey*

Baseline

(FY2015):

88.2%

FY 2016 Actual:

75%

FY 2017 Target:

80% FY 2017 Actual:

66.7%

*Indicator collected on an annual basis

MESP conducted the third round of the Annual Mission staff survey, the objective of which is to assess and document changes, potentially resulting from MESP interventions, in the utilization and availability of required performance data for management to achieve the Mission’s DOs. While the reported results for this indicator did not reach the estimated target, the majority of respondents reported improvement in this area, 66.7%, since last year, which indicates that the Mission continues to make progress in this critical area of practice.

The primary method of data collection is an online survey using 18 questions to elicit the feedback of

Mission staff who oversee Activities as part of their job. In order to maximize Mission ownership of and participation in the survey, PRO sent the online survey link via email to 30 Mission staff including PRO, AOR/CORs, and M&E focal points. Twelve Mission staff responded to the survey (40 percent response rate), and captured all six of the Mission’s technical offices (Water Resources and Environment (WRE), Population and Family Health (PFH), Economic Development and Energy (EDE), Education and Youth

(EDY), Democracy Rights and Governance (DRG), and Gender).

Responses to the survey provide data on a variety of performance indicators tracked by MESP. This year, the survey tool was modified to reflect indicators related to KaMP access. The indicators are as follows:

R 3.3 Percent of surveyed users responding Agree/Strongly Agree (on a five point Likert scale) to “My access to USAID M&E information including templates and tools is improved by access to the KaMP.”

R 3.4 Percent of surveyed users responding Agree/Strongly Agree (on a five point Likert scale) to “My access to information related to the development sector in Jordan is improved by access to the KaMP.”

Additionally, MESP included the A/COR DevResults learning agenda.

USAID/Jordan MESP – Annual Report (FY 2017) 7

As demonstrated in Graph A above, 66.7 percent of respondents indicated that they ‘agree,’ or ‘strongly agree,’ that the performance area had improved over the past 12 months, or since starting work with

USAID for those respondents who had worked less than 12 months in the Mission. This percentage of reported improvement is a general indication that performance in this critical area of Mission staff technical work is moving in a positive direction, though not as positive a change reported last year.

Although the 2017 reported results is an eight percentage point reduction from reported 2016 results, there was also a considerable increase (10 percentage points) in those respondents who Strongly Agree with the statement in 2017 (25 percent versus 15 percent). In addition to improvements to the response scales, several open-ended questions were added to the survey to provide more insight and depth into the availability and utilization of performance data. However the responses to the open ended questions were very limited and didn’t provide much insight.

25%

41.70%

25%

8.30%

15%

60%

15%

10%

Strongly agree

Agree

Neither agree or disagree

Disagree

Strongly disagree

Graph A. Mission Survey 2016 and 2017 results

2016 2017

Lesson Learned

MESP included open-ended questions in order to solicit greater understanding of the changes throughout the year, but the responses were limited. MESP believes that conducting in-depth interviews with Mission staff would have been more useful in yielding more detailed feedback and insight.

USAID/Jordan MESP – Annual Report (FY 2017) 8

R1. Strengthened PMP Implementation and Utility R

.1

Average rating of the availability of performance monitoring data that the surveyed mission staff need to do their job (on a scale from 1 to 10, one being no available data and ten being data is readily available)

Baseline*:

5.2

FY 2016 Actual:

6.8

Q4 FY 2017:

7.5

FY 2017 Actual†:

7.5

FY 2017 Target:

* Baseline was established at the same time as the 2015 data collection, asking respondents to retroactively rate the availability of performance monitoring data.

†Indicator collected on an annual basis.

Availability of Mission M&E Data for Decision-making

As described above, the M&E survey of Mission staff is designed to provide performance data in a variety of MESP support areas. Working in close partnership with PRO, MESP technical assistance is intended to support the Mission’s utilization of M&E data for portfolio management in multiple ways, including improving the availability of required performance monitoring data through the population of

DevResults. In order to document changes in data availability potentially resulting from MESP interventions, MESP’s revised AMELP includes the following performance indicator:

R 1.1 Average rating of the availability of performance monitoring data that the surveyed mission staff need to do their job (on a scale from one to 10, one being no available data and ten being data is readily available).

The continued positive trend for this indicator and other data collected during this year’s survey lends support to the belief that interventions in this area are having positive results.

Respondents of the Mission Survey were asked to rate the availability of performance data required to conduct a portfolio review or other M&E-related task data at the present time. The rating scale ranged from one (not available) to 10 (readily available). Last year, the average response to this question was 6.8 with a baseline of 5.2 (estimated retrospectively by the same respondent), while in response to the survey in 2017, the average was calculated as 7.9 (mode of seven with five responses, and a range from three to 10). This may demonstrate that positive change is taking place in targeted areas.

5.2

6.2

6.8

7.5

0 2 4 6 8 10

Graph B. Mission Survey 2017 Average respondent rating out of 10 of the availability of "performance monitoring data that you will need to do your job." (N=12)

Baseline 2015 2016 2017

USAID/Jordan MESP – Annual Report (FY 2017) 9

Additional questions were included in the survey to give more depth and understanding to the changes in the availability of performance monitoring data. When asked “Since you started at the Mission, has the availability of performance monitoring data that you need to do your job (for example, if you were performing a portfolio review or other M&E tasks) improved, worsened or stayed the same?” 66.7 percent of respondents reported that the availability of performance monitoring data had improved since they started at the Mission (Graph C) compared to 65 percent last year.

M&E Mission Training and Technical Assistance

Mission Staff Perceptions of MESP Technical Support

Over the course of three years of implementation, MESP has provided support to Mission staff through a combination of technical assistance, tools, guidelines, and workshops that are designed to improve

Mission and IP performance in MEL practices. MESP’s annual Mission survey includes several questions intended to help MESP and PRO understand the perceived utility of MESP support. As with last year’s assessment, the results of this year’s data collection to date are positive and indicate a high level of perceived utility by Mission staff of MESP’s technical support.

Mission staff were asked to state their level of agreement with the following statement: “MESP support helped improve my ability to implement M&E tasks.” Eighty-three percent (10 out of 12 respondents) indicated either agreement or strong agreement in this area (Graph D), compared to the previous years’ survey the results of 75 percent of respondents.

66.7%

65%

33.3%

25% 5% 5%

Graph C. Mission Survey 2017

“Since you started at the Mission, has the availability of performance monitoring data that you need to do your job improved, worsened or stayed the same?"

Improved Stayed the same Worsended Don’t know

15%

41.60%

60%

41.60%

20% 5%

8% 8%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

GRAPH D. Mission Survey 2017

To what extent do you agree with the following statement:

"MESP support helped improve my ability to implement

M&E tasks" (N=12)

Strongly agree Agree Neither agree or disagree Disagree Strongly disagree

USAID/Jordan MESP – Annual Report (FY 2017) 10

The survey also included questions intended to assess perceptions related to specific areas of technical support provided by MESP. One area of intensive work by MESP staff is in the area of AMELP reviews.

In order to limit responses to staff that had actually worked with MESP in this area, a filter question was first used to determine which respondents had overseen Activities that received MESP support with their AMELPs, of which slightly more than half had done (11 out of 20). Of these 11 respondents, all but one, or 91percent responded ‘yes’ when asked, ‘Do you think that MESP technical assistance helped produce a better product?’ This high level of perceived utility is a strong endorsement of MESP’s value in helping Mission managers work with IPs to produce higher-quality AMELPs. In addition, respondents also mentioned perceived improvements in portfolio reviews (5 responses) and DQAs (6 responses). While

MESP cannot directly attribute the positive changes to its support based on the survey results, MESP does provide support to these tasks.

Training for Mission Staff

R

.2 .5

Number of person-hours of training in Monitoring and/or Evaluation conducted by MESP

Baseline:

FY 2016 Actual:

Q4 FY 2017*:

371.5

FY 2017 Actual:

FY 2017 Target:

*Mission Staff: 27.5 hours IP Staff: 344 hours

Causal Logic/ Adaptive Management Training

Through MESP’s technical assistance and trainings in previous years, the need for additional capacity building on causal logic, indicator selection and adaptive management for Mission and IP staff was identified. To address the need, MESP developed a new three-day workshop focused on causal logic and adaptive management. Workshop materials were completed and reviewed by PRO and

USAID/Washington.

In October 2016, MESP conducted a pilot M&E training workshop with support from the MSI Home

Office for nine MESP and Mission staff (Table A). The pilot training was an opportunity to review the training materials, methodology, exercises, and other aspects of the training, and also provided feedback to improve, and make more Jordan context specific. It also served as a Training of Trainers (ToT) training for MESP staff in order to be able to organize future similar trainings for Mission and IP staff locally.

TABLE A: FY 2017 Training Workshops for Mission

Workshop

Title

Workshops

Offered

Person Hours/

Workshop

Attendance

Total M F Total Person Hours

Causal Logic/

Adaptive

Management 1 24 4 1 3 80

Following the successful delivery of the pilot workshop, MESP delivered four rounds of training sessions.

The workshops were attended by 42 participants; four Mission staff, resulting in 80 hours of person hours of training for Mission staff.

USAID/Jordan MESP – Annual Report (FY 2017) 11

A FAITAS application for the "Causal Logic and Adaptive

Management" training was also submitted and approved for 20

Continuous Learning Points (CLPs), where this course was published on the list of approved CLPs for Mission staff on March 31, 2017.

Further details on the content and results of the training sessions can be found under the “Workshops for IPs” section.

Mission Staff Technical Assistance

Review of Mission Site Visit Reports

At the request of PRO, MESP reviewed a sample of 50 site visit reports completed by Mission staff as part of their Activity management and data quality assurance responsibilities. The review was intended to provide feedback to PRO on the quality of information included in the reports. MESP’s M&E specialists compared the information in completed field visit reports with the template and guidance. The process helped identify potential methods of improving what and how information is reported, in addition to several areas of potential action in order to strengthen the effectiveness, efficiency, and utility of these visits, this includes providing a clear guidance in the template, MESP shared with the Mission a summary report covering the results of the reviews and priority observations and findings from the reviewed reports per DO.

Review of IP Quarterly and Annual Reports

At the request of PRO, USAID/Jordan Activities were scanned for accessible/available reports, MESP also checked the availability of the PITT, which is important for cross-checking DevResults and supporting the narrative around results. As part of data quality and consistency MESP is in the process of reviewing performance indicator data for a sample of seven Activities looking at Q2 and Q3 FY2016 performance data reporting to identify any issues or discrepancies compared with data in DevResults.

In order to systemize the reviewing process, MESP developed a tracker which captures and summarizes the key fields mentioned above. As part of the review process, MESP conducted a quality and consistency check of performance data for a sample of seven Activities which report on DevResults.

Performance data for Q2 and Q3 FY2016 were compared between quarterly reports PDT/PITT and

DevResults. MESP also looked into whether Activities are reporting any requests for revisions or additions to the data reported on DevResults (approved or pending) are being reported within the quarterly reports.

In general, MESP found a range of types of PITTs integrated into quarterly reports, some more comprehensive than others, but only one Activity has the PITT/PDT as a separate attachment, six

Activities report their indicator data in the report narrative, and have no PITT/PDT included, and five out of the 16 Activities which consistently incorporate PITT/PDT in their reports started to utilize

PITT/PDT post quarter one 2016. Moreover there are discrepancies in data reported between DevResults and PDT/PITT for most Activities.

Lessons Learned

In the process of reviewing IP quarterly reports, MESP discovered that data reported on DevResults may not match data reported by Activities in finalized quarterly and annual reports from the same period, and there may be no log of revisions made to the data available for review that could explain differences between data in a specific report and more recent data in DevResults.

USAID/Jordan MESP – Annual Report (FY 2017) 12

Support to DQAs

In Years 2 and 3 MESP provided technical assistance, tools, and guidelines targeted to improve the preparation and performance of IP and Mission staff in conducting DQAs. As capacity strengthened within the Mission around DQAs, MESP shifted its approach to a more observational role, with staff attending some DQA sessions conducted in FY 2017, providing support to Mission and IP staff as needed.

CLA Self-Assessment Workshops

Stemming from the MEL Conference in May, MESP began more deeply exploring USAID’s CLA

Framework. Capitalizing on the momentum of the conference and the presence of experts from

Washington, D.C. (Stacy Young, USAID Bureau for Policy, Planning and Learning, and Jessica Zeigler, USAID/LEARN Contract), six MESP and two PRO staff participated in and were trained as facilitators on the CLA self-assessment process. Originally designed to support USAID Missions around the globe, the process and associated Maturity Matrix tool aims to support actors “to think more deliberately about how to plan for and implement CLA approaches that fit [the actor’s] context and assist them in achieving their development objectives.”1 The Maturity Matrix tool cards, an example of which is pictured above, explains the spectrum of CLA practice, from Not Yet Present to Institutionalized, within each CLA sub-category, and supports actors through a series of exercises to identify, discuss, and prioritize CLA practices and action plans.

In undergoing the process, MESP and PRO selected several subcomponents: Technical Evidence Base, Adaptive Management, and Relationships and Networks. Ultimately, the Maturity Matrix Tool and associated discussions helped MESP and PRO recognize where we were and where we wanted to go with our CLA practice. Primarily, MESP and PRO perceptions of its current practices ranged from emergent to advanced in the selected sub-categories, and there was a consensus to continue to strengthen our practices in the selected areas. To this end, through the self-assessment process, MESP and PRO jointly developed a concrete CLA action plan that focused on ways to raise awareness and strengthen CLA practices within the USAID/Jordan community, particularly focusing on Mission staff and

IPs. These action plans have since been prioritized and integrated into MESP’s work plan. For more details on the results of the Self-Assessment process, see the

Jordan Self-Assessment and Action Planning report produced by

USAID/LEARN. Overall, it was a refreshing opportunity to discuss strengths and shortcomings related our CLA practices, brainstorm innovative ideas to support CLA within the

USAID/Jordan community, as well as integrate CLA throughout MESP’s activities.

Following this experience and facilitation training, MESP co-facilitated the process with USAID/PPL, USAID/LEARN and

PRO for the USAID/Jordan Youth Task Force. A total of seven members of the task force, representing EDE, EDY, WRE, PFH, DRG and Program Office, participated in the CLA Self-

Assessment process. To focus the discussion, participants selected the following sub-components, Internal Collaboration, Adaptive Management, and Relationships and Networks.

1 USAID Learning Lab, https://usaidlearninglab.org/library/collaborating%2C-learning%2C-and-adapting-cla-framework-and-maturity-matrix-overview.

CLA SELF-ASSESSMENT

✓ What areas of the CLA

Framework resonate with you?

✓ Where do you think your practices are along the CLA spectrum – ‘not yet present’ to

‘institutionalized’?

✓ Where do you want your practices to be in the future?

✓ What can you do to strengthen your CLA practices and your

Activity effectiveness?

USAID/Jordan MESP – Annual Report (FY 2017) 13

Within these areas, perceptions on current practices ranged from not yet present to emergent and expanding, with aspirations tending toward reaching the expanding or advanced levels of practice.

Facilitators then guided the task force representatives through the action planning process, ultimately developing 5 action items to be completed within the next 12 months. Action items focused on clarifying the task force’s purpose, mapping activities and stakeholders to improve coordination and strategic decision making, as well as updating and realigning USAID’s youth strategy to clarify USAID’s vision around supporting youth.

Through both these workshops, the self-assessment process and tool were appreciated and useful, and important lessons learned were identified.

MEL Support for Implementing Partners

R

Percent of interviewed IP technical assistance recipients that report that MESP support helped them to implement targeted M&E tasks

Baseline:

FY 2015 Actual

100%

FY 2016 Actual:

100%

FY 2017 Actual:

100%

FY 2017

Target:

75%

*Indicator collected on an annual basis

R

Percent of surveyed IP staff that report that MESP training workshops were helpful for them in doing their job

FY 2015 Actual:

100%

FY 2016 Actual:

84.6%

FY 2017 Actual*:

100%

FY 2017 Target:

80%

*Indicator collected on annual basis

R

.5

Number of person hours of training in Monitoring and/or Evaluation conducted by MESP

FY 2016 Actual:

Q4 FY2017 Actual*:

371.5 FY 2017 Actual†:

FY 2017

Target:

†Mission Staff: 27.5 hours IP Staff: 344 hours

Lessons Learned

• Participants should have a sufficient or deep understanding of the CLA framework, its subcomponents, and why CLA is useful to better support meaningful discussions on current practices and effectively prioritize actions;

• Greater clarity on the final look of the CLA Self-Assessment product (action plan) may also help participants provide relevant and effective input;

• Facilitators should have an understanding of the context in which the team undergoing the self-assessment process operates, to effectively guide the discussion and provide input where needed;

• Decision makers should be involved and integrated into the process to develop effective action plans that will be implemented; and

USAID/Jordan MESP – Annual Report (FY 2017) 14

IP Perceptions of MESP Technical Support

Over the course of the four years of implementation, MESP has provided support to Implementing

Partners (IPs) through technical assistance, tools, guidelines, and workshops that are designed to improve IPs’ performance in MEL practices. The IP assessment is conducted annually and aims to assess the utility and effectiveness of technical support provided by MESP, identify additional needs for technical support, and discuss the challenges IPs currently face in their MEL work. As with last year’s assessment, the results of this year’s data collection to date are very positive and indicate a high level of perceived utility by IPs of MESP’s technical support and workshops. In-depth interviews, which began in

September, consisted of interviews with 13 IP MEL staff, and the responses helped in collecting data related to MESP below indicators:

R 1.2.1 Percent of interviewed IP technical assistance recipients that report that MESP support helped them to implement targeted M&E tasks.

R 1.2.2 Percent of surveyed IP staff that report that MESP training workshops were helpful for them in doing their job.

R 1.2.4 Percent of interviewed IP staff who report that MESP-developed tools helped them implement performance monitoring tasks

This year the tool was modified to reflect indicators related to KaMP access. The indicators are as follows:

R 3.3 Percent of surveyed users responding Agree/Strongly Agree (on a five point Likert scale) to “My access to USAID M&E information including templates and tools is improved by access to the KaMP.”

R 3.4 Percent of surveyed users responding Agree/Strongly Agree (on a five point Likert scale) to “My access to information related to the development sector in Jordan is improved by access to the KaMP.”

In addition to collecting indicator data, the objectives of the IP assessment are to provide a greater understanding and insight to the following:

• Utility of MESP support

• Challenges facing IPs in M&E (identified needs/gaps)

• Changes/achievements influenced by MESP

• Needs assessment/calculating baseline to specific M&E practices

Respondents were asked to rate whether MESP technical assistance and workshops were helpful to do their jobs. This year, the average response to the question of whether MESP’s technical assistance and workshops are helpful to do their jobs was 100 percent, similar to last year.

Training for IPs MESP continued to provide training for USAID/Jordan IP staff to enhance the availability of performance data reported to the Mission. Building on the success of the previous years, in FY 2017, MESP delivered

15 training workshops for IP staff. The workshops were attended by 176 people2 representing 38

Activities for a total of 1549 person-hours of training. While this number is more than the target of 612 hours, the difference is due to conscious delays in the delivery of the Causal Logic/Adaptive Management workshop into FY 2017 based on substantial and essential collaboration on the details of the course

2 This is the overall total across all trainings. The total may include double counting.

USAID/Jordan MESP – Annual Report (FY 2017) 15 content and delivery approach. In addition, while it isn’t reflected in the reported numbers, MESP actually delivered more than 1,500 additional person-hours of training to IP staff in M&E through the

Apprenticeship workshops.

TABLE B: FY 2017 Training Workshops for IPs

Workshop Title Workshops

Offered

Person

Hours/

Workshop

Attendance Unique

Activities

Participating Total M F Total Person

Hours

Causal Logic/ Adaptive

Management 4 24 42 12 30 1008 22

AMLEP/PIRS 3 3.5 32 5 27 112 17

Selecting Indicators 2 3.5 31 5 26 108.5 12

Preparing for DQAs 4 3.5 43 12 31 150.5 26

Data Visualization 1 3.5 12 1 11 42 9

The Road to Better Skills 1 8 16 5 11 128 12

Total 15 176 40 136 1549 38

Causal Logic/ Adaptive Management Workshop

As mentioned above, the aim of the Causal Logic/Adaptive Management Workshop is to build Mission and IP staff capacity to practice adaptive management effectively through understanding, articulating, and periodically updating the projects’ and Activities’ underlying cause and effect logic in response to emerging evidence and learning. The main objectives of the workshop were to help participants:

• Have strong skills in articulating the underlying activity theory of change and understand and troubleshoot various types of activity logic models, including results frameworks, log-frames and others;

• Apply principles of “cause and effect” and “sufficient and necessary” in identifying and measuring key activity results, including in situations where the underlying cause and effect relationships are not well understood;

• Understand principles for selecting useful performance and contextual indicators for strategies, projects and activities; and

• Conduct data driven performance/portfolio reviews including analyzing monitoring data to inform adaptive management.

• To ensure the content of the training effectively engaged and met the needs of participants, MESP conducted a pilot workshop, which provided valuable insights regarding the practicality of the training materials, delivery methods, exercises/case studies and other aspects of the training. Following the pilot workshop, MESP delivered four rounds of training

USAID/Jordan MESP – Annual Report (FY 2017) 16 sessions strengthening capacity around causal logic/adaptive management for 42 participants.

Participants include: 4 Mission staff, 3 MESP subcontractor staff from Mindset and Integrated, 35

MEL practitioners from 22 different Activities. In total, MESP provided 1,008 person hours of training through the Causal Logic/Adaptive Management Workshop. During the training, case studies and group work exercises proved to be effective, as the participants were able to relate to workplace real-life examples, enabling them to apply newly acquired knowledge. In addition, many senior participants helped create a progressive learning environment as they injected their own experiences throughout these trainings. Such an environment helped other participants to enhance their learning through understanding the practical application of a number of topics, while examining a variety of modern M&E techniques. Due to great interest shown by Mission and IPs staff in the Causal Logic/ Adaptive Management training workshop, MESP will be organizing further training sessions on the topic in FY2018.

Introduction to Activity M&E Learning Plans and Documenting PIRS (AMELP/PIRS)

Workshop

The aim of the AMELP/PIRS workshop is to orient IP staff on how to develop and/or review their

AMELP, and to provide an overview of the USAID/Jordan IP PIRS template with specific guidance for completing each key section. More specifically, the objectives of the workshop are to help participating IP staff:

• Understand key USAID M&E terms;

• Distinguish between the different levels of M&E planning at USAID Jordan;

• Identify main components of AMELPs; and

• Work with other staff members to document their Activity PIRS.

The content and examples used in the workshop are adapted from real-life examples, including common problems with indicators and missing AMELP elements that the Mission has recently encountered. Some of the terminology was revised based on guidance received from PRO according to changes to ADS, for example, using the term “Activity Monitoring & Evaluating Learning Plan” instead of the term “Activity Monitoring & Evaluating Plan”.

During FY 2017, MESP conducted three AMELP/PIRS workshops that were attended by 32 (27 F/ 5 M) participants representing 17 different Activities totaling 112 person-hours of training.

USAID/Jordan MESP – Annual Report (FY 2017) 17

Developing/Selecting Performance

Indicators Workshop

The training materials were developed with the intention of helping participant IP staff to become more comfortable with the process of selecting and developing performance indicators to measure progress towards achieving their results. The main objectives of the workshop were to help participants:

• Understand what performance indicators are and their role in tracking progress;

• Understand the different ways of expressing performance indicators;

• Recognize the criteria of good performance indicators; and

• Learn how to incorporate best practices in identifying performance indicators.

During the reporting period, MESP conducted one

“Selecting Performance Indicators” workshop that was attended by 31participants (5 M/26 F) representing 12 different activities for a total of 108.5 person-hours of training.

Preparing for Data Quality Assessment

(DQA) Workshop

The ‘DQA’ workshop aims to support IP staff in the preparation for DQAs conducted by the Mission. The training develops participants understanding of USAID data quality criteria and the process of a DQA.

The main objectives of the workshop are to help participants:

• Recognize the importance of high quality data for decision making and reporting;

• Understand USAID data quality standards;

• Be familiar with the processes and best practice of conducting a DQA.

MESP conducted four DQA workshops this year. A total of 43 participants attended the workshops, (12 M/31 F) representing 26 different Activities for a total of 150.5 person-hours of training.

Data Visualization Workshop

This workshop aims to build the capacity of USAID/Jordan Implementing Partners’ staff to understand the guiding principles for data visualization and to strengthen participant skills to produce quality visualizations using Excel.

The main objectives of the workshop are to help participants:

USAID/Jordan MESP – Annual Report (FY 2017) 18

• Understand basic data visualization principles

• Be able to produce customize charts in Excel

• Know what resources are available to further their data visualization skills

In preparation for this training, MESP utilized MSI home-office technical support to design and conduct an internal pilot workshop for MESP and subcontractor staff

(Integrated and Mindset), which not only informed future workshops, but also strengthened the capacity of MESP staff in data visualizations. After the pilot was completed, MESP conducted one Data Visualization Workshop, reaching a total of 12 participants attended the workshops, (1 M/11 F)…