Attachment_1.pdf

FY 2015 ANNUAL REPORT

ACTIVITY YEAR 2

OCTOBER 2014 – SEPTEMBER 2015

USAID/JORDAN Monitoring and Evaluation Support Project

OCTOBER 30, 2015

This publication was produced for review by the United States Agency for International

Development. It was prepared by Management Systems International.

FY 2015 ANNUAL REPORT

ACTIVITY YEAR 2

OCTOBER 2014 – SEPTEMBER

USAID/JORDAN MONITORING AND EVALUATION

SUPPORT PROJECT

Management Systems International

Corporate Offices

200 12th Street, South Arlington, VA 22202 USA

Tel: + 1 703 979 7100

Contracted under AID-278-C-13-00009

Jordan/USAID Monitoring and Evaluation Support Project

DISCLAIMER

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 i

CONTENTS

List of Acronyms .................................................................................................................................................................ii

Introduction

Key Accomplishments

Detailed MESP Results MESP Goal: Enhanced utilization of quality data to effectively assess progress towards the achievement of the CDCS (by Mission staff) IR1. Strengthened PMP Implementation and Utility

A. M&E Mission Training and Technical Assistance B. M&E Support for IPs C. Review or Establish Baselines and Targets, including Surveys and Studies D. Support Implementation of the Mission PMP E. Collect, Analyze and Report PMP data F. Data Quality Assessment29 G. Establish a Mission M&E MIS/Geographical Information System (GIS)

IR2. Project/Activity Evaluation Strengthened A. Evaluation Assistance B. M&E Mission Training and Technical Assistance C. M&E Support for IPs D. Assessments, Special Studies and Surveys E. Evaluation Utilization

IR3. Effective Communication of Selected Mission Results Implemented A. Strategic Communication and Strengthened Collaboration

MESP Activity Monitoring and Evaluation Plan Implementation

Lessons Learned

Management and Administrative Issues MSI Home Office Support to the Field

Constraints and Limitations

Key Planned Activities for the Coming Quarter

Annex I. List of Major Project Deliverables for the Year to Date

Annex II: MESP M&E Technical Support to Activities Fiscal Year to Date

Annex III: MESP M&E Technical Support to Mission Fiscal Year to Date

Annex IV: MESP Monitoring, Evaluation and Learning Plan

Annex V: MESP Training Plan

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 ii

LIST OF ACRONYMS

AMEP Activity Monitoring and Evaluation Plan

AOR Agreement Officer’s Representative

CBIWDM Community-Based Initiative for Water Demand Management

CDCS Country Development Cooperation Strategy

CEP Community Engagement Project

CISLE Cultivating Inclusive and Supportive Learning Environments in Jordan’s Schools

Activity

CoP Community of Practice

COR Contract Officer’s Representative

CSOs Civil Society Organizations

DEC USAID’s Development Experience Clearinghouse

DO Development Objective

DQA Data Quality Assessment

EDE Economic Development and Energy

EDU Education

EDY Education and Youth

ERFKE Education Reform for Knowledge Economy

ESCB Energy Sector Capacity Building Activity

FRP II Fiscal Reform Project II

GGI Gender Gap Index

GIS Geographical Information System

GOJ Government of Jordan

IP Implementing Partner

IR Intermediate Results

JCAP Jordan Communication, Advocacy and Policy Project

JCP Jordan Competitiveness Project

KaMP Knowledge Management Portal

LENS Local Enterprise Support Activity

LETS Learning Environment Technical Support Program

M-PMP Mission PMP

M&E Monitoring and Evaluation

MESP Monitoring and Evaluation Support Project

MIS Management Information Systems

MSI Management Systems International

MoE Ministry of Education

MPWH Ministry of Public Works and Housing

P-MEP Project M&E Plan

PAD Project Appraisal Documents

PAP Public Action for Water, Energy, and Environment Project

PIRS Performance Indicator Reference Sheets

PMP Performance Management Plan

PRO Program Office

QRTA Queen Rania Teachers Academy

RF Results Framework

SDO Special Development Objective

SOW Statement of Work

STTA Short Term Technical Assistance

TLM Teaching and Learning Materials

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 iii

TNA Training Needs Assessment

ToC Theory of Change

UCYI UNICEF Child, Adolescents and Youth Initiatives in Jordan

UNICEF United Nations Children’s Fund

USG U.S. Government

WRE Water and Natural Resources

Y4F Youth for the Future

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 4

INTRODUCTION

Management Systems International (MSI) began the United States Agency for International Development

(USAID) Jordan Monitoring and Evaluation Support Project (MESP) on Dec. 1, 2013, under contract

AID-278-C-13-00009. The principal purpose of the work is to provide technical and advisory services to the USAID/Jordan Program Office (PRO), USAID/Jordan Development Objective (DO) teams, and implementing partners (IPs) in the areas of project monitoring, evaluation, research, organizational learning, and knowledge management to enable USAID/Jordan to fulfill its performance and evaluation requirements.

As the second fiscal year of project operations began in October 2014, MESP utilized the opportunity to review its’ operational approach to identify best practices and lessons learned, and to adjust and plan for upcoming activities. A key practice that emerged in the first year of implementation has continued to characterize MESP’s approach to every task it has undertaken over the course of the year. This practice is the successful focus on continually strengthening and institutionalizing collaboration between MESP and PRO, which has resulted in strong alignment in the two groups’ approaches to monitoring and evaluating (M&E), and has encouraged greater collaboration between MESP, the Mission, and IPs.

Over the course of a highly productive year, MESP provided expert technical assistance in the various areas of performance monitoring and management, including conducting multiple reviews of Mission results frameworks and indicators. MESP provided detailed feedback, hands-on capacity building, and other support to the development and/or revision of just over a dozen Activity M&E Plans. In the majority of cases, MESP was able to document improvements integrated into these documents that resulted from this input. Concurrently, utilizing the information collected during a Training Needs

Assessment (TNA) survey of IP staff, MESP revised its training plan, materials, and content in order to better align with prioritized areas of learning as well as introduced learning approaches that better meet participant needs, the results of which proved to be very effective. In anticipation of USAID Data Quality

Assessments (DQAs), MESP established a new workshop to coincide with the scheduled assessments, which recorded high attendance rates and markedly positive feedback. Throughout FY 2015, MESP conducted 22 training sessions for Mission and IP staff, totaling 1240.5 training hours.

As MESP matured into a trusted partner of IPs and Mission staff, MESP leveraged its close relationships to implement new means of assessing IP needs and learning from feedback and past experience. One example was demonstrated by the rebranding and reorganization of ‘Drop in-sessions’ into the more successful Community of Practice (CoP), that has brought together different perspectives, approaches, and experiences by establishing a mutually beneficial and open communication platform for IP, Mission, and MESP staff to collaborate, and ultimately improve practices in M&E.

Aligned with the project’s approach to incorporating best practices from lessons learned, in the latter part of the year, MESP conducted an assessment to establish the effectiveness of MESP assistance to partners. The assessment sought to solicit rich qualitative data, through in-depth interviews with 15 IPs, that will be vital for identifying needed changes to MESP’s assistance and highlighting areas for future focus as the project enters its third operational year. Preliminary analysis of the data indicated universal agreement among IPs that MESP’s support in all of its forms had been useful for IP M&E staff in their work. This corroborates the high levels of satisfaction with MESP’s workshops already observed through training evaluation forms and collected anecdotally throughout the year.

A prominent achievement of the project’s second year is the highly successful development and launch of DevResults, a platform to systematize Activity-level indicator data for increased availability of

Management Information Systems (MIS)/Geographical Information Systems (GIS) data. The success of the launch, in part due to the tireless efforts of PRO and MESP staff as well as a very positive response from the 17 IPs selected for the initial rollout, resulted in 13 of the partners successfully reporting their

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 5 data in the first reporting window. By Q4, an impressive 88 percent of the M-PMP indicators were reported in the system. By the end of FY 2015, MESP began introducing a second cohort of Activities to the system, including providing introductory sessions and technical assistance as needed.

Concurrently throughout the year, MESP continued to provide significant assistance to activities by conducting six evaluations and assessments, including the final performance evaluation of the five-year project Strengthening Health Outcomes in the Private Sector (SHOPS), in addition to the School

Construction Assessment, an assessment which was planned and implemented entirely by local staff.

Furthermore, as part of the continued focus on improving evaluation and data utilization, MESP conducted two utilization reviews with the Mission: one focused on the Fiscal Reform Project II (FRPII) evaluation and another consolidating the Education and Youth (EDY) evaluations and Youth Assessment.

The reviews were conducted with Contract/Agreement Officer Representatives (CORs/AORs) and

PRO, and followed a utilization template developed by MESP and based on USAID and MSI best practices. The utilization reviews highlighted important recommendations for both improving the quality of evaluations as well as enhancing the efficiency of the utilization assessment process itself.

As the project’s third year approaches, MESP stands ready to launch an additional key tool for improving the availability of M&E data and other resources to Mission and IP stakeholders as it prepares for the imminent finalization of USAID’s security review for the MESP-developed Knowledge Management

Portal (KaMP). Among other resources, the KaMP will host a number of MESP-developed M&E training packages, templates, tools and guidelines as well as a rapidly expanding pool of targeted communications products developed by MESP, such as the nearly half-dozen two-page summaries of evaluations and assessments produced over the past year.

As MESP strives to embody best practices in monitoring and evaluation, the activity’s second year has included detailed reviews of various components of MESP’s operational approaches and continual exploration of how to enhance the assistance and support provided to the Mission and IPs. The achievements reflect a highly productive year that has successfully enhanced the collaboration of M&E stakeholders and encouraged the utilization of M&E-related products and services in order to further enhance performance management at USAID/Jordan.

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 6

KEY ACCOMPLISHMENTS

Availability and Use of Mission Performance Data

MESP/PRO-surveyed Mission staff rated the current availability of required performance data at

6.1 and estimated that one year ago, the availability would have been rated at 5.2 on a scale of one to 10 (not available to readily available).

Eighty-eight percent of 18 Mission staff respondents to this survey reported that their utilization of M&E data for assessing progress against the Country Development Cooperation Strategy

(CDCS) had improved in the past year.

Technical Assistance for USAID PMP

Conducted seven CoP sessions for implementing partners in Education and Youth (EDY), Economic Development and Energy (EDE), Water and Natural Resources (WRE), Democracy

Rights and Governance (DRG), and Population and Family Health (PFH); the first five sessions introduced the USAID Evaluation Policy and Practices and two sessions introduced the process of conducting DQAs.

Reviewed and provided feedback to the Mission on the DO Results Frameworks and performance indicator reference sheets (PIRS) for all sectors.

Training and Technical Assistance for the Mission

Provided 612 person-hours training for the Mission, as Mission staff participated in a four-day

Performance Management (PM) workshop as well as a workshop on conducting and preparing for DQAs.

Conducted a four-day PM Workshop in collaboration with PRO and the Middle East Bureau’s

Strategic Planning Office in USAID/Washington.

Revised the content of the DQA workshop and conducted one training session for Mission staff.

Conducted IP training workshops that were also attended by Mission staff.

Training and Technical Assistance for IPs

Provided 629 person-hours training for IP staff.

Developed training materials for DQA training workshop for IPs.

Conducted 19 training workshops for IP staff: seven Activity monitoring and evaluation plan

(AMEP) workshops, seven PIRS workshops, and five DQA workshops.

IP staff took part in a three-day PM Workshop alongside their Mission counterparts.

Conducted a TNA for IP M&E staff to improve the quality of M&E data provided to the Mission and enhance decision-making.

Technical Assistance for IP AMEPs

Reviewed, discussed, and provided feedback to the 14 IPs in support of developing and enhancing AMEPs.

Evaluations

Conducted the Public Action in Water, Energy and Environment Project (PAP) final performance evaluation.

Completed the Youth for the Future (Y4F) final performance evaluation.

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 7

Conducted the SHOPS final performance evaluation.

Submitted a concept note for impact and performance evaluations dealing with job creation, learning outcomes, and non-revenue water based on consultations with the Mission, IPs and subcontractor Mathematica.

Conducted utilization reviews of the FRP II, Y4F, and Learning Environment Technical Support project (LETS) evaluations.

Continued the analysis of data for the Civic Initiatives Support Project (CIS) evaluation. A draft report will be submitted in Q1 of FY 2016.

Assessments, Studies and Surveys

Submitted the Civil Society Sector Assessment Report.

Completed the School Construction Stakeholder Assessment.

Completed the Youth Assessment and conducted a stakeholder dissemination event with 100 attendees.

Completed the Primary Data Analysis for the Youth Assessment.

Conducted utilization reviews of the Youth and School Construction Stakeholders Assessments.

Establishment of Mission MIS/GIS

Set up and rolled out the DevResults system to the Mission and IPs.

Produced improvements in the availability of Mission performance monitoring plan (PMP) data as

17 IPs successfully reported their Q2/2015 indicator data in DevResults.

Conducted USAID/Jordan AOR/COR training as well as Train the Trainers for MESP and

USAID/Jordan Program Office staff. Reem Omar, USAID/JORDAN MIS and GIS Specialist, Maha

Bseiso, MESP Performance Management Support Manager, and Shadi Tanash, MESP Training

Specialist, have been granted the Train the Trainers certification.

Completed, with DevResults’ assistance, successful introductory sessions for IPs as well as two training sessions for USAID/Jordan AOR/CORs managing DevResults Activities and other technical staff.

Collected map data for USAID Activities related to host communities, school construction and grants, and transformed the data into interactive maps in ArcGIS that were then presented to

Mission Director Beth Page.

Provided on-going support to IPs during the second implementer data entry pass in DevResults for Q3/2015.

Introduced DevResults to new ongoing Activities (CISLE and MESC).

MESP Deliverables

MESP developed a proposed workplan for the launch and management of USAID/Jordan’s

Knowledge Management Portal (KaMP) pending the USAID/Washington’s completion of the portal’s security review.

MESP began to collect a list of proposed resources and collections for each sector to be hosted on KaMP prior to the official launch.

Restructured and finalized MESP’s Year Two Work Plan, which was approved.

Finalized and submitted MESP’s AMEP indicators for consideration by PRO.

Submitted the MESP Strategic Communication Plan, which was approved.

Communications

Produced the following communication products in English and Arabic:

https://dec.usaid.gov/dec/content/Detail.aspx?ctID=ODVhZjk4NWQtM2YyMi00YjRmLTkxNjktZTcxMjM2NDBmY2Uy&rID=MzU5MDg0

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 8

- Two-page summary on the USAID/Jordan National Youth Assessment;

- Two-page summary on the LETS Final Performance Evaluation;

- Two-page summary on the Y4F Final Performance Evaluation;

- The Youth Assessment Executive Summary; and

- The Youth Assessment Methodology and Sample Selection Factsheet.

Submitted the first edition of MESP’s ‘M&E Matters’ Newsletter for review.

MESP Staffing Enhancements

Hired an MIS Assistant Officer

Hired a Strategic Communications Specialist

Hired an M&E Specialist

Hired an Administrative Assistant

MESP/PRO Planning and Management

Conducted a joint work planning and team building meeting to reinforce the “One Team” management and implementation approach.

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 9

DETAILED MESP RESULTS

MESP Goal: Enhanced utilization of quality data to effectively assess progress towards the achievement of the CDCS (by Mission staff)

Percent of surveyed Mission technical management staff reporting agreement that their utilization of M&E data to effectively assess progress towards achievement of the CDCS has improved since the last survey

FY 2015 Actuals: 88.2%

Baseline: 88.2%

FY 2016 Targets: 80%

Mission M&E Baseline Survey

MESP and PRO conducted a Baseline Survey of Mission staff with the objective of assessing the utilization and availability of required performance data for management to achieve the Mission’s development objectives. The primary method of data collection was an online survey using 11 questions to elicit the feedback of Mission staff who oversee Activities as part of their job. In order to maximize

Mission ownership of and participation in the survey, PRO sent the online survey link via email to candidates.

The survey was used to provide baseline data on three performance indicators tracked by MESP, including one at the goal level of the MESP Activity: ‘Percent of surveyed Mission technical management staff reporting agreement that their utilization of M&E data to assess progress towards achievement of the CDCS has improved since the last survey.’

While it is difficult to conclusively attribute any improvement to MESP or PRO interventions, responses to the survey question concerning utilization are encouraging. When asked “To what extent do you agree with the following statement: ‘My utilization of M&E data to assess progress towards achievement of the CDCS goal has improved in the past 12 months,” 88 percent of respondents indicated they somewhat agree, agree, or strongly agree, that their performance in this area has improved over the previous year. The graph below shows the breakdown of all the responses to this question.

Figure 1: Response to: "My utilization of M&E data to assess progress towards achievement of the CDCS goal has improved in the past 12 months." n=18

70%

60%

50%

40%

30%

20%

10%

0%

Strongly Agree Agree Somewhat Agree Somewhat Disagree

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 10

There are important limitations in the ability of MESP to accurately and reliably assess actual and improved utilization of data by USAID managers in addition to the added challenge of attributing any improvements to specific MESP assistance. These include a potential difficulty in distinguishing MESP-provided technical support from that of PRO staff, who are also providing high-quality M&E assistance to

Mission staff, and the technical challenge of measuring and quantifying actual performance in this role when it is neither the full-time responsibility of most USAID managers nor readily observable. MESP will discuss the results of this survey and the utility of data collected for this indicator with PRO in the next quarter as part of the continued refinement of MESP’s M&E and Learning agenda.

Forty-five Mission staff were invited to complete the survey, which resulted in a response from 18

Mission staff (40 percent response rate). Responses reflected all six Technical Offices within the Mission:

WRE, PFH, EDE, EDY, DRG, and Gender. The table below summarizes the response rates.

Mission M&E Baseline Survey - Response Summary

Category

Number of

Staff

Contacted

Total Number and percent of responses received

Total Number and percent of Female responses received

Total Number and percent of Male responses received

USAID/Jordan

Mission Staff 45 18 (40%) 10 (55%) 8 (45%)

Moreover, out of the 18 respondents, nine (50 percent) were CORs and five (28 percent) were PRO staff. Due to the relatively small number of respondents, detailed comparisons between groups such as

PRO and A/CORs are not statistically valid and have not been provided. The table below summarizes the roles of respondents:

Mission M&E Baseline Survey – Respondent’s Role

Description of Role Number and percent of responses

A/CORs 9 (50%)

Program Office Staff 5 (28%)

Mission M&E Focal Point 1 (7%)

Other 5 (17%)

In order to assess the importance of M&E-related tasks in the Mission’s work, staff were asked, “Which of the following M&E tasks have you performed in the past 12 months?” The responses highlight that

M&E is a component of all respondents’ work, as each one of them listed performing at least two M&E tasks in the past 12 months. As illustrated in the table below, the top three M&E-related tasks listed by respondents related to portfolio reviews, site visits, and Project Appraisal Documents (PADs).

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

120.0%

Conduct Data

Quality

Assessment

Review

Activity M&E

Plan

Prepare or participate in

Portfolio

Review

Review of

Activity

Qaurterly

Report

Conduct

Activity Site

Visit

Develop

Project M&E plan

Develop or review PAD

All others

Figure 2: M&E Tasks Performed in the Past 12 Months

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 11

Further analyses of the survey results are presented below in appropriate sections of the report.

IR1. Strengthened PMP Implementation and Utility

IR 1.1 Average rating of the availability of performance monitoring data that the surveyed mission staff need to do their job (on a scale from 1 to 10, one being no available data and ten being data is readily available)

FY 2015 Actuals: 6.2

Baseline: 5.2 (baseline was measured retroactively in the same survey)

FY 2016 Targets: 7

As described above, MESP/PRO’s M&E survey of Mission staff was designed to efficiently provide performance and other data in a variety of MESP support areas. Working in close partnership with PRO, MESP technical assistance is intended to support Mission utilization of M&E data for portfolio management in multiple ways, including improving the availability of required performance monitoring data. In order to document changes in data availability potentially resulting from MESP interventions, MESP’s AMEP includes the following performance indicator: 1.1 Average rating of the availability of performance monitoring data that the surveyed mission staff need to do their job (on a scale from one to ten, one being no available data and ten being data is readily available).

In order to provide current data and a retrospective baseline, respondents were asked to rate the availability of performance data required to conduct a portfolio review or other M&E-related task data at the present time as well as 12 months ago. The rating scale ranged from one (not available) to ten

(readily available). While the improvements cannot be unequivocally attributed to any MESP or PRO-specific interventions, the lower average response to this question of 5.2 regarding data availability one year ago compared to the current rating of 6.2, demonstrates that a positive change is taking place in targeted areas.

Based on the reported levels of current data availability, there is substantial room for improvement that will be supported by DevResults and other PMP implementation support.

MESP conducted a number of data collection exercises in order to solicit feedback on the quality and utility of its various capacity building approaches. Some of these are frequent, such as the routine collection of training evaluations and pre- and post-training scores on knowledge tests. Others are less frequent, such as the once a year Mission M&E survey and the annual in-depth interviews with assisted

IPs. In all instances, MESP technical support was assessed positively, reflecting a high-level of satisfaction with the quality of MESP’s work to date. All of the MESP team are committed to using every possible tool at their disposal to assess the effectiveness of our work and to identify any and all opportunities to make positive change.

A. M&E Mission Training and Technical Assistance

Indicator 1.2.3 Number of person hours of training in Monitoring and/or Evaluation conducted by MESP

FY 2015 Actuals: 1241 (Mission staff received 612 person hours of training)

Baseline: 0

FY 2016 Targets: 612

Indicator 1.3 Percent of surveyed Mission/IP staff who report utilizing MESP developed tools to support implementation of performance monitoring tasks

FY 2015 Actuals: For Mission staff 88.9%

Baseline: 0

FY 2016 Targets: 75%

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 12

Building on the success of the first year, MESP has expanded its training and capacity-building portfolio for Mission and IP staff. A TNA for IP staff was conducted in Q1 and Q2, and a four-day (four days for

Mission staff and three days for IP staff) Performance Monitoring workshop in collaboration with PRO and USAID/ Washington’s Middle East Bureau, was planned during Q2 and delivered during Q3 with great success. In addition, a training plan was developed based on the findings of the TNA; under the new training plan, a DQA workshop for Mission staff was conducted in June and a new DQA training workshop for IP staff was developed and delivered during Q4. MESP’s portfolio of capacity-building

Activities and approaches continues to encompass a blend of traditional formal workshops, group meetings, technical on-the-job coaching, and written feedback and suggestions on draft and final Mission-produced documents. Figure three provides an overview of the MESP training sessions provided over the course of FY 2015.

Mission Staff Training/Workshops in Activity Year 2

One of MESP’s main objectives is to provide training and capacity-building support to USAID/Jordan

Mission and IP staff. During the current reporting period, MESP continued to provide training specific to

Mission staff, including delivering two training workshops aimed at achieving this objective. In addition, several Mission staff have attended training workshops targeted primarily at IPs. This resulted in 612 person-hours of Mission staff training during the year.

Mission Staff Training FY 2015

Title Number of

Workshops

Person Hours per Workshop

Attendance Total Person

Hours Total M F

Performance Monitoring workshop 1 32 18 3 15 576

DQA Mission 1 1.5 11 1 10 17

Mission Staff attending

MESP trainings for IPs N/A N/A 6 0 6 19

Total 2 N/A 35 4 31 612

Figure 3: MESP Training Sessions FY 2015

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 13

Performance Monitoring Workshop

A four-day Performance Monitoring Workshop (32 training hours per participant) was developed in collaboration with USAID/Jordan PRO and USAID’s Middle East (ME) Bureau in Washington. The workshop was attended by Mission and IP staff, with the first day being exclusive for Mission staff. The overall workshop objectives were determined based on the team’s experience in working with Mission and IP staff and on findings from the TNA. The goal of the workshop was to enable participants to work together to manage, measure and report on program performance, and ultimately, to enhance the quality and utility of performance monitoring data.

The workshop built on materials developed by the ME Bureau. MESP provided valuable input to tailor the content based upon the Mission and IP training needs it had identified through the TNA, workshops, and direct assistance to IPs. Christine MacAulay, Monitoring and Evaluation Specialist from the Bureau, travelled to Jordan one week ahead of the workshop to help finalize the materials.

Mission participants in the workshop included 18 (three male/15 female) Staff members, who were joined by 15 (three male/12 female) IP staff members and five (female) MESP staff members.

Data Quality Assessment for Mission Staff Session

In response to the priority capacity-building needs identified with the MESP COR, MESP developed and delivered a one and a half-hour training session for Mission staff focusing on how to conduct DQAs.

The training materials were designed specifically for Mission staff utilizing the feedback that MESP received from the Mission after delivering two similar workshops last year and the feedback received from participants in the Performance Monitoring Workshop.

Eleven Mission staff (one male/ten female) participated in the training, and three PRO staff helped in the organization and facilitation in addition to answering some participant questions. The majority of participants had not conducted a DQA prior to the workshop, which made the session of particular interest.

The overall rating of the session was very positive. Out of the six participants who completed the evaluation form, two rated the session as excellent, and four rated it as good on a four-point scale

(Poor, Fair, Good, and Excellent). Almost all responders either agreed or strongly agreed that the training session was relevant to their work, that it added to their knowledge on the subject, and that they plan to implement what they had learned.

Lessons Learned

The blend of both Mission and IP staff for learning opportunities is highly effective and welcomed by both groups. This learning environment creates unique opportunities to discuss the challenges faced in operationalizing performance monitoring systems and enables the participants to share practical experiences.

Preparing and distributing pre-workshop reading materials to participants ahead of the training enables participants to familiarize themselves with topics in advance, improving their participation.

During workshops designed for IPs, the participation of Mission staff, in particular ‘newcomers’ to the Mission, presents an excellent learning opportunity, as well as improves IPs’ commitment to applying and implementing what they have learned in their day-to-day tasks.

‘I will be responsible for following up with

IPs from the DRG team to assemble missing or problematic PIRS, and today’s training has provided excellent context for that work”

‘DQA Workshop ’Post-training evaluation

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 14

B. M&E Support for IPs

IR 1.2.1 Percent of interviewed IP technical assistance recipients that report that MESP support helped them to implement targeted M&E tasks

FY 2015 Actuals: 100%

Baseline: 0

FY 2016 Targets: 75%

IR 1.2.2 Percent of surveyed IP staff that report that MESP training workshops were helpful for them in doing their job

FY 2015 Actuals: 100%

Baseline: 0

FY 2016 Targets: 80%

IR1.2.3 Number of person hours of training in Monitoring and/or Evaluation conducted by MESP

FY 2015 Actuals: 1241 (IP staff received 629 person hours)

Baseline: 0

FY 2016 Targets: 612

IR 1.3 Percent of surveyed Mission/IP staff who report utilizing MESP developed tools to support implementation of performance monitoring tasks

FY 2015 Actuals: For IP staff 93.3%

Baseline: 0

FY 2016 Targets: 75%

IP Training Plan

MESP analyzed the data collected through the IP M&E TNA in order to identify gaps in knowledge, priority areas for training, and the desired delivery methods of training. The results were utilized to develop a proposed IP training plan for the next 12 months (Annex V). The first IP training workshop, developed in response to the IP TNA focused on providing further information on data quality assurance and assessment procedures as requested, was initially offered in Q4. Additional topics for new workshops will include developing effective indicators, developing Activity and project theories of change, and analyzing performance data.

Emphasizing the continued need for practical training in USAID M&E best practices and concepts, more than half of IP respondents to the TNA survey reported that they had less than two years of experience with USAID M&E practices. Under the circumstances, it is therefore not surprising that the majority of the survey respondents indicated a need for further information, knowledge, and skills.

Lessons Learned

In the process of reviewing the results of the survey, some valuable lessons were learned from the assessment that will be considered in developing future training plans:

Identify the desired competency that potential trainees should be able to achieve, such as produce a results framework, produce an AMEP, and write PIRS. This is important for triangulating what respondents say they need training in in order to do their job.

Suggest a limited number of training topics that are required for the trainees to achieve the identified activities and performance targets rather than offering an open menu of topics. Be specific and indicate how each topic will help trainees reach the desired competency.

Limit survey questions to only those that will inform the development of the training plan rather than things that are just nice to know.

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 15

Keep questions simple, particularly when the information required is simple and straightforward.

For example, the TNA asked about topics that required further skills to do the job and a different question about priority training topics. The answers were inconsistent, complicating analysis.

Utilize personal interviews in conducting the survey, which could provide more richness on opinions and needs and potentially avoid unexplained inconsistencies in responses between similar questions.

Workshops for IPs

To achieve its goal of enhancing the availability of performance data reported to USAID, MESP works with IPs to build the capacity of their staff to perform targeted M&E tasks and functions. During FY

2015, MESP delivered 20 training workshops, which were attended by 100 participants from 19 different Activities, for a total of 629 person-hours of training.

IP Staff Training FY 2015

Title Number of

Workshops

Person Hours per Workshop

Attendance Total Person

Hours Total M F

AMEP 7 3 30 10 20 90

PIRS 7 3 27 7 20 81

Performance Monitoring (PM) 1 24 15 3 12 360

DQA IP 5 3.5 28 10 18 98

Total 20 N/A 100 30 70 629

Introduction to AMEPs Workshop

Number of workshops delivered: seven (Q1=3, Q2=3, Q3=1, Q4=0)

Total person-hours of training: 90

MESP continued to offer the “Introduction to AMEPs” workshop during FY 2015. The purpose of this half-day (three-hour) workshop is to build the capacity of IPs to develop AMEPs that respond better to

USAID/Jordan’s needs and requirements and that conform to USAID best practices. MESP developed the workshop based on priority gaps identified while reviewing the IPs AMEPs and reports. The workshop focuses on improving the understanding of how IP Activities fold into the Mission PMP and

CDCS; details the required sections that provide USAID with sufficient data for Activity and portfolio monitoring and management; and explains USAID M&E terminology and acronyms.

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 16

During FY 2015, MESP offered seven AMEP workshops attended by 30 participants representing 12

Activities, which translates into 90 person-hours of training. However, as almost all of the currently active USAID/Jordan Activities have participated in the workshop, MESP reduced the frequency of offering the workshop to one session per month during Q1 and Q2, and once during all of Q3. In anticipation of a continued need for the workshop and to improve its effectiveness, particularly for new

USAID/Jordan partners as well as new staff, MESP will combine the contents of the workshop with the

PIRS workshop and offer a single combined training workshop that covers both AMEP and PIRS starting in Q1 of the next fiscal year.

Documenting PIRS workshop

Number of workshops delivered: seven (Q1=3, Q2=3, Q3=1, Q4=0) Total person-hours of training: 81

MESP also continued to offer the “Documenting PIRS” workshop. The purpose of this half-day (three-hour) workshop is to build the capacity of IPs to complete the PIRS accurately for their AMEP indicators. The workshop provides an overview of the purpose and specific content of the

USAID/Jordan IP PIRS template with specific guidance for completing the key sections, lessons learned, and best practice tips.

During FY 2015, MESP offered seven of these workshops with 27 participants representing 13 Activities attending the trainings, which translates into 81 person-hours of training. Similar to the AMEP workshop, almost all the currently active USAID/Jordan Activities have participated in the workshop, Sex Function Number of Activities

Figure 4: AMEP Workshops Attendance FY 2015

Female Male M&E Program Senior Management Others

Sex Function Number of Activities

Figure 5: PIRS Workshops Attendance FY 2015

Female Male M&E Program Senior Management Others

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 17 which also led MESP to reduce the frequency of the workshop to match that of the AMEP session. As noted above, in the coming quarter, MESP will offer a combined workshop for AMEPs and PIRS.

Performance Monitoring Workshop

Number of workshops delivered: one three-day workshop (Q3) Total person-hours of training: 360

IP staff attended three of the four days of the Performance Monitoring Workshop (24 training hours per participant). During the workshop, IP staff had the opportunity to interact with and learn alongside

Mission staff. Practical group exercises helped both groups of participants to benefit from each other’s experience. IP staff comprised 15 of the workshop participants (three male/12 female) and represented 15 Activities; five (female) MESP staff members joined them.

Data Quality Assessment Workshop for IPs

Number of workshops delivered: five (Q4=5)

Total person-hours of training: 98

DQA preparation for IPs was identified as a priority-training topic by MESP through the IP TNA, discussions with IP staff, and post-workshop evaluation forms. In addition, anticipating that the Mission will conduct several DQAs for the revised PMP indicators, PRO asked MESP to design and deliver a training workshop for IP staff to help them prepare for the expected DQAs.

During Q4 FY 2015, MESP developed a new training course for IP staff in time for the planned DQAs by the Mission. The workshop was designed specifically to meet the needs of IP staff, based on MESP experience in delivering similar sessions to the Mission, and needs identified through the IP

TNA.

Sex Function Number of Activities

Figure 6: Performance Monitoring Workshops Attendance

FY 2015

Female Male M&E Program Senior Management Others

Training materials provided to participants included USAID tips and guidelines, as well as the

DQA checklist

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 18

While the TNA determined the training topics, MESP sought the opportunity to incorporate more detailed IP suggestions and feedback to establish the content of each training topic. As part of this new approach, MESP reached out to several IP M&E staff and discussed their expectations as well as suggestions for the key content they hoped to learn about during the training workshop. Coupling the responses with the feedback gathered during two CoP sessions on

DQAs, MESP developed the training materials accordingly. The process highlighted specific subjects that were not previously identified as potential content, such as data quality criteria, and proved to be very effective, as reflected in the participants’ interaction during training, post-training evaluation, and the pre- and post-test scores.

The workshop employed several adult learning methodologies including PowerPoint presentations, individual and group exercises, examples from previously conducted DQAs in Jordan, USAID TIPS, review of the USAID/Jordan DQA checklist, and peer-to-peer learning.

In Q4, five workshops were delivered with 28 (18 female, 10 male) attendees representing 10 Activities.

This is equivalent to 98 person-hours of training.

Training Evaluation

MESP utilizes several approaches to evaluate the training workshops it delivers including through discussion and interaction during the workshop, requesting participants to complete a post-training evaluation form, pre-and post- training tests, and conducting personal interviews with participants several months after the workshop. In addition, MESP keeps track of participation records and disaggregates them by sex, Activity, sector, and workshop. However, as post-training evaluation forms and pre- and post- training quizzes are anonymous, it is not possible to disaggregate responders in any way.

Studying the information collected through the various types of training evaluations helps MESP make decisions on the frequency for conducting each workshop, topics to be offered, and it helps MESP adjust

Sex Function Number of Activities

Figure 7: DQA Preparation Workshop Attendance

FY 2015

Female Male M&E Program Senior Management Others

As part of the DQA Workshop, IP staff took part in several individual and group exercises

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 19 individual workshops in a way that helps to improve the learning gain. For example, the PIRS workshop had a small section on DQA for IPs. However, with the identified need to introduce a stand-alone DQA workshop, MESP developed a new workshop and stopped offering the DQA component in the PIRS workshop. In addition, when realizing the majority of Activities have attended both the AMEP and the

PIRS workshops, MESP reduced the frequency of the two workshops to once every quarter from once every month. MESP will also combine the two workshops in the next quarter.

Post-Training Evaluation

MESP’s post-training evaluation form contains five positive statements and, with the use of a five-point scale, seeks to assess to what extent participants agree with them. The statements gather participant feedback on the suitability of the workshop to their roles, the learning process during the workshop, the organization of the workshop, and participants’ intention to put what they have learned into practice.

The more participants who agree or strongly agree with the statements, the greater the overall satisfaction with the training and intention to implement what they have learned in their work.

Analyzing the post-training evaluation forms showed that participants were very satisfied with all the parameters assessed. Ninety-one percent of all participants (n=83) stated that they either agree or strongly agree with the statements provided. None of the participants strongly disagreed with any of the statements and the average response to the statements across all workshops was 4.57, as demonstrated in the chart below.

In addition, participants were asked for their overall rating of the workshop on a four-point scale (Poor, Fair, Good, and Excellent). Sixty-nine out of 83 participants thought the workshops they attended were

“Excellent,” while 14 participants thought they were “Good.” No participant rated any of the workshops as ‘Poor” or “Fair.” The overall rating of all the workshops was 3.83.

Analyzing the narrative part of the forms showed similar results in regards to participants’ satisfaction.

When asked, “How do you expect to apply what you have learned in this course,” most of the participants in the AMEP and PIRS workshops explained that it would help them develop and revise their

Activity AMEP and complete PIRS. Participants in the two courses stated that the new concepts, USAID terminology, and policies they learned would be helpful for them in their work. An interesting response came from one of Mission staff participants, who mentioned that attending the AMEP and PIRS

4.30 4.80 4.72 4.48 4.55 4.57

The course was relevant to what I do on the Job

The course was well organized

The course has added to my knowledge of the subject

I anticipate sharing my learning with my colleagues

I plan to implement the relevant sections in my job

Average of all workshops

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Figure 8: Combined Post-Training Evaluation Responses for All Workshops

FY 2015 (n=83)

Five-point scale questions 1=strongly disagree, 2= disagree, 3= somewhat agree, 4= agree, 5= strongly agree

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 20 workshop would help her in managing the Activities for which she is responsible. Participants in the

DQA workshops indicated that the training would help them to prepare for the expected USAID DQAs next quarter, ensure their performance data meet the data quality standards, and conduct internal DQAs.

When asked about the barriers or challenges they expected to face while implementing what they had learned, some of the respondents mentioned that the development of an AMEP, and the process of documenting PIRS requires collaboration of many staff members and that they have to rely on their cooperation. Others added that it requires a substantial investment of time to update an AMEP and to document PIRS, particularly for Activities about to close.

Pre- and Post-Training Test

In addition to completing the post-training evaluation forms, in FY2015 MESP introduced a pre- and post-training test designed to assess the change in participants’ knowledge and skills as a result of the training based on the workshops’ learning objectives. Administration of the tests as part of each training course has helped MESP to better understand the extent of the advancement or change in the participants’ knowledge after training.

To calculate the change between the pre- and post-training scores, MESP uses the following formula1:

𝑳𝒆𝒂𝒓𝒏𝒊𝒏𝒈 𝑮𝒂𝒊𝒏 = 𝑷𝒐𝒔𝒕 𝑻𝒆𝒔𝒕 𝑺𝒄𝒐𝒓𝒆 − 𝑷𝒓𝒆 𝑻𝒆𝒔𝒕 𝑺𝒄𝒐𝒓𝒆

𝑴𝒂𝒙𝒊𝒎𝒖𝒎 𝑺𝒄𝒐𝒓𝒆 − 𝑷𝒓𝒆 𝑻𝒆𝒔𝒕 𝑺𝒄𝒐𝒓𝒆

The formula compares pre- and post-test scores, dividing them by the maximum learning potential

(maximum score – pre-test score). MESP calculates the learning gain for each participant, and averages the gains for each workshop, course, and for all workshops together. Eighty-three participants completed the pre-and post-training tests during the reporting year. The average learning gain for all MESP workshops was 73.4 percent.

Analyzing the pre- and post-training test scores showed that post-test scores are substantially higher for all workshops. For example, the average post-test score for the DQA workshops was 94.1 percent while the average pre-test score was 58.1 percent, from which the calculated learning gain is 86 percent.

It is interesting to note that the average pre-test score for PIRS workshops was 74.6 percent, which is much higher than the average for the other two workshops (AMEP and DQA). One possible reason for

1 This a standard formula used to calculate the learning gained by participants out of the maximum that they could have gained. The formula is used by many training and educational institutions, including universities. Examples can be accessed through the following links:

http://www.trainingcheck.com/help-centre-2/guide-to-training-evaluation/creating-evaluations-at-the-different-levels/level-2-learning/.

https://www.slu.edu/Documents/professional_studies/EducatorWorkSample.pdf

0%

20%

40%

60%

80%

100%

Figure 9: Learning Gain for MESP Workshops FY 2015

AMEP (n=27) PIRS (n=29) DQA (n=27) Average for All Workshops (n=83) http://www.trainingcheck.com/help-centre-2/guide-to-training-evaluation/creating-evaluations-at-the-different-levels/level-2-learning/

USAID/Jordan MESP – Annual Report (FY 2015) October 2014 – September 2015 21 this is that MESP traditionally offered PIRS workshops following AMEP workshops, and, in many cases, participants in PIRS workshops had previously attended an AMEP workshop; some of the concepts and learning topics are common between the two workshops.

Lessons Learned

Reaching out to and soliciting more in-depth feedback in terms of participant needs and expectations during the training schedule and materials design phase results in training sessions that more effectively meet the needs of IP staff and ultimately help to improve training outcomes, participant satisfaction, and overall learning.

Training and training evaluation information collected through various tools helped MESP to learn and improve the design and delivery of its courses, as well as to make adjustments to the training plan schedule and content accordingly.

While previously an assumption, the training workshops through FY 2015 demonstrated that the relevance and the timing of a workshop has a direct effect on the participation of attendees and their learning. For example, attendees from Activities who were in the process of developing their AMEP were more invested in learning than other participants.