Attachment_2.pdf

ACTIVITY YEAR 1 ANNUAL REPORT

JANUARY – DECEMBER 2014

USAID/JORDAN MONITORING AND EVALUATION

SUPPORT PROJECT

February 23, 2015

This publication was produced for review by the United States Agency for International

Development. It was prepared by Management Systems International.

ACTIVITY YEAR 1 ANNUAL

REPORT

JANUARY – DECEMBER, 2014

USAID/JORDAN MONITORING AND EVALUATION

SUPPORT PROJECT

Management Systems International

Corporate Offices

200 12th Street, South

Arlington, VA 22202 USA

Tel: + 1 703 979 7100

Contracted under AID-278-C-13-00009

Jordan/USAID Monitoring and Evaluation Support Project

DISCLAIMER

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

USAID/Jordan MESP – January - December 2014 Annual Report i

CONTENTS

List of Acronyms ............................................................................................................................ ii

Introduction

Key Accomplishments

Detailed MESP Results for January – December 2014 IR1. Strengthened PMP Implementation and Utility

A. M&E Mission Training and TA B. M&E Support for IPs C. Review or Establish Baselines and Targets, including Surveys and Studies D. Support Implementation of the Mission PMP E. Collect, Analyze and Report PMP data F. Data Quality Assessment G. Establish a Mission M&E MIS/Geographical Information System (GIS)

IR2. Project/Activity Evaluation Strengthened A. Evaluation Assistance B. M&E Mission Training and TA C. M&E Support for IPs D. Assessments, Special Studies and Surveys

IR3. Enhanced Communication and Collaboration A. Strategic Communication and Strengthened Collaboration

MESP Activity Monitoring and Evaluation Plan Implementation

Management and Administrative Issues MSI Home Office Support to the Field

Constraints and Limitations

Key Planned Activities for the Coming Quarter

Annex I. List of Major Project Deliverables for the Year to Date

Annex II: MESP M&E Technical Support to Activities In Year 1

Annex III: MESP M&E Technical Support to Mission In Year 1

Annex IV: MESP Organizational Chart

USAID/Jordan MESP – January - December 2014 Annual Report ii

LIST OF ACRONYMS

AMEP Activity Monitoring and Evaluation Plan

AOR Agreement Officer’s Representative

CBIWDM Community-Based Initiative for Water Demand Management

CDCS Country Development Cooperation Strategy

CEP Community Engagement Project

CISLE Cultivating Inclusive and Supportive Learning Environments in Jordan’s Schools

Activity

COR Contract Officer’s Representative

DO Development Objective

DQA Data Quality Assessment

EDE Economic Development and Energy

EDU Education

ESCB Energy Sector Capacity Building Activity

FRP II Fiscal Reform Project II

GGI Gender Gap Index

GIS Geographical Information System

GOJ Government of Jordan

IP Implementing Partner

IR Intermediate Results

JCAP Jordan Communication, Advocacy and Policy Project

JCP Jordan Competitiveness Project

KaMP Knowledge Management Portal

LENS Local Enterprise Support Activity

LETS Learning Environment Technical Support Program

M-PMP Mission PMP

M&E Monitoring and Evaluation

MESP Monitoring and Evaluation Support Project

MIS Management Information Systems

MSI Management Systems International

MoE Ministry of Education

P-MEP Project M&E Plan

PAP Public Action for Water, Energy, and Environment Project

PMP Performance Management Plan

PRO Program Office

QRTA Queen Rania Teachers Academy

RF Results Framework

SDO Special Development Objective

SOW Statement of Work

STTA Short Term Technical Assistance

TNA Training Needs Assessment

UCYI UNICEF Child, Adolescents and Youth Initiatives in Jordan

UNICEF United Nations Children’s Fund

USAID U.S. Agency for International Development

USG U.S. Government

Y4F Youth for the Future

USAID/Jordan MESP – January - December 2014 Annual Report 3

INTRODUCTION

Management Systems International (MSI) began the United States Agency for International Development

(USAID) Jordan Monitoring and Evaluation Support Project (MESP) on Dec. 1, 2013, under contract

AID-278-C-13-00009. The principal purpose of the work is to provide technical and advisory services to the USAID/Jordan Program Office (PRO), USAID/Jordan Development Objective (DO) teams and implementing partners (IPs) in the areas of project monitoring, evaluation, research, organizational learning and knowledge management to enable USAID/Jordan to fulfill its performance and evaluation requirements.

December 2013 was devoted to administrative and start-up operations and technical implementation began in January 2014. During its first full year of operations MESP launched into full operational mode to complete four performance evaluations (Fiscal Reform Project (FRP) II, Learning Environment

Technical Support (LETS), Youth for the Future (Y4F) and Public Action for Water, Energy and

Environment Project (PAP)) and a sector assessment focused on youth, conducting 18 workshops attended by 130 IP staff and 30 USAID staff, providing direct one-on-one technical assistance to seven

IPs in revising their results frameworks and activity monitoring and evaluation plans (AMEPs), facilitating USAID/Jordan’s Mission-wide retreat to kick-start the revision of the Country Development

Cooperation Strategy (CDCS), and laying the foundation for the roll-out of DevResults, which will systematize activity-level indicator data for increased availability of MIS/GIS data. As MESP strives to embody best practices in monitoring and evaluation, the last quarter of the first year of MESP operations has included a detailed examination of MESP’s own indicators of success, and a continuing exploration of how to further the utilization of M&E-related products and services to enhance the USAID/Jordan

Mission’s Performance Management, the ultimate measure of success.

As a first step towards enhancing utilization of its assistance, MESP has contributed to increasing accessibility and availability of performance data through posting its performance evaluations of the

LETS and FRP II projects on the USAID Development Experience Clearinghouse web site https://dec.usaid.gov. MESP also gave a presentation of findings from the youth assessment attended by over 40 U.S. Government (USG) employees, with a larger stakeholder presentation planned in February

2015. Further awareness and availability of performance data results will be made available by MESP converting activity-level indicator data into standardized templates for Management Information Systems

(MIS) and Geographical Information System (GIS) analysis through DevResults. As USAID/Jordan completes the CDCS revision process based on input from the MESP-facilitated retreat, DevResults templates will be finalized accordingly and rolled out for Mission-wide use. There is increasing evidence that USAID/Jordan and IPs are using and benefitting from the AMEP, PIRS and DQA tools and templates created by MESP. In collaboration with USAID, MESP also designed and tested the Knowledge

Management Portal (KaMP) which is anticipated to be fully functional in Year Two. With increased accessibility and availability of needed M&E data to USAID/Jordan staff, it is anticipated that the coming year will present even greater and clearer evidence of utilization.

KEY ACCOMPLISHMENTS

Training and Technical Assistance to USAID

Provided support to USAID in defining and revising the results frameworks for Special

Development Objective 4 (SDO 4) that focuses on Gender Equality and Female Empowerment.

https://dec.usaid.gov/

USAID/Jordan MESP – January - December 2014 Annual Report 4

Led a consultative process with the USAID/Jordan Democracy, Rights and Governance staff and implementing partners to develop a revised results framework and indicators for the Democracy, Rights and Governance DO.

Provided detailed feedback and suggestions for improvements of the results frameworks for

Water and for Health.

Conducted systematic reviews of the Mission PMP and available AMEPs in order to prioritize technical assistance and capacity building topics for the first years of the activity.

Developed a draft USAID/Jordan Activity Quarterly Report Template.

Based on an M&E Training Needs Assessment (TNA) survey of USAID/Jordan staff, developed materials and conducted training with USAID staff on three M&E topics: working with IPs on improving AMEPs, conducting DQAs and Building Better Evaluation Statements of Work (SOW) with corresponding templates and guidelines.

Supported the Mission in evaluating available software packages for its MIS/GIS and in the selection and deployment of the DevResults system.

Facilitated the USAID/Jordan Mission-wide retreat that laid the foundation for revising the

Country Development Cooperation Strategy (CDCS).

Designed and built a web-based knowledge management portal for development-related resources in Jordan.

Training and Technical Assistance to IPs

Promoted opportunities for intra-IP learning and collaboration in M&E through ten sector-specific drop-in sessions.

Conducted 18 total sessions of two structured training workshops attended by representatives of

85% of USAID/Jordan activities on priority topics M&E topics: AMEPs and PIRS (Performance

Indicator Reference Sheets)/DQAs.

Provided activity-specific technical assistance to the Energy Sector Capacity Building (ESCB), Civic Engagement Project (CEP), Local Enterprise Support (LENS) Activity, Cultivating

Inclusive and Supportive Learning Environments (CISLE), Takamol (Gender project), Jordan

Communication, Advocacy, and Policy Project (J-CAP), and Adolescents and Youth Initiatives in

Jordan (UCYI) activities in support of revising AMEPs, PIRS, results frameworks, and baselines.

Provided direct technical assistance to guide activity staff in gender-related M&E for two activities: Community-Based Initiative for Water Demand Management (CBIWDM) and Jordan

Competitiveness Project (JCP).

Developed and launched an M&E Training Needs Assessment (TNA) survey for all

USAID/Jordan IP staff that will identify the IPs’ preferred content and characteristics of M&E training from a design and delivery perspective.

Evaluations

Completed the Learning Environment Technical Support (LETS) final performance evaluation, available on http://pdf.usaid.gov/pdf_docs/pa00k1qb.pdf.http://pdf.usaid.gov/pdf_docs/pa00k1qb.pdf

Completed the Fiscal Reform Project II (FRP II) performance evaluation, available on http://pdf.usaid.gov/pdf_docs/pa00k5pg.pdfhttp://pdf.usaid.gov/pdf_docs/pa00k5pg.pdf

Submitted the draft report of the Public Action in Water, Energy and Environment Project (PAP) final performance evaluation to USAID (December 2014) for comment.

Revised the draft report of the Youth for the Future (Y4F) final performance evaluation based on comments from USAID (December 2014). Awaiting further comments before final approval.

http://pdf.usaid.gov/pdf_docs/pa00k1qb.pdf http://pdf.usaid.gov/pdf_docs/pa00k1qb.pdf http://pdf.usaid.gov/pdf_docs/pa00k5pg.pdf

USAID/Jordan MESP – January - December 2014 Annual Report 5

Conducted a comprehensive and innovative Youth Assessment in six governorates, polling more than 700 youth, presented the findings to over 40 USG staff and submitted the draft report to

USAID (December 2014).

USAID/Jordan MESP – January - December 2014 Annual Report 6

DETAILED MESP RESULTS FOR JANUARY – DECEMBER

IR1. Strengthened PMP Implementation and Utility

A. M&E Mission Training and TA

Conducted a detailed TNA through a survey of Mission staff to help identify priority training topics and formats.

Supported USAID in refining and/or revising DO results frameworks in Gender, Democracy, Rights and Governance (DRG), Health and Water.

Completed analysis of the World Economic Forum’s Gender Gap Index to help USAID assess its appropriateness as a top-line indicator for Special Development Objective (SDO) 4.

Conducted workshops with USAID staff on Building Better Evaluation Statements of Work

(SOW), conducting DQAs and working with IPs on AMEPs with accompanying templates, tools and guidelines in each area.

Developed a USAID/Jordan Activity Quarterly Report Template.

Facilitated the Mission-wide CDCS revision kick-off retreat.

Beginning with a TNA conducted with Mission staff in the first and second quarters of MESP, the portfolio of capacity building activities and approaches expanded to encompass traditional formal workshops, group meetings, facilitated retreats, technical on-the-job coaching and written feedback and suggestions on draft and final Mission-produced documents. MESP’s Year Two work plan builds on this foundation of versatile and responsive support with replication of standardized trainings, a new co-facilitation approach to Mission training with USAID/Washington experts and expansion of capacity building into new requirements as new tools, including DevResults, provide additional opportunities for

MESP to perform its core functions.

Mission Staff Training/Workshops in Activity Year 1

A total of five formal classroom workshops were offered by MESP trainers to Mission staff over the year.

Feedback on the sessions was generally positive although the attendance figures were less than expected.

Specific assessment of the utility of the content and accompanying materials will be solicited in an upcoming survey of attendees and other staff.

Topic Number of workshops

Number of participants

USAID DQA requirements 2 16

Reviewing and Improving Partner Activity Monitoring and Evaluation Plans

2 11

Building Better Evaluation Statement of Work 1 6

Total 5 33

MESP has been asked to co-facilitate a multi-day workshop on performance management with a trainer from USAID/Washington’s Strategic Planning Office (SPO) in the Middle East Bureau in the next

USAID/Jordan MESP – January - December 2014 Annual Report 7 calendar year. A summary table of workshops and other technical support to the Mission is presented in

Annex III.

B. M&E Support for IPs

Over the first year of project activities MESP provided M&E-related technical support in various forms and intensities to a total of 11 USAID/Jordan activities, not including capacity building workshops and drop-in sessions where multiple IPs were invited. This support ranged from comprehensive reviews of

M&E plans that resulted in formal reports to the Mission with observations and recommendations, to reviews of PIRS and results frameworks to help partners identify and address potential gaps prior to formal submission to the AOR/COR. A full list of activities assisted by MESP technical experts is contained in Annex II.

While a complete assessment of the utility and effectiveness of this support is not possible in many instances due to a lack of final approved AMEPs and/or multiple instances where feedback was provided to Mission staff without a request for in-person follow-up by a member of the project team, there are several instances where improvements were noted in revised documents that corresponded to areas of technical support provided. Examples include the addition of mid-level outcome indicators in the most recent draft of the ESCB AMEP; the production of a substantially more complete AMEP using the

Mission’s template by CISLE (replacing a spreadsheet of indicators-based M&E plan); and significant improvements to the results framework of the UNICEF-implemented UCYI initiative.

MESP and PRO’s deliberate and incremental approach to building trust in and demand for technical support among both Mission and IP staff paid off throughout the first year as demand steadily increased for project services, particularly at the crucial early stages of new activities producing their first

AMEPs. This momentum is expected to continue to build in Year Two as the new Mission CDCS and

PMP are finalized and additional areas for adding technical value are identified and implemented.

Workshops for IPs

In order to build capacity among IPs to improve performance data reported to USAID, MESP offered two training courses to IPs during the first year. Topics for these workshops were selected based on MESP’s review of existing AMEPs and available data in the Mission PMP during the activity’s first quarter.

Priorities identified during this review focused on the importance of clearly linking AMEPs to required

Mission indicators, the need for more robust AMEP narrative and structure beyond simple tables of performance indicators and the goal of encouraging IPs to prepare PIRS in order to improve reporting across activities on shared indicators and facilitate USAID/Jordan DQA implementation. The total attendance at these two workshops for the year was 130, representing 83 unique individuals, the majority of whom were female. Coverage of USAID/Jordan IPs by these workshops was approximately 85%.

Attendance data are summarized below.

USAID/Jordan MESP – January - December 2014 Annual Report 8

AMEP PIRS/DQA Total

Total IP workshops in 2014 9 9 18

Attendance Total 73 57 130

Attendance Female 54 42 96

Attendance Male 19 15 34

Attendance M&E Staff 31 28 59

Attendance Senior Management 13 7 20

Attendance Program 16 11 27

Attendance Other 13 11 27

Introduction to AMEPs Workshop

The purpose of this workshop is to build the capacity of IPs on how to develop AMEPs that better respond to USAID/Jordan’s needs and requirements, and conform to USAID best practices. MESP developed the workshop based on priority gaps identified while reviewing the IPs’ AMEPs and reports.

The workshop focuses on improving IPs’ understanding of how their activities fold into the Mission PMP and CDCS; describes the sections that are required to provide USAID with sufficient data for activity and portfolio monitoring and management; and explains USAID M&E terminology and acronyms. Sixteen new IP staff (ten females, six males) attended the three workshops that were conducted during the final quarter, bringing the total participants for the year 2014 to seventy-four. As Figure 1 below illustrates, attendance at these workshops exhibited wide interest in M&E related topics by a variety of professionals as more than half of the attendees were not specifically M&E staff in their activities.

Documenting PIRS and Preparation for DQA

The purpose of this workshop, which was launched in the second quarter of MESP’s first year, is to build

Sex Function

AMEP Workshop Attendance for Activity Year 1

Other

Program

Senior Management

M&E

Attended - Male

Attended - Female

USAID/Jordan MESP – January - December 2014 Annual Report 9 the capacity of USAID/Jordan IPs to accurately complete the PIRS for their AMEP indicators, and to help them prepare for USAID DQAs. The PIRS/DQA workshop material was developed based upon the needs uncovered during the AMEP workshops and further builds upon information presented at these workshops. The workshop provides an overview of the purpose and specific content of the USAID/Jordan

IP PIRS template with specific guidance for completing key sections of the template.

Three PIRS/DQA workshops were conducted during the final quarter with a total of ten new IP staff (7

Females, 3 Males) attending, bringing the total participants for the year 2014 to fifty-seven. Figure 2 below shows the characteristics of the PIRS/DQA workshop participants for the first year of the project.

As with the attendees in the AMEP workshops, a substantial portion of attendees were not dedicated

M&E staff within their activities.

Training Evaluation

Analysis of the training workshop evaluation forms shows that participants in MESP workshops have reported very high levels of satisfaction with the content and format of the sessions. When asked to rate the overall satisfaction with the course, nine out of 13 participants (69%) who completed the evaluation form for the AMEP workshops stated it was “Excellent,” while four out of 13 participants (31%) reported it was “Good.” As for the PIRS/DQA workshops, all participants (ten out of ten) reported it to be

“Excellent.” More importantly, when assessing the specific relevance of the workshop content to their work, on a scale from one to five (one being the lowest), scores averaged 4.0 and 4.5 respectively for

AMEP and PIRS/DQA workshops. Similar scores were reported for participants’ plans to share the knowledge they learned with their colleagues with an average of 4.5 for both course types.

The graphs below show more details of participants’ reaction to the workshops they attended.

Sex Function

PIRS/DQA Workshop Atendance for Activity Year 1

Other

Program

Senior Management

M&E

Attended - Male

Attended - Female

USAID/Jordan MESP – January - December 2014 Annual Report 10

In the evaluation form, participants are given a list of eight M&E topics and are asked to select up to three of them as their top priority to receive training and additional support. The analysis of the completed forms show that the top two topics selected are:

1- Development Hypothesis and Theory of Change (selected by 20% of the responders)

2- M&E fundamentals (selected by 18% of the responders)

As MESP has recently launched the IP M&E TNA, the above figures will be considered when determining what new courses will be offered to IPs. MESP has also noticed an interest in the topic of

“Development Hypothesis and Theory of Change” from USAID/Jordan Mission staff.

Pre- and Post-training tests

In the fourth quarter of the project year, MESP began to utilize pre- and post-training tests in order to assess how much learning is happening and identify areas where additional practice or information is needed.

The pre-test and post-test were designed using the learning objectives established for each course. Both tests consist of the same set of questions. However, the pre-test is given to participants before training as a

1.0 2.0 3.0 4.0 5.0

The course was relevant to what I do on the Job

The course was well organized

The course has added to my knowledge of the subject

I anticipate sharing my learning with my colleagues

I plan to implement the relevant sections in my job

1= Strongly Disagree; 5=Strongly Agree

Participants' post-training feedback: AMEP Course Oct-Dec 2014

1.0 2.0 3.0 4.0 5.0

The course was relevant to what I do on the Job

The course was well organized

The course has added to my knowledge of the subject

I anticipate sharing my learning with my colleagues

I plan to implement the relevant sections in my job

1= Strongly Disagree; 5=Strongly Agree

Participants' post-training feedback: PIRS Course Oct-Dec 2014

USAID/Jordan MESP – January - December 2014 Annual Report 11 means of measuring how much they know about the topic. At the conclusion of the training, the participants take the post-test.

As part of the analysis process for the pre- and post-training tests, MESP calculates the “learning gain” from each workshop. This shows the improvement between the tests after attending the training. The formula used to calculate the learning gain is:

Learning Gain (%) = Post training Score - Pre training Score

Maximum Score - Pre training Score x 100%

Analysis of the pre- and post-training tests scores show that there was an average of 77% learning gain in assessed AMEP workshops and 58% learning gain in assessed PIRS/DQA workshops. The table below shows the details of pre- and post-training tests scores for all AMEP and PIRS workshops.

Workshop AMEP PIRS/DQA

Average Pre-training score (out of 100) 52 73

Average Post-training score (out of 100) 89 88

Average Learning Gain (%) 77% 58%

It is interesting to note that the pre-training score for the PIRS/DQA workshops is much higher than for the AMEP workshops (73 to 52 out of a maximum 100). This might be in part, because many of the participants to the PIRS/DQA workshops had already been to the AMEP workshop, during which they gained new knowledge that helped them score higher on the PIRS/DQA pre-training test. This shows that those participants had retained the knowledge they learned and were able to apply it in a different situation. MESP has always recommended participants to attend the AMEP workshop before coming to the PIRS/DQA one and has usually offered the AMEP workshop earlier during the month than the

PIRS/DQA.

Training Coverage

By the close of the year, analysis of participation records show that more than 80% of USAID’s activities

(approximate figure as the number of activities changes over time) have had staff attend both the AMEP and PIRS/DQA courses as of December 31, 2014 (see Annex II). Nevertheless, demand for both workshops remains stable as new staff are joining or changing roles at the activities. In addition, as new activities are launched, their new staff will seek the support of MESP in developing their AMEP or completing their PIRS. For this reason, MESP will continue to offer both courses at the rate of one workshop per quarter each.

In the next quarter, MESP will implement a survey with USAID and IP attendees to comprehensively assess their application and use of targeted knowledge, skills and tools in their M&E work since their participation in the courses, and as a result, of other MESP technical support.

IP Training Needs Assessment

As attendance at workshops began to approach saturation with existing IPs, MESP launched a TNA for

IPs to determine topics and format for future training. The TNA is intended to 1) identify gaps in M&E knowledge and skills among USAID/Jordan IP staff; 2) determine the preferred characteristics of M&E-related training courses; 3) understand IPs’ priorities for M&E training topics; and 4) uncover challenges

USAID/Jordan MESP – January - December 2014 Annual Report 12 and barriers experienced by IP staff in conducting M&E work that may be outside their realm of capacity.

The TNA was developed with the support of MSI Home Office and was launched during the last week of

December 2014.

The TNA will conduct data collection through an online survey with IP staff (Senior Management, M&E, and Technical/Programmatic staff), and for a second survey with USAID/Jordan Mission staff. MESP sent invitations to 25 Mission staff and requested the nomination of at least three staff from each activity.

In addition to the online surveys, MESP will conduct thematic focus group interviews with selected IPs’ staff to identify the training needs among IPs. MESP will use the results of the TNA to develop new training courses, create new tools and guidelines, and work with IPs and the Mission to improve M&E performance and data use.

Drop-in Sessions

In order to provide practice-based application of learning in workshops, MESP offered ten sector-specific drop-in sessions that facilitated inter-activity dialogue and experience sharing among sector M&E staff and supporting them in applying the best practices in M&E. Based on feedback from participants, the frequency and structure of future drop-in sessions will be modified. These meetings will take place quarterly in Year Two and each session will be equally divided between a MESP-developed presentation on a pre-selected M&E topic and free discussion between session attendees.

Activity M&E Plan Reviews and Related Technical Assistance

As a result of the workshops and in-depth technical assistance and/or requests from USAID, MESP provided technical assistance related to AMEPs in a variety of formats. In the first month of on-the-ground implementation, MESP provided a summary overview of the AMEP for the FRP II activity to their COR. In addition, MESP has worked closely with the following four activities to revise existing AMEPs to address recommendations for strengthening of M&E systems and performance data identified by

Mission and MESP reviews: the Community Engagement Project

(CEP), Energy Sector Capacity Building (ESCB), Cultivating

Inclusive and Supportive Learning Environments in Jordan’s

Schools (CISLE) and Local Enterprise Support (LENS) Activity. In addition, MESP provided substantial technical support for IP development of AMEPs for new activities: Takamol, Jordan

Communication, Advocacy and Policy project (J-CAP) and

UNICEF Child, Adolescents and Youth Initiatives in Jordan

(UCYI). During the quarter, MESP M&E Specialists met with M&E and program staff from each of these activities in order to discuss recommendations and best practices related to AMEP content. While MESP is awaiting receipt of the final versions of several of these AMEPs in order to completely assess actions taken in response to MESP recommendations, ESCB, CEP, Takamol and UCYI staff expressed agreement with guidance provided by MESP in subsequent meetings. More importantly, draft revisions of

AMEPs produced after the reviews by these activities demonstrated significant improvements corresponding to the guidance provided by MESP to date.

In order to improve its understanding of the M&E systems of one of USAID/Jordan’s largest DRG activities, MESP requested Mission approval for and conducted several meetings with staff of the Civic

Initiatives Support (CIS) activity. As MESP was not tasked by the Mission for this work, a formal report was not prepared.

Technical Assistance Provided to 7

IPs (AMEPs, PIRs, RFs, Baselines)

Community Engagement Project

(CEP)

Energy Sector Capacity Building

(ESCB)

Cultivating Inclusive and

Supportive Learning Environments in Jordan’s Schools (CISLE)

Local Enterprise Support (LENS)

Activity

Takamol

Jordan Communication, Advocacy and Policy project (J-CAP)

UNICEF Child, Adolescents and

Youth Initiatives in Jordan (UCYI)

USAID/Jordan MESP – January - December 2014 Annual Report 13

Jordan Communication, Advocacy and Policy Project (J-CAP)

The Jordan Communication, Advocacy and Policy Project is a new five-year health activity implemented by Abt Associates. In November 2014 MESP was asked to provide feedback on J-CAP’s M&E plan and baseline KAP Survey questionnaire. MESP provided direct initial feedback through a meeting with the J-

CAP COR and M&E staff. The main recommendations provided to J-CAP in that meeting included the need to strengthen the logical organization of J-CAP’s results framework and the importance of explaining the causal path represented by that framework in a detailed written development hypothesis.

MESP’s technical support through the end of the year was focused on assisting J-CAP to revise and improve its results framework and indicators. MESP will continue providing appropriate technical assistance to help this activity finalize its AMEP and KAP Survey Questionnaire.

Local Enterprise Support (LENS) Activity

In the second quarter of MESP activities, a meeting was arranged with FHI360, the Local Enterprise

Support (LENS) activity implementer and the LENS COR to present and discuss best practices around the development of AMEPs, including links between the AMEP and Mission PMP (M-PMP)/Log

Frame/CDCS in the form of required Mission indicators. Recommendations from the workshop included that the LENS AOR should provide the LENS M&E team with a list of required Mission indicators from the M-PMP and PPR so that these can be inserted into the LENS M&E plan as it is developed. Six months later, MESP was asked to review the proposed draft LENS AMEP by their COR. In a meeting attended by the LENS COR, the EDE M&E PoC, M&E staff from PRO, and MESP’s M&E Specialist, strengths and weaknesses of the draft AMEP were discussed and a unified series of recommendations were agreed upon. Once MESP receives a revised AMEP from USAID, it will complete a detailed review to assess how its assistance was utilized in developing the document and whether additional assistance is required.

Community Engagement Project (CEP)

In March, at the request of the Community Engagement Project’s (CEP’s) AOR, MESP met with Global

Communities, the CEP implementing partner, in order to complete an analysis of strengths and weaknesses of CEP’s AMEP and baseline study. This consultation was followed by a report on findings and recommendations submitted to MESP’s COR. Recommendations from the analysis included:

developing additional outcome-level indicators to bridge an observed gap between existing output indicators and population-level impact measures, including required Mission indicators for CEP in the

AMEP, and completing targets and PIRS for all CEP indicators, among other suggestions for strengthening CEP and USAID/Jordan M&E data quality. Subsequent drafts of CEP’s revised AMEP that were shared unofficially with the MESP team included revisions that addressed many of the observations and recommendations made in MESP’s original assessment. Once MESP receives the final approved or officially submitted revised AMEP from USAID, it will complete a detailed review to assess how its technical support was utilized in the new document.

Takamol (The Gender Project)

In August 2014 MESP began work with Takamol - USAID/Jordan’s flagship Gender project - in order to assist them in developing their AMEP and to facilitate cross-cutting work by both Takamol and MESP in supporting results reporting under the Mission’s Special DO4. Over the course of several meetings, MESP technical assistance focused on helping M&E staff to better understand and articulate Takamol’s theory of change and in strengthening the activity’s results framework and performance indicators. An assessment of the effectiveness of this support will be conducted once Takamol’s final approved AMEP is shared with MESP.

UNICEF Child, Adolescents and Youth Initiatives in Jordan (UCYI)

USAID/Jordan MESP – January - December 2014 Annual Report 14

After UNICEF attended AMEP training in November 2014, USAID/Jordan’s EDY team encouraged

UNICEF to invite MESP’s M&E Specialist to provide additional support to the development of their

Child, Adolescents and Youth Initiatives in Jordan (UCYI) AMEP. MESP provided a number of comments and suggestions on how to enhance the AMEP, with specific focus on improving the logic of the proposed RF and the utility of the selected performance indicators. Since it is funding existing UCYI activities for which a reporting structure already exists, it was determined that substantial changes to conform to USAID’s best practices were not necessary. However, UNICEF is taking MESP’s recommendations into consideration as they revise the AMEP.

Cultivating Inclusive and Supportive Learning Environments in Jordan’s Schools (CISLE)

Based on a request by USAID to support the ability of the newly initiated CISLE Jordan education activity to respond to an upcoming USAID DQA, a meeting was organized at Queen Rania Teachers

Academy (QRTA), the implementing partner of the activity. The MESP M&E team developed and delivered a two-hour presentation on best practices in USAID IP AMEP development and preparing for a

USAID DQA that was specifically tailored to the objectives and required indicators of CISLE. In addition to this workshop, several telephone and email consultations between QRTA and MESP took place to answer additional AMEP-related questions. MESP received the final AMEP for CISLE in mid-December.

Gender-Specific Technical Assistance

During the past year MESP provided consultations to several IPs relative to gender issues in performance management. The CBIWDM activity was provided with information on best practices for reporting on gender indicators. MESP has also provided consultation with Takamol, the Jordan Competitiveness

Project (JCP), and J-CAP to provide guidance on using appropriate indicators for gender mainstreaming and gender-specific indicators.

C. Review or Establish Baselines and Targets, including Surveys and Studies

The first year of MESP implementation resulted in limited focus on specific support related to baselines and targets. In Year Two and beyond, the team anticipates that much more work in these areas will be required for both new indicators identified in the revised CDCS and Mission PMP, as well as for the design and implementation of impact evaluations in multiple areas of the portfolio.

D. Support Implementation of the Mission PMP

Mission CDCS Revision Support

In October 2014, MESP fielded two highly experienced consultants – Drew Lent and Michelle Adams-

Matson – to assist the Mission in designing and conducting an all-staff retreat intended to launch the revision process for the CDCS. Over the last two years, Jordan experienced a rapidly evolving development environment, including the crisis in Syria. At this mid-point in strategy implementation, the

Mission chose to conduct a retreat to provide an opportunity to examine what, if any, revisions are necessary in order to respond effectively to the changing context. This retreat provided an opportunity for staff to step away from a fast-paced and demanding daily environment to consider larger strategic and management issues that are critical for success over the long term.

The explicit retreat objectives were to: Ensure that all staff understand the strategy and the evolving development context (integrating new staff); Ensure that staff understand how their work better supports the strategy; Adapt or strengthen program synergies; Determine if the strategy is relevant or whether changes are necessary and identify how to be more effective; and Team Building.

The retreat organizers met with DO teams prior to the retreat to get the needed background on questions

USAID/Jordan MESP – January - December 2014 Annual Report 15 of strategy as well as of performance challenges and successes. In the retreat itself, staff attendees were organized by DO when considering DO issues and in cross-DO teams when considering performance issues. The key retreat accomplishment was the identification of a core team of Mission staff with a detailed timeline to lead the revision process of the CDCS. An extensive and complete report was submitted with all retreat materials and transcriptions of all retreat products.

Mission DO Team Results Framework Revision Support

MESP’s input into the Mission’s RF revision process covered a full-range, from providing written feedback and suggestions for improvement via email to fielding multi-week STTA where completely revised frameworks and indicators were developed and shared with the relevant team. While formal feedback on the value of this assistance was not provided in most instances, comments shared by the

Water team highlighted the desirability of a more focused approach in some instances where detailed suggestions on each and every component of the RF may have been overwhelming. An additional lesson learned through this and other similar exercises was the essential requirement that any specific review of results language and/or indicators be informed by a detailed presentation of the intended results and key interventions under the corresponding component of the framework. The health team shared a format for presenting such results and activities with a revised results framework that MESP recommended for use by other DOs working on similar tasks.

Health

In response to a request from PRO, MESP prepared detailed feedback to a draft revised results framework in health in Q3 of the fiscal year. Key modifications suggested by MESP included separating the IRs into more specific sub-IRs to highlight the importance of maternal and neonatal health interventions as well as proposed modifications and additions to higher-level indicators to reflect the specific outcomes of

USAID’s work in new technical areas that would not be reflected in the current family planning-focused metrics.

Water

MESP submitted multiple comprehensive comments and suggested potential revisions to the draft RFs that were shared with the office for input. This process was somewhat hampered by difficulty in identifying the specific intended results and activities designed to produce those results for each of the IRs and sub-IRs of the Water DO’s RF. While MESP held several meetings with the Water team to try and improve its understanding of the required information, sensitivities around upcoming procurements and other barriers to full transparency may have impeded the effectiveness of the discussions.

Democracy, Rights and Governance (DRG)

In mid-June, STTA was fielded for two weeks to assist the DRG team in revising the DO RF and corresponding performance indicators. Working with IPs and Mission staff in multiple small and large group meetings, the consultant produced a draft revised results framework with detailed narrative of intended results under each IR and sub-IR and corresponding indicators. The final list of indicators delivered in July included innovative use of Jordanian expert panels for assessing progress in key components of the portfolio’s intended results. Feedback on the final deliverables has not been provided by the Mission to date.

Gender Equality and Female Empowerment (SDO 4)

MESP worked with USAID’s SDO 4 team to help the Mission finalize the performance indicators included in the Mission PMP and results framework relative to gender. One outcome of these discussions was a request for MESP to conduct a detailed assessment of the sensitivity and utility of the World

USAID/Jordan MESP – January - December 2014 Annual Report 16

Economic Forum’s Gender Gap Index (GGI), which is currently a top-level indicator in the SDO 4 framework. MESP’s analysis highlighted that many of the indicators and sub-indicators utilized in the calculation of the GGI do not directly correspond to specific results targeted by USAID/Jordan interventions across the portfolio. While there is some direct correspondence between GGI indicators and current USAID/Jordan activities in the area of economic development, intersection in most other areas of the GGI is indirect. Given USAID’s current programming, it is unlikely that the GGI would register significant change in the overall GGI resulting from USAID’s portfolio, which limits its utility for performance monitoring. One approach to improving the utility of the GGI for performance monitoring by the Mission would be to pull out and focus on the most directly impacted sub-indicators used in the overall index rather than focusing on just the top-level numbers. With this input, the Mission has decided to maintain the GGI as a context indicator and rely on other indicators in the PMP to measure performance.

Economic Development and Energy Sector

During the USAID Economic Development and Energy (EDE) retreat that was held at the Dead Sea in

September 2014, MESP facilitated a session where it presented best practices in the identification and development of indicators at the IR level of the USAID Mission PMP beyond those listed in the F

Standard Indicators Matrix.

E. Collect, Analyze and Report PMP data

Mission Portfolio Review Support

In April of Q2, MESP prepared a series of slides illustrating a more visual way to present Mission PMP data to improve rapid comprehension of progress against targets and the current status of PMP indicator achievement within each DO team. USAID approved a subset of these slides for use in the Mission’s June

DO-level Portfolio Review (PR). At the request of the MESP COR, the home office and field office teams provided feedback to the proposed overall presentation template for use by each DO team and revised it to address agreed-upon improvements and to integrate the new slides for indicator data. In May, MESP was tasked with populating the DO PMP indicator slides with data provided by the Water and DRG teams. Many of the Water and DRG PR indicator slides were removed due to a lack of data for many of the indicators in the current approved PMP for these teams.

F. Data Quality Assessment

Project support of Mission-conducted DQAs took a variety of forms tailored to the needs and requests of

PRO and other USAID staff. Over the course of the year, MESP conducted DQA-related workshops for both Mission and IP staff targeted at improving the preparation of both parties for conducting DQAs as well as the quality of completed assessments. In addition, a revised DQA checklist was prepared for use along with a detailed set of guidelines focused on planning for and implementing all stages of the DQA process. MESP staff attended multiple DQA sessions in order to provide in-person support to Mission staff while they used the new tools. Detailed written comments on a number of completed checklists sent by PRO for feedback were also provided. Support in Year Two is expected to be substantial as the

Mission works to implement DQAs for a number of new indicators required by the revised CDCS.

G. Establish a Mission M&E MIS/Geographical Information System (GIS)

One of MESP’s earliest and most effective uses of home office STTA was the mobilization of MSI’s

Head of Knowledge Management & Information Systems in March to assist the Mission in assessing the suitability and potential costs of available MIS/GIS options for deployment in Jordan. In mid-April 2014, the MESP COR notified the activity that USAID has exercised Option 1 of the contract and thus added

USAID/Jordan MESP – January - December 2014 Annual Report 17 responsibility for supporting the development and deploying the USAID/Jordan MIS/GIS using

DevResults to the MESP portfolio.

In August 2014, the Performance Management Support Manager joined MESP as a long-term staff member to implement the web-based system implemented on the DevResults platform to track tasks and workflows and to collect, analyze, and report on USAID and IP M&E data. The Manager will also work with DO teams to develop effective processes for using monitoring data to improve program performance and will collaborate with the Monitoring and Evaluation Data/GIS Analyst, who has also been hired in

January 2015, to assist in creation of relevant, effective visualizations of monitoring data.

MESP initiated work to populate the DevResults, the Management Information Systems (MIS) /

Geographical Information System (GIS), with Mission and IP frameworks and indicators. MESP completed converting available information for 17 ongoing USAID activities in the DevResults Setup

Template format prior to the Core team’s “DevResults Power User/Admin” training, which took place during November 2014. Later, MESP decided in consultation with USAID, that only 14 ongoing priority activities (large, new, or having at least 18 months of activity time to go) would participate in the implementation of the DevResults program. Accordingly, these activities would start populating their FY

2015 second quarter results data into DevResults, and thus, the first reporting period would be during

April 2015. Additionally, MESP will start exploring the DevResults mapping function with the creation of maps in three areas: school construction, grants, and Syrian population concentrations in Jordan. The latter map and related data will be used as an overlay/layer to provide contextual information for other

USAID results data.

IR2. Project/Activity Evaluation Strengthened

A. Evaluation Assistance

In its first year, MESP has begun to cultivate a standardized yet flexible process for planning and conducting evaluations and assessments. Four activity evaluations and one assessment were conducted in the first year; two evaluations were accepted as final and published on USAID’s Development Experience Clearinghouse web portal;

draft reports were submitted to USAID for the remaining two evaluations and assessment.

Through the process of completing these studies, MESP Technical

Specialists became more familiar with tools, processes, and management of these assignments, and assumed increasing responsibility with consulting teams. These assignments have helped to shape a formalized scope of work for Technical Specialists when working on evaluations, including details on roles and responsibilities, and standard operating procedures. Concurrently, areas in which there are gaps in capacity have been identified for intensive capacity building prior to the next wave of evaluations expected in the coming year. Areas identified for capacity building include preparation and SOW best practices; getting the most out of

MSI evaluation templates; linking Findings, Conclusions and

Recommendations; and elements of the final report. The ultimate objective of these efforts is for MESP

Technical Specialists to assume leadership and management of assignments, with MESP international

The team that would be handling the program in Jordan.

Evaluations Completed in Year 1

Learning Environment Technical

Support (LETS) final performance evaluation, Fiscal Reform Project II (FRP II) mid-term performance evaluation, Draft Public Action in Water, Energy and Environment Project

(PAP) final performance evaluation, Draft Youth for the Future (Y4F) final performance evaluation, Draft comprehensive Youth

Assessment in six governorates with more than 700 youth.

USAID/Jordan MESP – January - December 2014 Annual Report 18 staff playing a role that is more advisory than supervisory.

While MESP will continue to implement evaluations and assessments in subsequent years, an important new phase of evaluation assistance focused on assessing and improving evaluation utilization will be launched in the next year. The first phase of this work will consist of interviews with evaluation stakeholders intended to elicit details of how and where the Mission has utilized evaluation and assessment results and, concomitantly, perceptions of what aspects of future studies could be modified to improve their potential for use in Mission decision-making and design. MESP and PRO will also work together to establish a practical standardized system to track and report actions taken in response to specific evaluation recommendations. Highlights of evaluation and assessment activities to-date follow.

Learning Environment Technical Support Final Performance Evaluation

The Final Performance Evaluation of the Learning Environment Technical Support (LETS) activity was conducted, approved and published on the USAID DEC web portal:

http://pdf.usaid.gov/pdf_docs/pa00k1qb.pdf. The purpose of this evaluation was to assess the performance of LETS implementation of learning environment activities and results-based benchmark monitoring systems, while involving stakeholders both locally through schools and parents, and nationally through the Ministry of Education (MoE), so as to inform future programing.

Among LETS’ most notable achievements are community stakeholders attributing improvements in the learning environment to…