ATTACHMENT_J_7_MESP_Learning_Session_Notes.pdf

MESP Learning Session Notes

Thursday September 20, 2018

Overview/background:

• Introduction/purpose of session/overview

• Focal technical work components

• MESP approach: MESP began in Dec 2013, and was first M&E platform of its kind in Jordan. It was designed as a demand-driven M&E support platform, focusing on all four components mentioned below, among others. Because it was the first of its kind, establishing systems and building foundations were a critical focus of MESP’s early years and necessary to enable its current depth and breadth of work.

Cross-cutting lessons learned:

• A focus on learning and adaptive implementation models were key

• Relationships matter including Program Office, Implementing Partners, other stakeholders

• Flexibility in design and implementation of MESP and its interventions was critical

• Dedicated MESP teams and integration across units are crucial to effectively strengthen implementation and data utilization

Capacity Building:

• A hybrid model is most effective o Mix of demand driven vs. supply driven, formal vs. informal capacity building sessions o In next phase of MESP, need to have a unified vision/progression of capacity building based on needs

• Tailored trainings are needed to effectively address varying capacity gaps at all levels o Conducting a needs assessment before training topic decided o There are different knowledge gaps among junior, senior M&E and technical staff that require customized training o Conduct assessments on potential participants before enrolling them to ensure there is a homogenous group of learners.

• Although majority of training is held in English, some trainings are required in Arabic. It is important to have in-house capacity to conduct translations and interpretations. Putting knowledge and skills into practice:

o Practical-based trainings have proved most effective o Ensuring opportunities for practice are critical and remain a challenge

USAID requirements provide concrete opportunities to do so (AMELPs, DQAs) Challenges remain within IPs and other target organizations to ensure M&E staff have the opportunities. MESP and USAID should explore how to better support trained M&E staff o Strengthening MEL at the Activity level requires a wider target audience Beyond M&E staff and beyond Amman

Wider involvement of IP leadership and technical staff is needed to encourage wider Activity/organization-level adoption of skills/practices and data-driven decision making/ learning. Strengthen buy-in and practice of M&E

• Sustainability of Capacity building:

o Strong communities support stronger outcomes

E.g. Apprenticeship program- the importance of follow-up and community building

Community of Practice – a new idea for M&E practitioners, needed guidance and facilitation to get this going.

o Sustainability of capacity building requires thinking beyond the classroom Engage strategic communications personnel, online platforms, think about job aids, guidebooks/reference materials, etc.

• Engaging Non-USAID Stakeholders: a culmination of MESP lessons learned o Relationships matter (especially because MESP doesn’t have leverage outside of USAID stakeholders, but towards the end of the project, MESP was able to reach local civil society organizations that were enthusiastic to take part in MESP’s training sessions.)

o Understand your new audience and tailor interventions o Diagnostics are important to tailor trainings to new target participants for example the

Certificate Program for Evaluations (CPE) had a wide array of participants with different skills so it was a challenge to keep the training on track, the same was felt during the Apprenticeship program.

• Working through IPs has been a fruitful avenue to engage other stakeholders for example through the Apprenticeship program and the CPE training. Follow-up and community building is key for strong relationships needed to expand reach and strengthen outcomes. Dedicated MESP team/staff were critical to expand capacity building offerings (beyond a training specialist)

• Training Topics that were not tackled during MESP and are of interest o Data analysis for learning and adapting o Data visualization o Soft skills, negotiation skills, leadership skills to empower M&E professionals at USAID

Activities with the skills to engage and get buy-in from senior management and technical staff o Remote Monitoring o Basic excel, SPSS, different quantitative data collection methodologies o Training specific for Activities held at their offices for M&E and technical team members.

Evaluation:

• Early and on-going collaboration with USAID was key o Commissioning evaluations, designing SOWs, managing stakeholder (USAID) expectations

• Ensuring quality evaluations and evaluation products require strong evaluation teams and intensive MESP support o Teams: Unpredictable quality of external evaluators until they are on the ground. Key skills: local experts are best when they have a deep understanding of local stakeholders and networks; team leaders have strong management and communication skills and a focus on synergy (among team, between qualitative/quantitative data) as well as having strong evaluation expertise versus technical expertise as well as write up skills; team must have strong writing/note taking skills, Arabic language skills (more in-depth, open, and efficient data collection, quality analysis); Team lead should finalize the write up of the report. The team should have a person manage the logistics as this takes a lot of time and effort from the MESP staff member who is “managing” the evaluation.

Recruitment is an on-going challenge: pay scales vs. desired skills, need for greater knowledge management on consultant strengths/weaknesses Finding the balance between theoretical and practical knowledge/expertise.

Need structured quality control of evaluations, and have a dedicated team member to do quality control and inter-reliability testing o MESP support:

Dedicated Home Office support, dedicated unit/staff proved to be useful due to timeframes and intensity of workflows (administrative, writing/editing) Support teams with activity background/understanding and history (once on the ground, there is not enough time in an evaluation to fully understand Activities. Have MESP staff play a role in teams in order to provide that background information along the way)

Quality control – understanding and meeting USAID requirements and preferences (knowing the audience/writing to the audience), integration of quant/qual data, ensuring all questions are answered, ethics and international best practices in various aspects of evaluations ie conducting and dissemination; Findings Conclusions and Recommendations are all good additions to the evaluation process

Building local evaluation capacity is a long-term process, but can help strengthen evaluation quality and can only be learnt through practice. Have to learn by doing/being involved in evaluations

Link MESP M&E capacity building efforts/target groups with MESP evaluations (involve same target audiences as relevant)

Invest in junior evaluators/staff Build on expertise pool (example RAMP local large scale data collectors)

• Evaluation utilization requires an integrated approach and innovation o Program Office and MESP introduced Co-generation of recommendations which ensured evaluation findings were utilized by USAID. Need for both formal and informal work with partners on evaluation utilization (formal: utilization follow-up, dissemination events and products; informal: follow-up and work with stakeholders before final evaluation report approved, MESP technical support) o Timing of evaluations (mid-term or linked to strategic decision making cycles/opportunities) have been important o Don’t let evaluation bi-products sit on the shelf (survey results, etc.)

o Have a CLA person to manage this

• Strengthening evaluation practice requires regular, intentional learning processes o Institutionalize After Action Reviews/reflections on process to improve o Institutionalize evaluation processes; example six-months reviews – reviews of successes, challenges of a specific evaluation post conducting each evaluation.

o To be flexible in applied research and theoretical research best practices o Improving quality within time constraints by building a pool of knowledge in preparation of an evaluation (ie summaries of report reviews, knowledge of an activity, desk reviews…etc.)

Performance Data and Information systems (MIS/GIS/KaMP)

• Deep understanding of information systems and partner data supported an efficient and effective establishment of MIS/GIS system

• Relationships and one-on-one support were crucial during system roll-out and to encourage utilization

• Having a dedicated core team of M&E and GIS/MIS specialists was key, however 4 people was not enough to have time to think strategically about data utilization and analysis and be creative.

• MESP focused on establishing the systems and the next step is to focus on utilization of this data

• DevResults and KaMP need to be more interactive and sustainable

• Data quality remains a challenge for MIS/GIS systems o Work with AORs/CORs and strategize within MESP to review performance data entered into DevResults o GIS data collection system needs to be strengthened (data collection software and IP capacity strengthening), and managed through a relational database

• Now that the information systems are established, there are opportunities to strengthen utilization o A hybrid approach to utilization is needed, responding to demand while also pushing out new ways to utilize data to inform and strengthen demand e.g. Performance data visualization tailored to user needs and timeframes, interactive maps o Integrate information management and technical M&E teams to design and implement strategies to increase utilization o Deeper understanding of stakeholder uses and constraints is needed to further increase effective utilization of data for learning/decision making o Expand target audience for utilization to include the Mission and IPs for DevResults, GIS and KaMP, as well as the wider development community for KaMP

Strategic Communications

• Stronger integration with technical M&E teams are needed to strategically push relevant information out to the M&E community through the various communications channels

• Strategic communications need to be integrated as a required element into any intervention from the design phase to maximize

• Greater human resources dedicated to strategic communications can expand the level of involvement of communications in MESP interventions and variety of products produced

• Strategic communication will be critical of the success of the project which will enhance learning