Attachment_B_Draft Statement of Work.pdf

DRAFT STATEMENT OF WORK

Performance Monitoring and Evaluation Support (PMES) Activity

1 - Purpose

The Performance Monitoring & Evaluation Support (PMES) Activity will provide high-quality, reliable, timely, and objective performance monitoring services to help ensure sufficient oversight of USAID/Pakistan activities. More specifically, the Activity will enable USAID/Pakistan Agreement/Contracting Officer Representatives (A/CORs) to use independent monitors to verify and report on activity implementation when USAID staff are unable to conduct on-site monitoring visits due to a number of reasons, including but not limited to: non-permissive security conditions; difficulty obtaining Pakistani government approval to visit certain geographic locations, and unavailability of Embassy assets to facilitate site visits.

The primary focus of the Activity will be independent monitoring; however, pending available funding, and based on the capacity of the award holder(s), the mechanism also may be used for select evaluations or assessments related to learning priorities of the Mission.

This Activity will help inform and improve implementation of the Mission’s Country Development Cooperation Strategy (CDCS) and support achievement of the strategy’s development objectives. It also will assist USAID fulfill its responsibilities as a steward of taxpayer dollars by helping to prevent waste, fraud, and abuse in the administration of activities.

2 - Background

USAID/Pakistan Portfolio The CDCS, which defines USAID’s development assistance priorities for the Islamic Republic of Pakistan for the period 2018-2023, aims to achieve a more stable, peaceful, and prosperous Pakistan.

The strategy focuses on helping to meet some of Pakistan’s most pressing needs, including greater security, especially along the Afghanistan/Pakistan border, more inclusive economic growth, and improved governance, all of which will advance the country’s journey to self-reliance.

USAID/Pakistan works in a number of sectors, including health, education, infrastructure, economic growth and agriculture, energy, peace, and governance, nationwide. The Mission currently has around 60 activities, though its overall portfolio likely will be reduced in size over the course of the current

CDCS.

Monitoring and Evaluation For the past eight years, USAID/Pakistan has had a monitoring, evaluation and learning support contract, the latest iteration of which was called PERFORM. PERFORM and its predecessors provided a number of services to support the Mission’s monitoring, evaluation and learning (MEL), including evaluation and assessment support, third-party monitoring, and geographic information system assistance, including mapping. Moving forward, USAID/Pakistan requires a more calibrated scope and approach to M&E support services in line with the Mission’s portfolio of activities, focusing on performance monitoring.

This Statement of Work (SOW) reflects this revised scope.

The Mission has historically used third-party mechanisms to conduct infrastructure Quality Assurance/Quality Control (QA/QC) and will continue to do so until all infrastructure activities are completed. While information sharing between the QA/QC and M&E support contracts managed by Mission offices is encouraged, this Activity will operate independently of any other M&E support mechanisms. The Contractor for this Activity is solely responsible for providing the deliverables described in this SOW.

USAID/Pakistan fully complies with Agency guidance on monitoring, evaluation, and learning, including the USAID Evaluation Policy (2011) and the Automated Directives System (ADS) 201 Program Cycle Operational Policy. In addition, the Mission has implemented an approach to enhance its performance monitoring, which is based on Agency best practices and recognizes that, particularly in environments where it is difficult to access activity sites, it is important to rely on a variety of monitoring actors to gather activity data. Under this Multi-Tiered Monitoring (MTM) approach, activity managers gather and analyze data from multiple sources, or “Tiers,” including: 1) USAID contact with the Implementing Partner, 2) Document Review, 3) USAID Site Visit, 4) Independent Monitoring, and 5) External Sources, to document findings, triangulate information to ensure confidence in the reporting, and use the results to inform programmatic decisions. This Activity will provide the requisite services to support activity managers and other USAID staff members in meeting “Tier 4” of the MTM approach.

This Activity is expected to provide performance monitoring support across the entire Mission; offices that will utilize this Activity include: the Offices of Health, Population and Nutrition (OHPN), Education (OED), Infrastructure and Engineering (OIE), Economic Growth and Agriculture (EGA), Energy (ENR), Stabilization and Governance (OSG), Community Resilience Office (CRO), and regional offices, including Khyber Pakhtunkhwa, Lahore, and Karachi.

3 - Scope of Work

The Activity will perform tasks in accordance with requirements in USAID/Pakistan’s CDCS, Performance Management Plan (PMP), Mission Notices and Mission Orders on Performance Monitoring and Evaluation, individual project- and activity-level Monitoring, Evaluation and Learning (“MEL”) plans and activity-level Multi-Tiered Monitoring (MTM) plans. The Contractor’s performance of the components below must also comply with quality standards outlined in ADS Chapter 201 and the USAID Evaluation Policy.

3.1 - Component 1: Monitoring Support

This Activity will provide high-quality, timely, and useful performance monitoring support services to help USAID/Pakistan meet its performance monitoring responsibilities. Contracting Officer Representatives (CORs), Agreement Officer Representatives (AORs), Activity Managers (AMs), and other USAID/Pakistan staff members (together, “Technical Representatives”) require performance monitoring data to help identify activities with implementation challenges; improve and adapt activities during implementation; and inform new activity designs.

The Activity will utilize creative, innovative approaches to mitigate the challenges of monitoring in remote or difficult-to-reach areas. These approaches must remain sensitive to the operating environment in Pakistan and may include a variety of tools such as: Global Positioning System (“GPS”) location-stamped photos, data from GPS tracking devices, and/or remote sensing, surveys (e.g., rapidly deployed crowd sourced/community mobilization or longer-term door-to-door household surveys), remote data collection using mobile devices (e.g., SMS, text message, or Interactive Voice Response), and/or crowdsourcing.

The Activity will also provide monitoring support services for direct Government-to-Government (G2G) agreements that USAID/Pakistan has executed with the Government of Pakistan (GOP). Where government capacity and/or systems are limited, restricting the Contractor’s ability to conduct monitoring support tasks, the Contractor will utilize alternative methods that mitigate these weaknesses.

In the end, the Contractor must adapt and conform its approaches and processes to the requirements in the host country agreements (i.e., some agreements have very specific M&E and performance management requirements and other agreements cursorily mention them) depending on the scope and scale of the activity. Further, the Activity must consider the use of host government systems for data collection, as determined by USAID, but remain prepared to supplement this data to ensure that systems yield high-quality information.

The Mission developed a number of monitoring data collection and analysis tools under the previous M&E contracts. These tools will be shared with the Contractor for reference and use under this Activity.

Specific performance monitoring support tasks will include, but are not limited to:

1 - Conduct Site Visits and Verify Activity Implementation:

The Contractor will conduct site visits and activity verification according to the activity MEL Plans, MTM plans and/or work plans. This includes monitoring support services for projects and activities located in environments that pose challenges to accessibility, especially in Khyber Pakhtunkhwa (KP) province, including the Merged Areas (MAs), South Punjab and Northern Sindh. This monitoring will provide technical representatives with the necessary data to verify partner compliance with award requirements, including but not limited to, activities conducted under the work plan, results achieved, participant training requirements, environmental requirements and plans, climate change adaptation, and marking and branding compliance. For example, the Contractor will verify activities such as commodities and equipment delivery and usage, community outreach and mobilization, and capacity building. The Contractor will analyze the performance data it collects from site visits and will flag inconsistencies or red flags that arise during monitoring immediately to USAID.

The Contractor will avoid the duplication of data collection efforts by actively coordinating with USAID/Pakistan and its implementing partners to establish a protocol or schedule for joint data collection across sectors or geographical areas, and will create and utilize performance monitoring tools, such as a standard checklist for use during regular site visits. The Contractor’s reports will utilize a standard template and will be made available to Mission technical teams and implementing partners in electronic form, as guided by USAID.

In all cases when dealing with Implementing Partners, the Contractor shall be careful not to give the impression that it is, or is acting as, a representative of the U.S. government (USG), nor shall it attempt to enforce USAID policies or regulations.

For infrastructure activities, USAID/Pakistan has executed agreements with engineering firms to conduct oversight of construction tasks. Nevertheless, the Contractor will support the monitoring of higher-level results of those construction activities (e.g., roads, dams, power plants, schools or clinics constructed) on beneficiaries.

2 - Conduct Data Quality Assessments:

The Activity will support the completion of Data Quality Assessments (DQA), which include recommendations to ensure the soundness of data collection methods and common definitions, and facilitate aggregation of data from multiple partners/sources. The Mission anticipates the need for annual DQAs that will contain not more than 20 new or updated indicators. The contractor will use standard checklist and guidance documents, provided by the Mission, to conduct and record the DQAs.

3.2 - Component 2: Evaluation and Assessment Services

If required, and pending available funding, this Activity will complete high-quality, timely, useful, and select (no more than 2-3 per year) performance evaluations and assessments of USAID/Pakistan’s development projects and activities, which will enable the Mission to improve the implementation of these projects and activities, and plan future projects and activities in support of the results defined in the CDCS.

The Contractor will design and implement methodologically rigorous evaluations and assessments, including the following:

● Mid-term and final evaluations; assessments or special studies to inform USAID’s decision-making process such as on resource allocation, geographic focus and the design of a new project or Mission strategy;

● Ad-hoc analyses and/or surveys to inform a development objective or cross-cutting programs;

● Recurring assessments in various areas including but not limited to, Knowledge, Attitudes and

Perceptions (“KAP”); gender; early grade reading; and sectors relevant to the Mission’s work, as requested by USAID.

● Baseline, midline and end-line data collection across the development portfolio to evaluate the outcomes of USAID/Pakistan programs, projects, and activities, including the impact on beneficiaries, communities and local organizations.

The Contractor will use rigorous methods that meet international and Agency standards for evaluations and assessments. The Contractor will adopt a participatory approach when conducting evaluations and assessments, involving USAID implementing partners and other stakeholders, as appropriate.

The Contractor’s evaluations of G2G awards may utilize specialized approaches. G2G activities require a close working partnership with government officials, and evaluations of these activities must support this relationship. Further, the Contractor must consider the use of host government systems for data collection, but remain prepared to supplement this data if systems cannot yield high-quality information.

4 - Implementation Approach

Each task under this Activity will be implemented as follows:

I - USAID/Pakistan SOW: For each M&E task, the requesting USAID technical office will develop an SOW that will describe the context, purpose, learning objective and audience of the requested services, background information, activity description, list of already existing information, key M&E questions to be addressed, list of deliverables required, frequency of reporting, expected time for deliverables and other requirements as applicable.

II - Workplan: Within two weeks of receiving the SOW, the Contractor will develop a comprehensive work plan describing the proposed implementation approach. The work plan must include detailed methodologies for data collection and data analysis, sampling approach, potential limitations, proposed staffing structure, list of deliverables, anticipated schedule of activities, estimated level of effort and proposed budget.

III - Data Collection and Analysis Plan: In the task work plan, the Contractor will clearly explain all data collection methods it plans to utilize in the field (e.g., surveys, interviews, direct observation), and provide the tools (e.g., questionnaires, interview protocols) for USAID’s review and approval.

Where applicable, the methods should be grouped to similar M&E questions and data triangulation should be considered. The contractor will also develop a data analysis plan that will be applied when processing the obtained data. This plan will directly address each key M&E question with specific methods for collecting and for analyzing the data that will be used to answer it.

IV - Reporting and Data Sharing: The Contractor will provide the performance data it has collected, as well as the analysis of this performance data, which will include lessons learned and recommendations for improving activity performance (developed in collaboration with Mission staff), in monitoring and evaluation reports and in formats and frequencies, as guided by USAID. Reporting may also include short briefers, regular check-in meetings, presentations/debriefs to USG or other stakeholders, evaluation recommendation workshops, submission of raw data, and upload of final deliverables to the USAID Development Experience Clearinghouse (DEC). Reporting should be clear and accessible to various stakeholder audiences. Evaluation reports must meet the standards outlined in ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report.

The regular information gathering that the Contractor will conduct throughout Pakistan will generate a large amount of data. The Contractor must make this data readily available to USAID personnel in an easily accessible and understandable format. The Contractor must work closely with the COR of PMES activity to ensure that appropriate data transfer methods are in place between implementing partners, USAID/Pakistan, and other stakeholders.

V - Human Resources and Planning: For all monitoring and evaluation services, the Contractor will provide sufficient human and technical resources for team planning meetings at the beginning of the task as well as for sufficient team preparation prior to fieldwork. The Contractor will employ methodologies that ensure systematic stakeholder input at appropriate stages of planning and implementation. This includes a review of the qualifications of the proposed field teams and the provision of methodological tools and input from requesting Mission staff. The Contractor will budget sufficient time for both data collection and analysis and will provide the human and technical resources necessary for drafting quality reports and presentations of conclusions and recommendations. The Contractor must maintain professional ethics and a business code of conduct for the performance of fieldwork by consultants.

The Contractor will provide specialized staff for appropriate time periods for sector-specific tasks.

Possible sector-specific knowledge should include: energy, water, agriculture, service delivery, private sector engagement, conflict prevention, education, health, social cohesion, governance, economic growth, gender, environment, climate change, micro-finance, and competitiveness, among others.

The anticipated positions for M&E services include:

● Task Manager

● Performance Monitoring Specialist

● Evaluation Specialist

● Sector Specialist

● Gender Specialist

● Data Analyst

● Report writer

VI - Logistics and Resolution of Discrepancies

The Contractor will manage the logistics associated with executing each task, including but not limited to, arranging planes, trains, rental cars, visas, etc.—including field visits to remote or insecure areas.

The Contractor will need to interact with many of USAID/Pakistan’s implementing partners.

Consequently, the Contractor must work with the PMES COR and the implementing partners to establish a coordinated system of communication and information sharing, as well as a protocol to de-conflict data collection. The Contractor will develop an approach for engaging multiple key stakeholders in task development that makes the most efficient use of limited time, such as multi-stakeholder engagement processes. The Contractor should ensure coordination and non-duplication of effort with other similar activities conducted through the Mission, particularly site visits.

On occasion, disagreements over the findings of the Contractor’s tasks may arise. At the PMES COR’s request, the Contractor will appear for meetings with USAID staff and/or technical project implementing partners to explain any discrepancies between their findings and the findings of the technical project implementing partner’s. At such meetings, the Program Office’s Performance Management Unit (PMU) and relevant technical office staff will discuss how the discrepancy will be resolved.

5 - Contractor Coordination with the PMES Activity COR

The PMES Activity COR (referred as “the COR”) will coordinate with USAID Technical Representatives and the Contractor on individual activities required by the Activity. The Contractor will work closely with the COR on all relevant coordination issues related to implementation of activities.

The COR will define a workflow process, which will be finalized with the technical representatives upon start up.

The Contractor must establish an internal system to track time and expenditures on the required components and specific activities ordered in the Activity. The Contractor will define the exact requirements for this system in consultation with the COR and incorporate it in a workflow document.

6 - Mobilization of Support Services

The COR will develop a list of the activities that will need monitoring, evaluation and assessment services, along with activity-specific requirements, and related documentation (e.g., activity work statements) during the revision of the USAID/Pakistan PMP, which should be completed in Summer 2020 and will be regularly updated by the end of each fiscal year.

The COR will base his/her determination of the needed services on the following criteria:

● Well-defined, clearly stated purpose (e.g., to inform new projects/activities, test a development hypothesis, meet current administration priorities, global lessons, etc.) and supports learning on Mission identified cross-cutting themes and special issues;

● Achievable within the expected timeframe, geographic location, and budget, with sufficient quality and comprehensiveness that will provide useful, quality results;

● Supports clarity for programs/projects/activities that have more complex monitoring needs or are in difficult to reach geographic locations;

● Timing aligns with the appropriate CDCS program/project/activity phase;

● Achieves a sufficient degree of cost benefit (including factors such as location, level of effort, potential replication, etc.) given the size and importance of the program/project/activity;

● Balances support to all interested technical and regional offices and Development Objectives; and,

● Responds to or follows up on audit findings.

In order to expedite startup, the Contractor will work with the COR to develop templates for routine activities, such as site visits, leveraging templates that have been developed in the past. The COR will bundle similar requests for services into one request or summary document, when possible.

7 - Gender Considerations

USAID requires Contractors to undertake efforts to prevent discrimination and violence against women and girls, provide economic and leadership opportunities, increase participation of women in the political process, improve security for women and girls, promote education, health and well-being, and other efforts designed to directly benefit women and girls.

The Contractor will support USAID/Pakistan’s implementation of the Agency’s Gender Equality and Female Empowerment Policy with respect to monitoring, evaluation, and learning, as outlined in ADS

205.3.8. This support will include incorporating gender-specific requirements in all monitoring and evaluation tasks and advice on any course corrections that could further close gender gaps and promote gender equality and female empowerment.

The Contractor will support the Mission in ensuring that Agency and CDCS gender requirements are met by verifying participation of women through its monitoring support activities, disaggregating by sex all people-level indicator data it collects, and analyzing data for gender differential effects.

The Contractor will ensure that both female and male voices are addressed in methodologies, data collection and analysis; specifically, the Contractor will undertake gender and culturally appropriate approaches (e.g., separate focus groups, female data collectors, effective utilization of technology, etc.)

for all monitoring, evaluation, and assessment tasks. Furthermore, the Contractor will ensure that the statements of work for all evaluations it conducts identify evaluation questions for which gender specific or gender differential impacts are expected and design the evaluation according to Agency requirements for gender implicated monitoring, evaluation, and learning.

In addition, the Contractor will specifically support USAID/Pakistan to ensure that its programming appropriately addresses gender by conducting gender assessments and analyses for specific sectors.

In addition to the above responsibilities, the Contractor will perform the below activities:

● Collect, or identify possibilities for collecting, sex-disaggregated data and geographic disaggregates (province and district).

● Review and analyze data against planned targets with attention to whether there are any gaps between the extent to which females and males are participating in and benefitting from projects and activities, as appropriate, and discuss the findings with implementing partners;

● Analyze unexpected results (positive or negative) affecting females, males or both, and discuss the findings with implementing partners;

● Include questions, or encourage the use of questions, in evaluations and assessments about male and female roles to uncover intended and unintended positive and negative changes;

● Identify all evaluation questions for which sex-disaggregated data are needed;

● Consider and advise USAID/Pakistan on whether key evaluation questions examine the extent to which closing gender gaps has improved project outcomes and whether the project has transformed gender norms and reduced gender gaps, and if requested, provide guidance on how to incorporate this aspect into evaluation question(s);

● Ensure that evaluation and assessment designs, methodologies, data collection and analyses adequately capture the situations and experiences of both males and females;

● Field interview teams that are comprised of both males and females appropriate for the cultural context and data being collected;

● Ensure equal participation of women and men in activities under this activity.

8 - Estimated Anticipated Deliverables

(a) The contractor must electronically submit all written deliverables to the COR.

Additionally, the COR may request bound or color deliverables. These deliverables may include large maps or other innovative displays of data, as needed.

(b) The below table represents an estimate of the number and types of deliverables anticipated under this activity. The Activity Manager will work with the Contractor and USAID/Pakistan staff to update the deliverables required under the terms of the agreement, which will be incorporated into the Annual Work Plan.

Estimated Anticipated Deliverables Table

Component/Section Estimated

No.

Estimated Frequency

Deliverables per Item

1a: Monitoring - Site visit verification and compliance-related services

500 site visits (locations)

Annual ● Task Work Plan, including detailed methodology, sampling, data collection plan, data analysis plan and other deliverables, as required by USAID;

● Data collection tools;

● Site visit checklist/ reports;

● Consolidated report/s (frequency and format to be determined by USAID based on the individual task); and

● Debrief, presentation and/or meetings on findings and recommendations.

1b: Monitoring - Data Quality Assessments

20 Indicators Annual ● 1 DQA form/indicator at partner level;

● A consolidated report, if requested by USAID; and

● Debrief, presentation and/or meetings on findings and recommendations.

2: Evaluation 2-3 Annual ● Implementation Plan and methodology, including sampling plan, data collection plan and data analysis plan;

● Data collection instruments;

● An initial findings debriefing discussion;

● A draft evaluation report;

● A final evaluation report consistent with USAID standards for evaluation reports (outlined below) and no more than 30 pages (excluding annexes);

● A presentation to USAID,IPs and other stakeholders on the final report;

● Raw data available electronically;

● All data sets submitted to the Data Development Library;

● Evaluation data utilization workshop and

● Final Report uploaded to DEC.

3: Assessment 2 Annual ● Implementation Plan and methodology, including sampling plan, data collection plan and data analysis plan;

● Data collection instruments;

● An initial findings debriefing discussion;

● A draft report;

● A final report;

● A presentation to USAID and stakeholders;

● Raw data available electronically; and

● Final Report uploaded to DEC, if required.

9 - Log Framework

NARRATIVE SUMMARY INDICATORS DATA SOURCE ASSUMPTIONS & MITIGATION

Goal: Effective and sufficient monitoring of USAID/Pakistan portfolio

Purpose: Provide high-quality analysis and evidence-based information to support oversight of USAID/Pakistan activities

Percentage of USAID/Pakistan staff satisfied with the services

Program Office will gather data from a survey of USAID/Pakistan recipients of contract services and products.

● USAID/Pakistan staff turn-over may bring in staff with new perspectives or differing requirements. This may affect satisfaction.

● Satisfaction may be defined differently by various USAID/Pakistan offices.

How are these risks addressed?

● Ensure that FSN USAID staff is a part of the assignment team and included in the survey.

● Clearly established SOQs (standards of quality) that are consistently applied in COR and technical staff’s review of products.

Output: High-quality products and services provided for effective performance management in timely manner

Percentage of final products that meet the stated purpose and requirements outlined in the approved task work plan.

Percentage of final products received in the timely manner for decision-making process.

Final products that are approved by the COR

● Assignments are designed to take place in a timely manner to be relevant for decision-making.

● Assignments are developed in collaboration with technical offices.

How are these risks addressed?

● Quality standards are clearly understood and integrated into the SOW of each task.

● Ensure that the criteria are robust and applicable to different types of tasks.

● Timelines are clearly defined in the SOW and work plans