Metrics_Project_-_Draft_SOW_2.17.2017.pdf

Section C: Statement of Work

C.1 PURPOSE

USAID/Guatemala’s Metrics Project (Metrics) contract will provide monitoring, evaluation design, and learning support services to the USAID/Guatemala mission. Metrics is the successor project to USAID’s Monitoring and Evaluation Program (MEP) that ends in September 2017 and is managed by DevTech Systems Inc. The period of performance is expected to be from January 2018 to December 2022.

The overall purpose of this project is to enhance USAID/Guatemala’s monitoring and evaluation (M&E) and learning systems and capacity for accountability and adaptation within projects and mechanisms. USAID/Guatemala will use the products and deliverables from this contract to:

inform decision making within the Mission, enable USAID/Guatemala to comply with Agency M&E requirements, enhance the use of evidence in project design and management, and achieve consistency with respect to how data is obtained, organized, validated, and reported to the Mission and Washington.

Generally, Metrics will provide services that fit into three areas; (1) Evaluation Design, (2) Performance Monitoring, and (3) Learning and Adaptation.

C.2 BACKGROUND

C.2.1 Introduction In 2013, USAID/Guatemala awarded the MEP project to DevTech Systems Inc. to support USAID/Guatemala’s Economic Growth (EG) Office and other offices upon request, in particular the implementation of the Feed the Future (FTF) and Global Climate Change (GCC) initiatives.

The primary objective was to provide M&E services for continued performance monitoring, verification, and impact evaluation of program results. MEP also provides support for monitoring environmental compliance of USAID mission-wide programs and activities, completion of periodic reviews, and permitted buy-in from other USAID offices upon request. In 2015, the Mission executed the option period that expires in September 2017.

The current Country Development Cooperation Strategy (CDCS) aims to maximize USAID contributions to initiatives and strategies, including Feed the Future (FTF), Global Health Initiative Plus (GHI) [including the President's Emergency Plan for AIDS Relief (PEPFAR)], the Central America Regional Security Initiative (CARSI), the Global Climate Change (GCC)

Initiative, and Goals 1 and 3 of USAID’s Education Strategy.1 In addition, the U.S. Strategy for Engagement with Central American Strategy (CEN strategy) and the Central America-led Alliance for Prosperity emphasize increasing economic opportunity, reducing crime, and improving governance, which complement the CDCS’s overarching goal and development objectives.

Each of these initiatives and strategies comes with specific data collection and reporting requirements, and several provide specialized tools and resources. For example, USAID’s Bureau for Food Security (BFS) and the State Department’s Office of the U.S. Global AIDS Coordinator (OGAC) require regular reporting on a set of standardized indicators and provide access to web-based and other tools for collecting, maintaining, and reporting these data.

As such, currently USAID/Guatemala and its implementing partners collect a vast amount of highly relevant data in the course of activity implementation, including performance data reported by implementing partners as well as data collected by third-party evaluation, assessment, and survey teams. However, the methods and approaches by which these data are collected, analyzed, and utilized for decision making and reporting vary from team to team, project to project, and activity to activity. This lack of consistency in how the Mission plans for, conducts, and uses M&E products and information for both learning, adapting, and accountability presents management and operational challenges.

Importantly, the Economic Growth Office utilizes an “M&E Clearinghouse”—which includes a web-based Management Information System (MIS), a mapping tool, and an environmental compliance monitoring tool—developed under the MEP contract.2 The Health and Education and Democracy and Governance Offices utilize their own customized performance data management tools. Mindful of the Agency’s moratorium on spending on new information technology systems3 and other requirements, the Mission expects to retain one or more of these systems4 for maintaining and analyzing performance management and environmental compliance data, as well as generating reports for Mission portfolio review and external reporting (e.g., Performance Plan and Report) beyond the life of the current MEP contract.

C.2.2 Theory of Change

1 Goal 1 of USAID’s Education Strategy is “Improved reading skills in early grades”, and Goal 3 is “Increased access to equitable education in crisis and conflict environments for youth.”

2 The MEP contract states that the web-based monitoring and reporting system developed by the contractor must be designed to be directly transferred to another entity upon completion of the contract.

3 See Agency Notice 22423 (February 3, 2012) and Agency Notice 24827 (February 13, 2013), both of which can be found HERE.

4 This could include the implementation of the Development Information System (DIS) currently being developed by USAID/Washington.

IF USAID/Guatemala improves its systems, capacity, and processes for M&E and Learning, THEN Mission staff and implementing partners will be better equipped to understand the outcomes and adapt current projects and activities as necessary, design new projects and activities, prepare for follow-on CDCS development, and contribute to learning and accountability within the Mission and among stakeholders (e.g., Government of Guatemala, Washington, U.S. Congress, and others). Greater M&E and Learning capacity will enhance USAID program management and, in turn, maximize development outcomes.

C.2.3 Mission Overview USAID/Guatemala’s Country Development Cooperation Strategy (CDCS), approved in March 2012, holds as its overarching goal “A more secure Guatemala that fosters greater socio-economic development in the Western Highlands and sustainably manages its natural resources.”

To achieve this goal, the strategy focuses on three development objectives (DO):

DO 1 Greater Security and Justice for Citizens

DO 2 Improved Levels of Economic Growth and Social Development in the Western Highlands

DO 3 Improved Management of Natural Resources to Mitigate Impacts of Global Climate Change

USAID/Guatemala’s CDCS states that the Mission “will invest in high quality monitoring of its development hypotheses and program effectiveness through well-designed performance management systems, as well as rigorous external evaluations of its projects. USAID/Guatemala will use internal monitoring and evaluation systems as well as external performance and impact evaluations to increase accountability and to test concrete hypotheses.” In addition, the CDCS also states that the Mission will use a Collaborating, Learning and Adapting (CLA) approach to provide USAID/Guatemala with the guidance and reference points needed for successful implementation of the overall strategy. The Mission’s Performance Management Plan (PMP) underscores and operationalizes this strategic approach and commitment.

By supporting the Mission’s execution of these approaches to performance management, Metrics will indirectly contribute to achievement of results as outlined in the CDCS Results Framework.

In addition, Metrics will contribute to achievement of the objectives and intermediate results yet to be defined in USAID/Guatemala’s future CDCS by providing targeted and timely data and analysis that will inform strategic planning as well as project and activity design and implementation.

USAID/Guatemala also manages the Regional HIV/AIDS programs that are included in the Regional Development Cooperation Strategy (RDCS). Between implementation of the CDCS and the RDCS, USAID/Guatemala has approximately 35 prime awards, primarily focused in local governance, food security, rural development, humanitarian assistance, economic growth, security and justice, institutional development, civil society strengthening, environment, health, education and youth workforce development.

C.3 SCOPE OF WORK

The overall purpose of this mechanism is to enhance USAID/Guatemala M&E and Learning systems and capacity for accountability and adaptation within projects and mechanisms. The Contractor shall provide flexible, demand-driven services. These services must include technical, analytical, advisory, training, monitoring, and design support services. These services will assist the Mission in complying with evaluation, monitoring, and project design requirements set out in USAID’s Automated Directives System (ADS), chapter 201.

Specific services to be provided by the Contractor include;

M&E technical support to the Mission for performance monitoring at the Goal, DO, Project, and Activity levels, Designing evaluations5, Designing and conducting assessments6, informal reviews, surveys, and other studies, Strengthening knowledge management, and performance monitoring systems, and

Building capacity for M&E and learning among USAID/Guatemala staff and its implementing partners.

The Contractor must use flexible and adaptable planning in case of unanticipated changes. The Contractor shall provide the level of expertise specified to advise on M&E tasks within development sectors that include; agriculture, economic growth, environment, humanitarian assistance, democracy, institutional capacity building, security and justice, health and education programs. The Contractor shall work closely with the Planning and Program Support Office (PPS), and technical teams augmenting their M&E capacity but not substituting their overall

5 Evaluation is the systematic collection and analysis of information about the characteristics and outcomes of strategies, projects, and activities conducted as a basis for judgments to improve effectiveness and timed to inform decisions about current and future programming. Evaluation is distinct from assessment or an informal review of projects.

6 Assessment is defined as a forward-looking process that may be designed to examine country or sector context to inform strategic planning or project design, or an informal review of a strategy, project, or activity. It is distinct from evaluation.

leadership of M&E processes. The Contractor will complement efforts of existing USAID/Guatemala awardees currently conducting their own M&E activities in Guatemala.

C.3.1 Evaluation Design

Tasks:

1. The Contractor shall provide technical and administrative assistance to

USAID/Guatemala’s Planning and Program Support Office (PPS) and the technical teams to design rigorous performance and impact evaluations, including all aspects of evaluation design such as; identifying evaluation questions, drafting evaluation statements of work (SOW), suggesting evaluation methods and other requirements, and advising on data collection and dissemination of findings.

2. The Contractor shall help USAID identify opportunities to integrate the use of mobile technology, and the collection of geo-location data into evaluation SOWs, where applicable and useful.

3. The Contractor shall develop and maintain a registry of evaluation recommendations in coordination with USAID/Guatemala.

Expected Results

● 15 high quality external evaluation designs, both impact and performance, completed between 2017 and 2022.

● USAID/Guatemala evaluation recommendations are logged, tracked, and addressed where applicable.

● USAID/Guatemala staff capacity is improved in areas of evaluation design, methods, and implementation through participation in evaluation design teams and other on-the-job training opportunities.

● Geo-located data incorporated into analyses and evaluation reports.

● New technologies (through integrating mobile and other technologies) are applied to facilitate cost effective, high-frequency, and quality evaluation data collection.

C.3.2 Performance Monitoring7 USAID is currently developing an Agency tool called Development Information Solution (DIS) that will have a performance management module and will be implemented overseas and in

7 Performance monitoring is the ongoing and systematic collection of performance indicator data and other quantitative or qualitative information to reveal whether implementation is on track and whether expected results are being achieved. Performance monitoring includes monitoring the quantity, quality, and timeliness of activity outputs within the control of USAID or its implementers, as well as the monitoring of project and strategic outcomes that are expected to result from the combination of these outputs and other factors. Performance monitoring continues throughout strategies, projects, and activities.

USAID/Washington. The DIS performance management module will lower the burden of data capture and organization and is currently planned to be rolled out to Missions in late 2017. At the same time, USAID/Guatemala’s MEP project developed a performance monitoring system, with especially useful components in GIS mapping and environmental compliance tracking. The system, including these two components is property of USAID/Guatemala. The Mission is considering maintaining those two components in partnership with a local institution or bringing them within USAID’s IT structure if such components are not part of the DIS performance management module. Given the uncertainty around the rollout of the new DIS, USAID plans to include an option in this SOW (see section C.4. Optional Service 1) described in detail below.

Tasks:

1. The Contractor shall provide technical and administrative assistance to the Mission in identifying, tracking, and transferring performance monitoring data to the DIS performance management module.

2. The Contractor shall provide technical and administrative assistance to USAID technical teams in ensuring data quality assessments are completed with the associated performance indicator reference sheets for all indicators reported to Washington.

3. The Contractor shall provide technical and administrative assistance to USAID/Guatemala in its maintenance and regular updates of the mission-wide Performance Management Plan (PMP), and Project level M&E plans. The Contractor shall review project and activity level M&E plans and the Mission level PMP at least twice per year at times identified in coordination with USAID/Guatemala and its technical teams.

4. The Contractor shall provide technical and administrative assistance to technical teams and PPS on indicator selection and development of Performance Indicator Reference sheets (PIRs).

5. The Contractor shall provide on-the-job training to build USAID/Guatemala capacity in performance monitoring through training and technical assistance directly to USAID staff.

Expected Results

● Data management and performance monitoring system (DIS or other) fully functional including, environmental compliance monitoring system, and incorporating the entirety of USAID/Guatemala’s strategy, projects, and activities.

● USAID/Guatemala has on demand access to GIS/mapping data integrated through direct support from the Metrics Contractor or as integrated in performance data management system.

● Data quality assessments completed for existing and new mechanisms (USAID/Guatemala’s current portfolio includes approximately 35 awards to international and local partners).

● Better informed decision making based on accurate and up-to-date performance monitoring information.

● Performance Indicator Reference Sheets (PIRS) are maintained and shared.

● USAID/Guatemala staff and implementing partners’ capacity increased in understanding of monitoring system software and indicator development.

C.3.3 Learning and Adaptation USAID/Guatemala’s current CDCS includes a focus on Collaboration, Learning, and Adaptation (CLA) to continually develop best practices and lessons learned during the strategy period.

USAID/Guatemala requires assistance implementing its CLA agenda to formalize the approach to internal and external collaboration, document the technical evidence base (and disseminate in easy to digest format), and encourage reflection and more adaptive management. The CLA approach to performance management should contribute to overall program management in the following ways:

Collaboration

Identifying new opportunities for traditional and non-traditional partners to work with USAID and with each other towards the goals highlighted in this strategy.

Learning Identifying gaps in quality and availability of data needed to improve implementation, assess impact, and inform program and strategic adjustments.

Reinforcing project-level evaluations and strategically targeted impact evaluations.

Facilitating discussions of hypotheses and evaluation findings within USAID Guatemala, among partners, and alongside other donors and Government of Guatemala counterparts to improve the sharing of key data that inform implementation and partnerships.

Adapting

Using lessons learned to reshape projects, and when need be, DOs.

Cultivating an environment where incentives reward continuous improvements in our work.

Tasks:

1. The Contractor shall conduct surveys, assessments, analyses, and other studies of varying lengths and complexity, as directed by USAID, that contribute to USAID’s learning and adaptation agenda. These could include responding to rapid requests for analysis of existing activity-generated or longer term exercises that require primary data collection and analysis.

2. The Contractor shall provide mapping services to the Mission to demonstrate USAID’s geographic focus, with overlays of socio-economic, security, demographic, and other information that supports USAID’s learning and adaptation agenda.

3. The Contractor shall analyze evaluation data, running correlation and other comparative analyses to draw conclusions and lessons that look beyond a particular evaluation and more broadly at issues that impact other USAID activities and the operating context.

4. In partnership with the PPS Office, the Contractor shall disseminate findings (such as best practices, techniques, and other successes or failures) to ensure that relevant, reliable data reaches the appropriate decision makers in a timely and user-friendly format.

5. The Contractor shall design and implement processes to document and disseminate information. Dissemination will take various forms, including; community of practice events (including giving or facilitating at least twice-yearly presentations to the Mission on changes and developments in Guatemala’s political and economic context), or seminars with implementing partners, program participants, and other stakeholders. This would also include developing tools for integrating and adapting graphics, video, and audio into data collection and dissemination of evaluation and assessment results and better integrating the technical evidence base, lessons learned, and project successes and failures, into USAID’s external and internal communications activities.

Expected Results

● USAID/Guatemala staff capacity improved in synthesizing data to generate information products, infographics, and other messages to disseminate learning and other findings to target audiences.

● Findings from evaluations, studies and other reports are better incorporated into USAID’s project design, communications, and reporting.

● USAID is more responsive and prepared to use and analyze data and analytics/infographics in responding to taskers and other requests for information about projects, planning, and the Guatemalan context.

● Implementing partners are better informed of relevant learning from USAID’s portfolio of activities.

● Relevant, reliable, and timely data and information is available for USG decision makers in a user-friendly format.

● Processes designed and implemented to document and disseminate information.

C.4. OPTIONAL SERVICE 1

With the development and roll-out of any system there is considerable risk of delays or even changing priorities. In the case of significant delays or even cancellation of the DIS roll-out, or to the extent that the DIS platform does not include mapping and environmental compliance aspects within its performance management module, the Contractor shall maintain those modules available to USAID/Guatemala and its partners. Furthermore, in this scenario, the Contractor shall assess the other components of the MEP performance monitoring platform and expand or develop other compliance monitoring and indicator tracking components to be used by each technical team to harmonize reporting definitions, facilitate data entry by implementing partners, and to generate reports, graphics and time series analysis. The Metrics project will have a special option for services to provide additional funding to support improvements to the existing performance monitoring system8.

C.5. CROSS-CUTTING ISSUES ANALYSIS

Metrics is focused on improving information, capacity, and data available for project management and adaptation. The project will help USAID obtain better information on gender equality and empowerment, environmental compliance, and project sustainability that will help the Mission better adapt to improve development outcomes.

USAID’s commitment to advance gender equality and pursue gender as a key development issue must be fully reflected in this activity. Gender considerations must be integrated into this activity. The Contractor must ensure that all USAID policies and strategies, particularly the 2012 Gender Equality and Female Empowerment Policy and ADS 205 are applied during the implementation of this award. These include three main objectives per USAID’s Gender Policy:

(1) highlighting any reductions in gender disparities in access to, control over, and benefit from resources, wealth, opportunities, and services; (2) reductions in gender based violence (GBV);

and (3) increased capability of women and girls to realize their rights, determine their life outcomes, and influence decision-making in households, communities and societies. All individual subject level indicators and data must be disaggregated by sex and age. As such, the Contractor shall integrate gender analysis and propose additional or alternative gender-related evaluation or research questions, as applicable, to measure USAID/Guatemala’s outcomes and impact related to gender issues. The integration of gender issues may include further disaggregation (i.e. vulnerable populations, age, Lesbian, Gay, Bisexual and Transgender and Inter-sex populations, or others).

8 Technical specs and additional information on the Dev Tech performance monitoring system are provided in Annex A.

C.6. COORDINATION AND COLLABORATION

The Contractor shall be responsible for ensuring achievement of all results, products, and reports required under this SOW. The Chief of Party (COP) shall be responsible for the activities of all long and short-term personnel in coordinating with USAID and its implementing partners. The COP will receive technical direction from the COR at USAID. The Contractor shall interact with the relevant technical team members or officers from USAID’s implementing partners. The Contractor shall maintain a strong working relationship across multiple offices within USAID/Guatemala, and with USAID’s implementing partners. The Contractor shall keep the COR or designee informed about the results of those interactions and consultations.

USAID/Guatemala does not envision the Metrics Contractor carrying out any evaluations under this contract. Evaluations will be contracted through separate procurements under a competitive process. The Metrics Contractor may not be eligible to bid on those evaluations due to its role in evaluation design and CLA within USAID/Guatemala.

[END OF SECTION C]

Annex

Resumen del Sistema de Información MONITOR‐G

(USAID/Guatemala M&E Clearinghouse)

Descripción del sistema:

MONITOR‐G es un sistema de información alojado en la nube, accesible a través de internet, desarrollado por el Programa de Monitoreo & Evaluación de USAID/Guatemala, a través de su socio local GEOTECNOLOGICA DE CENTROAMERICA, para suministrar a la Misión de USAID/Guatemala y sus socios, herramientas y servicios de apoyo al monitoreo y evaluación de sus programas de asistencia.

MONITOR‐G fue desarrollado haciendo un fuerte uso de las experiencias y lecciones aprendidas por

DevTech en el proceso de desarrollo de MONITOR Colombia. La lógica de muchos de los procesos clave de funcionamiento del sistema MONITOR‐G ha sido diseñada siguiendo las pautas de diseño establecidas por MONITOR Colombia. En consistencia con el concepto de mejora continua, MONITOR‐G utiliza una plataforma de desarrollo de software más moderna que la utilizada por MONITOR Colombia.

De forma similar, todos los procesos y funcionalidades de MONITOR‐G fueron revisados y optimizados para adecuarse a los requerimientos particulares de USAID/Guatemala y para cumplir con las versiones vigentes de varias regulaciones de USAID, relacionadas con M&E y cumplimiento ambiental.

Especificaciones de la Arquitectura Técnica del sistema:

Nombre del dominio: monitor‐g.net (categorized by Blue Coat as Government/Legal)

Hosting: El sistema está alojado en un servicio de hosting en la nube, denominado winnethost.com; cuya infraestructura está ubicada en California, Estados Unidos

Plataforma de desarrollo: Microsoft Visual Studio .NET

Sistema de Base de Datos: Microsoft SQL Server

Tecnología de Web‐mapping: ArcGIS Online

Desarrollo: El Sistema ha sido desarrollado por el Programa de M&E de

USAID Guatemala, ejecutado por DevTech, a través de su socio local GEOTECNOLOGICA DE

CENTROAMERICA, S.A.

Administración: El Sistema es administrado actualmente por el Programa de

M&E de USAID Guatemala

Estructura del sistema:

De acuerdo con el análisis de requerimientos emergentes de la Misión de USAID Guatemala realizado por el MEP, el sistema MONITOR‐G ha sido estructurado en 5 módulos:

Módulo de M&E: Este módulo tiene como propósito la automatización del manejo de datos de indicadores de los proyectos de USAID/Guatemala, de acuerdo con sus respectivos M&E Plans. Esto incluye la estandarización, ingreso, almacenamiento, sistematización y reporte de datos de indicadores por parte de los socios implementadores. La Misión utilizará el sistema para dar seguimiento al avance y desempeño de sus programas, así como para la generación de reportes consolidados para cumplir con sus compromisos de reporte a Washington.

Módulo de Cumplimiento Ambiental: Tiene como propósito promover el cumplimiento de la Regulación

Ambiental 216 por parte de los socios implementadores mediante: i) la generación semi‐automática de

Environmental Mitigation Plan and Report (EMMPs); ii) automatización del proceso de aprobación de los

EMPRs; iii) reporte en línea sobre el cumplimiento de la implementación de medidas de mitigación ambiental establecidas en los EMPRs; iv) verificación de cumplimiento por parte de la Misión o evaluaciones externas. El diseño y funcionamiento de este módulo fue analizado y aprobado por el REA‐

ECAM, quien además lo propuso como modelo para el desarrollo de la plataforma regional de cumplimiento ambiental que actualmente desarrolla el Buró de Ambiente para LAC.

Módulo de GIS: Este módulo tiene como propósito la utilización de información geográfica generada por los socios implementadores y otras fuentes locales para mapear la cobertura geográfica de las distintas intervenciones los programas de la Misión, a distintos niveles de detalle dependiendo de las necesidades de cada programa. Los mapas web interactivos que componen el modulo GIS del clearinghouse permiten analizar en un contexto geográfico la distribución espacial de los esfuerzos de asistencia de

USAID y sus programas, apoyando la planificación, monitoreo y evaluación de las mismas. También son una innovadora y poderosa herramienta para la planificación de nuevas actividades de asistencia, así como la coordinación de esfuerzos y la construcción de sinergias entre programas que trabajan en un mismo territorio. Actualmente el módulo de GIS contiene información geográfica de todos los programas de EGO y de seguridad alimentaria y nutricional. El módulo ha sido diseñado y construido en cumplimiento de las directrices técnicas emitidas por el GeoCenter de USAID para el manejo de información geográfica.

Módulo de Manejo de Documentos Electrónicos: Este módulo tiene como propósito servir como repositorio organizado de documentos varios relacionados con las actividades de la Misión y sus programas. A través de internet, los usuarios pueden compartir y acceder a documentos electrónicos, los cuales son etiquetados por su creador para facilitar su búsqueda, la cual se hace a través de un motor de búsqueda contextual diseñado para el efecto. De este modo, cada documento cargado en esta librería está identificado con etiquetas de búsqueda que lo relacionan con elementos como: DO, oficina técnica, programa, ubicación geográfica, temática, entre otros. Este módulo se enfoca actualmente en fortalecer las diversas iniciativas de manejo del conocimiento, aprendizaje y creación de comunidades.

Módulo de Comunicaciones: Este módulo tiene como propósito brindar a la Misión y sus programas de un vehículo para comunicar información y contenidos diversos, tanto dentro del sistema USAID, como hacia otros socios externos como entidades de Gobierno y el público en general. A través de internet, los usuarios pueden acceder a contenidos ofrecidos por la Misión y sus socios y que se organizan en 4 categorías: i) documentos electrónicos; ii) mapas; iii) social media y iv) imágenes digitales. Al igual que el anterior, este módulo se enfoca en fortalecer las diversas iniciativas de manejo del conocimiento, aprendizaje y creación de comunidades.