M E_Concept_Paper_01.31.14.pdf

MEASURE USAID/Bosnia and Herzegovina (BiH)

Monitoring and Evaluation Support Activity

Concept (01.31.14)

I. PURPOSE

The MEASURE USAID/Bosnia and Herzegovina Monitoring and Evaluation Support Project has two primary objectives. The first is to provide USAID/Bosnia and Herzegovina (BiH) with technical, analytic, advisory, training, monitoring, evaluation and related support services to assist USAID/BiH in effectively monitoring, evaluating, and relaying information about interventions. This includes designing and implementing both quantitative and qualitative evaluation studies and assessments, as well as providing monitoring and evaluation technical assistance to USAID/BiH’s programs. The contract will support the Mission to implement activities that promote inclusive development, in line with the Agency’s new policies on gender and youth. USAID/BiH will use the products and deliverables to (a) rebuild the Mission’s result-oriented culture, (b) facilitate informed program management, (c) shape the longer term strategic direction of programs and decision-making within the Mission, (d) and enable USAID/BiH to comply with USAID/Forward principles.

The second objective is to build the independent, non-partisan public policy analysis capacity in BiH.

This mechanism will develop local capacity to conduct high quality independent evaluations and other studies for USAID/BiH and other donors. By doing so, the local capacity to use tools and to execute the processes of conducting politically independent analysis of public policy initiatives and the challenges facing BiH will be strengthened. This mechanism and the partnerships it facilitates will help foster a culture of data-driven, results-oriented decision- making.

By the end of this activity, USAID will have created a ‘center of excellence’ for policy analysis in

BiH. This activity will facilitate one or more partnerships between US and BiH universities.

Knowledge, skills, and methods will be transferred to a center of higher learning in BiH. An independent public institution of higher learning is best poised to be the beneficiary of this transfer of knowledge and skills in order to provide the most benefit to the public. This is the best way to ensure that the next generation of Bosnians and a new cadre of political leaders can take advantage of these tools. This will also help promote a public culture of independent and data-driven analysis to take root.

This activity will also help to ensure USAID’s enduring legacy in BiH. Developing the capacity of evaluating the impact of development interventions is the proximal purpose of this activity. However, development projects will not last forever in BiH. The skills to evaluate development interventions lend themselves to be easily transferred into skills to evaluate the impact of public policy, proposed legislation, and organic solutions to the challenges facing the country.

II. BACKGROUND

Rigorous monitoring and evaluation (M&E) is a major focus of USAID Agency reforms to improve accountability and learning as part of the ambitious USAID Forward effort. Towards this goal, Agency performance measurement regulations and evaluation policy mandate rigorous measurement of the performance and impact of project interventions, including evidence-based decision making.

Notable requirements include:

Identifying objectively verifiable indicators, during the project design phase, to effectively measure whether desired results are being achieved;

Identifying evaluation questions during the project design phase;

Collecting high-quality baseline data for variables that correspond to key project outcomes and impacts;

Collecting, maintaining and reviewing performance data;

Reviewing and analyzing performance results and targets at least annually, and updating if needed;

Conducting data quality assessments as needed;

Establishing, in cases where impact evaluations are undertaken, a parallel evaluation mechanism at project inception to accompany implementation;

Conducting at least one performance evaluation of each large project (defined as equal or exceeding in dollar value the mean (average) project size for the Mission ); and

Conducting an impact evaluation of any activity with a project involving untested hypotheses or designed as a “pilot” or “proof of conduct.”

Consistent with this guidance, USAID/BiH is committed to the key principles and practices that foster a results-oriented culture and will make every effort to strengthen its M&E oversight. Towards this end, the Contractor will be expected to work collaboratively with Mission staff and implementing partners (IPs) to provide fast, flexible, robust and rigorous monitoring, evaluation and related support under a single integrated umbrella contract. Additionally, USAID/BiH will make every effort to not only strengthen its monitoring and evaluation oversight, but to also ensure that the local capacity is built to carry on these functions and make them sustainable. The Contractor will not directly monitor projects. The Contractor will neither replace nor substitute for (1) the contractual obligations of

USAID/BiH IPs for their monitoring and (2) the responsibilities for Contracting Officer

Representatives (CORs)/technical offices to carry out their monitoring.

USAID/BiH IPs have developed a range of activity-level monitoring approaches as a basis for collecting, tracking, and reporting performance information to show progress towards identified targets. Most of these activity-level monitoring plans provide updated performance data on a quarterly basis. The various components of the Mission ’s performance information, however, are not linked or reported in a consistent and systematic manner, making it burdensome to track compliance with

Agency guidelines, and difficult to obtain performance information for ad hoc requests. Performance data are not always submitted in the same format by IPs, and the Mission often stores this data in simple text documents or Excel tables that have limited use in the aggregate, making it difficult and slow to report on overall results at the strategic level.

USAID/BiH’s current capacity to expand staff is constrained by a limited USG footprint in BiH.

Thus, in the short-to medium term, USAID has identified certain monitoring, evaluation, reporting, and performance management support functions for delivery through external contracting. These functions will be contracted as a single integrated activity to reduce the burden on staff and to endeavor to meet Agency guidelines.

Equally as important as rigorous M&E is the principle of local sustainability. Therefore, this activity pursues two complementary primary objectives: providing USAID/BiH with the highest quality of monitoring and evaluation services; and encouraging the local capacity to conduct politically independent, publically beneficial policy analysis. The knowledge, skills, and experience built locally to achieve objective one, will be directly transferable and catalyze the achievement of objective two.

The need for local non-biased research capacity in BiH has been magnified since the 2008 economic crisis. According to the “Researchers’ Report 2013 Country Profile: Bosnia and Herzegovina” , BiH has “suffered from the departure of its most expert and highly qualified young people” in the area of research and development leaving a dearth of experienced researchers. The report also indicates that research and development jobs remain unattractive due to low salaries, a lack of social security coverage, limited career incentives, and poor economic opportunities in this arena. The report also http://ec.europa.eu/euraxess/pdf/research_policies/country_files/Bosnia%20Herzegovina_Country_Profile_RR

2013_FINAL.pdf http://ec.europa.eu/euraxess/pdf/research_policies/country_files/Bosnia%20Herzegovina_Country_Profile_RR2013_FINAL.pdf http://ec.europa.eu/euraxess/pdf/research_policies/country_files/Bosnia%20Herzegovina_Country_Profile_RR2013_FINAL.pdf highlights the lack of women in research positions, explaining that the GoBiH has not put in place

“concrete measures to promote gender equality in the research profession or to increase the number of female researchers in top-level positions/decision making bodies. The European Commission’s 2013

Progress Report on BiH also states that “very little progress was made in the area of research and innovation” explaining that almost no actions have been taken to strengthen the capacity of research and innovation at the national level.

The policy brief “Social Sciences and Research in Bosnia and Herzegovina: Between (Lack of)

Financing and (Lack of) Quality” finds that financial support for social science research in BiH is inadequate at around 0.1% of GDP (a drop from 1.5% of GDP in 1990) and is linked to poor research quality. It was also identified in the policy brief “Social Sciences and Research in Bosnia and

Herzegovina: (Is There?) Sufficiently Developed Personnel Potential” that there is no system for training young personnel in the methodology of scientific research in social sciences and that there are an insufficient number of research institutions in BiH. It is therefore necessary to build the capacity of local universities and institutions to fill these gaps. By building local capacity to provide

USAID/BiH with the highest quality of M&E services, we will also leave behind a legacy of independent, results-oriented, data-driven analysis of development problems that will be directly transferrable to the broader realm of public policy decision-making.

III. SCOPE

MEASURE will provide USAID/BiH with flexible, demand-driven services. These services will include technical, analytic, advisory, training, monitoring, and evaluation, and related support services. These are necessary for performance measurement, planning, monitoring, evaluation, reporting, activity mapping, and performance information management and dissemination activities.

These services will assist the Mission in complying with new evaluation, monitoring, and project design requirements under USAID Forward. They include monitoring and evaluation technical support to the Mission for performance management at three levels: the Mission Results Framework, which includes all Development Objectives (DOs); individual project level Results Frameworks; and sectoral requirements – including sectors in which the Mission in engaged and other sectors that the

Mission wishes to explore. MEASURE will support programs initiated under the USAID/BiH

Country Development Cooperation Strategy (CDCS) 2012 – 2016 and future strategy revisions. In addition, it may include assessments, analytical research, and related services required for new project designs, sector strategies, and broader strategy revisions.

It is critical that the Contractor use flexible and adaptable systems and work plans in case of unanticipated changes. Should new M&E priorities arise during the course of this contract, the

Contractor will recognize and adhere to adjustment of tasks in line with the overall M&E contract

Statement of Work. In conducting work under this contract, it is expected that the Contractor will seek input, technical contributions, and other pertinent work or information from relevant BiH and regional stakeholders, including civil society organizations, other academic institutions, Government of BiH (GoBiH), and other public and private organizations, consistent with and supportive of the nature of the work to be conducted.

The Contractor is encouraged, to the extent possible, to partner with other local institutions and consultants in completing the monitoring and evaluation assistance tasks outlined under this contract.

The contractor will be responsible for facilitating and developing the partnership(s) between US-based university (ies) and a BiH public university. For a period of three years, the primary contractor will ultimately be held responsible for carrying-out the components of this activity as illustrated http://ec.europa.eu/enlargement/pdf/key_documents/2013/package/ba_rapport_2013.pdf http://www.rrpp-westernbalkans.net/en/News/Policy-Dialogue-Bosnia-and-Herzegovina1.html http://www.rrpp-westernbalkans.net/en/News/Policy-Dialogue-Bosnia-and-

Herzegovina1/mainColumnParagraphs/00/text_files/file/pol_brief_finances_eng.pdf http://ec.europa.eu/enlargement/pdf/key_documents/2013/package/ba_rapport_2013.pdf http://www.rrpp-westernbalkans.net/en/News/Policy-Dialogue-Bosnia-and-Herzegovina1.html http://www.rrpp-westernbalkans.net/en/News/Policy-Dialogue-Bosnia-and-Herzegovina1/mainColumnParagraphs/00/text_files/file/pol_brief_finances_eng.pdf http://www.rrpp-westernbalkans.net/en/News/Policy-Dialogue-Bosnia-and-Herzegovina1/mainColumnParagraphs/00/text_files/file/pol_brief_finances_eng.pdf below. During this time, and to support the realization of Component 5, the primary contractor will have succeeded in building the capacity of a local public institution of higher learning (most likely through an ‘institute’ or other management unit affiliated with the university) to carry-out these activities on its own. At that time, USAID will provide a direct award to this university-affiliated institute as a transitional award, relying then fully on locally-implemented services. A time period of three years is necessary in order to successfully transition the management of these activities.

At the end of the contract period, the primary contractor will have succeeded in the transition of management of MEASURE by facilitating a mentoring relationship between one or more US universities with a BiH university for the purpose of developing an institute for public policy analysis and encouraging the capacity to manage contracts and other agreements. This relationship will foster a culture of conducting independent, non-partisan, data-driven public policy analysis for public benefit.

At the end of the three-year period, the primary contractor will have also facilitated the development of a center of excellence for public policy analysis in BiH. The higher education institution will have the capacity to conduct politically independent data and results-driven public policy analysis and development intervention evaluations. It will have the capacity and be expected to carry-on all of the activities illustrated below. The institution will also be recognized and respected as a leader in non-partisan public policy analysis on a wide range of national and local challenges, policies, and proposed legislation.

A regional approach should be expected. Other USAID missions in the Europe and Eurasia (E&E) region may work with USAID/BiH to utilize the contractor based in Sarajevo to conduct services illustrated below outside of BiH. The task of facilitating partnerships with universities may also be expanded to other countries, as required. The potential is to create a regional network of politically independent, publically-beneficial centers of public policy analysis. Working collaboratively with

The World Bank-supported Regional Centers for Learning on Evaluation and Results (CLEAR) and other similar institutes may also prove useful in order to expand this network and assure its sustainability.

IV. PROJECT COMPONENTS

The project has five major components as outlined below.

Component 1: Provide technical assistance for the Mission’s Performance Management Plan (PMP), Project M&E Plans, and Activity M&E Plans

Component 2: Design and conduct evaluations, surveys, studies, and sector assessments

A. Performance and Impact Evaluations

B. Surveys, studies, and sector assessments

Component 3: Geographic Information System (GIS) mapping

Component 4: Provide M&E training and technical assistance

A. M&E support to implementing partners

B. M&E support to Mission staff

Component 5: Strengthen local M&E capacity and develop a ‘center of excellence’ http://www.theclearinitiative.org

1. COMPONENT 1: PROVIDE TECHNICAL ASSISTANCE FOR THE MISSION’S

PERFORMANCE MANAGEMENT PLAN (PMP), PROJECT M&E PLANS, AND

ACTIVITY M&E PLANS

The USAID/BiH Mission is revising a Mission‐wide PMP covering the Goal, DO and

Intermediate Result (IR) levels of the Mission’s results framework to incorporate four new project-level logical frameworks. Those logical frameworks are a basis for four new projects. It is anticipated that the four projects will be fully developed by the time this contract is awarded. The

Contractor will be responsible for supporting USAID/BiH in the management of the PMP and project-level M&E plans.

Within three months after Contract award, the Contractor will assess USAID/BiH’s methods of collecting data from implementing partners and monitoring the progress of activities. Based on this review, the Contractor shall recommend a comprehensive and integrated monitoring and reporting system for USAID/BiH that promotes adaptive learning and generates analysis that informs management decisions. The proposed system must streamline data gathering and analysis to promote synergies across sectors, as well as enable the Mission to effectively map interventions.

Illustratively, and in accordance with approved work plans, the Contractor will:

Assist the USAID/BiH Program Office and technical teams in (a) conducting performance reviews, (b) reviewing performance reports; (c) collecting and analyzing data, (d) providing technical assistance in conducting data quality assessments, and (e) operationalizing indicators by assisting in the formulation of definitions and data collection methods.

Assist the USAID/BiH Program Office and technical teams in (a) reviewing all Mission, as well as activity-level baselines and targets; (b) collecting baseline figures where such data does not exist; (c) suggestion revisions of targets; and (d) engage in necessary data collection efforts to ensure that high quality data is presented on all PMP indicators.

Recommend indicators to the USAID/BiH Program Office and technical teams that effectively measure outputs, outcomes, and impacts. The Contractor must take special care to propose appropriate gender-sensitive indicators that measure the effects of activities upon women and men.

Facilitate Data Quality Assessments (DQAs). The goal of the DQA is to ensure that decision makers are fully aware of data strengths and weaknesses and the extent to which data can be trusted when making management decisions and reporting. The major decision point in conducting a DQA is to determine what level of data quality is acceptable, weighing tradeoffs in terms of time and cost of pursuing data. It needs to verify the reliability, accuracy, timeliness, precision and integrity of all performance indicator data.

As needed, train USAID staff by drawing upon the USAID Performance Management Toolkit in (a) data collection, analysis, retention, and use; (b) performance reporting; (c) Performance

Management Plans; and (Data Quality Assessments).

COMPONENT 2: DESIGN AND CONDUCT EVALUATIONS, SURVEYS, STUDIES AND

SECTOR ASSESSMENTS

A. PERFORMANCE AND IMPACT EVALUATIONS

The Contractor will be expected to design and conduct a wide array of performance and impact evaluations. The Contractor will also be required to work on identifying key evaluation questions during the project/activity design.

For each evaluation, USAID/BiH requires the contractor to prepare a Scope of Work (SOW) and work plan for review and approval to ensure that evaluations are objective and meet the USAID requirements for rigorous evaluations. This SOW should include evaluation questions that will address the most important and relevant questions about project/activity performance, and linking questions to specific future decision to be made by USAID, partner government institutions regarding the development hypothesis underlying the USAID/BiH programs.

Illustratively, and in accordance with approved work plans, the Contract will conduct:

Performance Evaluations: The Contractor will conduct at least 10 performance evaluations during the period of performance of this contract. Performance evaluations focus on descriptive and normative questions describing what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period): how it is being implemented; whether expected results are occurring; how it is perceived and valued including consideration of all significant stakeholders including those in Bosnia and other questions that are pertinent to program design, management and operational decision making. Performance evaluations incorporate before-after comparisons, but generally lack a rigorously defined counterfactual. See

Annex B for an Illustrative Schedule of Evaluations for FY 2014 - FY2108.

Impact Evaluations: The Contractor will design and conduct at least two impact evaluations during the period of performance of this contract. Impact evaluations measure the change in a development outcome that is attributable to a defined intervention or a set of interventions. Impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a “treatment” or a “control” groups provide the strongest evidence of a relationship between the intervention under study and the outcome measured. This task will include collection of the baseline data that will include variables that correspond to key outcomes, developing baseline data collection scenarios (use data from established baselines, conduct baseline surveys, reconstruct data, or use other techniques as appropriate), and advising USAID/Bosnia on setting up precise and measurable indicators of the intended changes. See Annex B for an Illustrative Schedule of Evaluations FY2014-

2018.

Cost-Benefit Analysis (CBA) and Cost-Effectiveness Analysis (CEA) are powerful tools in program analysis and evaluation. CBA weights costs against benefits, taking into account long-term impacts and CEA helps to reveal the most cost-effective method when the development objective is already given. These types of analyses can help to establish expected development outcomes and set realistic and measurable targets to have a well informed and evidence-based decision making regarding allocation of funds in project design. These types of analyses are useful for project design, program interventions options, and implementation, e.g., cross-projects analysis of cost-benefit of target beneficiaries, broad-based benefits, including indirect benefits or multiplier effects, to analyze what USAID is buying with its resources

The Contractor will do a CBA as part of every impact evaluation. Additional CBAs and CEAs will be done on as-needed basis.

B. SURVEYS, STUDIES, AND SECTOR ASSESSMENTS

The Contractor will perform sector assessments and provide short-term technical expertise to identify opportunities for Mission interventions to enhance sector performance. Potential sectors include governance, agriculture, energy, industry, education, finance, trade, environment, and conflict management and mitigation. Sector assessments will include an analysis of a sector’s policy and legal framework and institutional capabilities. In addition to the main national players in a sector, the

Contractor will review and assess programs by bilateral and multilateral development partners and make recommendations for the Mission on areas in which it can have the most significant impact while leveraging its financial and human resources with existing programs. Assessments will entail a status update on a particular sector or sub-sector and its weaknesses, strengthens, gaps, and opportunities for value-added intervention by USAID/BiH based on its comparative advantage. The

Contractor could potentially review any of BiH’s sectors – as directed by the MEASURE COR – in order to inform Mission strategic planning, decision-making, and project implementation. Specific details on the questions of interest will vary across assessments and be provided in TOPs prepared by the Mission for the contractor.

In addition, the Contractor will perform thematic assessments on issues that are relevant to

USAID/BiH interests. For example, the Contractor will conduct various gender analyses to assess gaps between genders in BiH, providing recommendations to enhance the leadership and expertise of women and girls, and strategies for meeting their needs. The Contractor may also conduct assessments to analyze issues affecting youth; members of BiH’s lesbian, gay, bisexual, and transgender community; the Roma population: marginalized groups; persons with disabilities;

trafficked individuals; and minority groups.

Illustratively, and in accordance with approved work plans, the Contract will:

Provide short-term and long-term technical assistance on special projects as requested by the

Mission, which may include joint donor assessments.

Design and conduct assessments and special studies, and provide technical expertise, to inform USAID/BiH on strategic planning and project design. Recommend creative and unorthodox approaches to measure project achievements in the complex political environment in BiH.

The Contractor will propose, design, and conduct an annual public opinion or similar survey per year to analyze trends and in attitudes towards civil society issues, rule of law, corruption, and other subjects. More frequent surveys may be required to understand the effectiveness of the mission’s communications and outreach activities.

The Contractor will be responsible for designing, selecting (with participation of USAID/BiH staff), managing, and maintaining (including communicating with participants) and panel and cohort studies. Panels may be an important tool for the mission particularly as it tries to understand the results of policy-level interventions.

The Contractor will design at least three indices (i.e., Bosnian NGO sustainability index, Local Economic Development Index, Judicial Capacity Index, Business Enabling

Environment Index) and collect and report annual data.

Recommend opportunities for focusing resources geographically, within a specific sector or across sectors or on specific populations and beneficiaries within regions.

Conduct analysis to identify opportunities to apply science, technology, and innovation strategically, including opportunities for engaging with the private sector and the university community.

COMPONENT 3: GEOGRAPHIC INFORMATION SYSTEM (GIS) MAPPING

The Contractor will provide and use geospatial methods and information technologies as they are particularly powerful tools for assessing programs and making evidence-based management decisions.

Mapping all the related interventions in a target geographic area can help identify where synergies in programming may occur and help plan for an evaluation sampling plan to investigate if synergies exist. The Contractor will work with USAID/BiH staff and implementing partners to identify opportunities for use of geospatial data and analysis.

The Contractor will ensure that any technology products procured for the conduct of its work are of sufficiently high quality to complete all tasks within a reliable and timely manner as defined by a process agreed upon with the COR. As with other data managed by the Contractor for the purposes of fulfilling the tasks under this contract, the Contractor must collaborate with USAID to ensure compatibility with USAID systems (such as AIDTracker) to allow for migration to a USAID system either during the period of the contract, if warranted, or as part of close out of this contract. All

Geographic Data must be submitted in industry standard formats such as a Shapefile and include metadata. Metadata is a summary document providing content, quality, type, creation, and spatial information about a data set. It represents who, what, when, where, why and how of the resource. It can be stored in any format such as a text file, Extensible Markup Language (XML), or database record. Metadata records include core library catalog elements such as Title, Abstract, and

Publication Data; geographic elements such as Geographic Extent and Projection Information; and database elements such as Attribute Label Definitions and Attribute Domain Values. The suggested metadata format is the XML schema, ISO 19139, which was developed to provide a consistent manner for presenting the ISO 19115 standard.

All Geographic Data must be projected to the Geographic Coordinate System World Geodetic System

1984 (GCS WGS 1984). All data must use the World Geodetic System 1984 (WGS 1984) datum. At the completion of the Contractor’s period of performance, a local partner will have the capacity to manage all the GIS data collected and analyzed under this contract. The local partner will also have the capacity to enter into agreements with other development agencies, public institutions, international and local NGOs, and governments in order to collect, analyze, store, and synthesize data from several sources. It is envisioned that the primary local academic institution in which the

Contractor selects as a partner will be the ultimate manager of this data. However, this is not a requirement and USAID encourages proposals for innovative partnerships with the private sector and civil society to maximize local ownership and ultimately local utility of this data and its analysis.

COMPONENT 4: PROVIDE M&E TRAINING AND TECHNICAL ASSISTANCE

A. M&E SUPPORT TO IMPLEMENTING PARTNERS

This task undertaken for IPs will not replace the M&E requirements that they need to carry out in fulfillment of their contractual agreements with USAID. For example, data collection and reporting requirements are included in acquisition and assistance instruments. The purpose of this task will be to develop and implement a monitoring and training plan for current IPs which will: consider their

M&E needs, include appropriate methods for delivering the training and identify follow‐on actions to ensure the long‐term sustainability of M&E.

Support IPs to to improve their data identification, collection, analysis and reporting, especiallyfocusing on new activities at their early stages.

Help out indicator selection, especially in cross‐cutting areas where data needs to be aggregated to tell a larger story across the Mission portfolio of projects.

Ensure activity-level M&E plans adequately reflect the outputs and outcomes and are aligned with the project M&E plans. Well‐crafted activity-level M&E plans ensure an appropriate means for assessing progress, including having quality indicators with clearly established baselines and targets.

Ensure identified indicators incorporate consideration of gender sensitivities, geographic focus, population segmentation, and similar cross‐cutting strategic issues of importance to

USAID/Bosnia.

Help with indicator selection, especially in cross‐cutting areas where data needs to be aggregated to tell a larger story across the Mission portfolio.

Work with IPs to establish baselines and develop challenging, yet attainable targets.

Training content may include developing and using performance management tools, developing M&E plans, selecting indicators, developing scopes of work for internal evaluations, using performance information for accountability, outreach and decision‐ making.

B. M&E SUPPORT TO MISSION STAFF

The purpose of this task is to provide TA to improve the performance M&E capacities of cognizant

Mission staff, catalyzing USAID/Bosnia’s transformation into an effective learning organization, one that continuously learns through evaluation and research, applying outcomes from that learning to the core business of the USAID/Bosnia Mission. The TA will develop practices, processes, guidelines and policies that support monitoring, evaluation, learning and adaptation.

Build the capacity of Mission staff for M&E through training and technical assistance using a participatory approach to designing and implementing the M&E program.

Provide tailored, results‐based management training covering such topics as performance management, logical frameworks, planning and managing evaluations, designing and testing survey instruments, etc. The teaching approach could include presentations, case studies, and peer learning through group exercises in a face‐to‐face format. Training could include designing and implementing core evaluation training, with a focus on crafting scopes of work, developing evaluation questions, and managing evaluation processes. Training should introduce fundamental M&E concepts, practices and tools which Mission staff can implement immediately. Topics may include: research designs, data collection strategies, determination of indicators and measurement strategies, constructing baseline data, data analysis strategies, reporting results and using results for effective decision making.

Help project design teams and technical offices as needed to develop strategy‐and project‐ level indicators and performance targets, which will eventually feed into the Mission’s

Performance Management Plan (PMP) and project design documents.

Improve organizational practices and capacity for learning and adaptation with the Mission staff and with respect to business processes, deepening coordination, collaboration and synergies within the Mission regarding M&E lessons learned.

COMPONENT 5: STRENGTHEN LOCAL M&E CAPACITY AND DEVELOP A ‘CENTER

OF EXCELLENCE’

Component 5 is directly related to achieving the second objective of this instrument. It is left to the end not to diminish its importance, but to highlight that at the end of the primary contractor’s period of performance, local capacity should exist to conduct all of the above tasks for USAID/BiH. A guiding principle of this task is to support a core function of USAID Forward, enabling local organizations to become stronger, more strategically‐focused entities that may become eligible to receive a direct evaluation award from USAID or other donors in the future. USAID recognizes the critical lack of evaluation capacity in BiH and realizes the importance of local capacity building.

The intent will be to develop the critical capacity of select BiH partners to carry out independent evaluation work. The Contractor will identify the prime institution that will become a ‘center of excellence’ in the field of independent public policy analysis.

The Contractor is encouraged to identify and involve other local companies, organizations, and institutions in order to promote a broad culture shift in BiH to a results-oriented, data-driven analysis of developmental and public policy challenges. USAID discourages local institutions and NGOs to sign exclusive partnership agreements with prime applicants at the proposal phase. The Contractor will work on all tasks specified in this RFP in a “learning‐by‐doing” partnership with local partners, building their capacity to carry out evidence‐based evaluations and assessments, focusing on methodologies to strengthen their analysis and reporting of data.

At the end of the period of performance, the Contractor will have developed the capacity of a local public academically-affiliated institution to manage all of the tasks identified in this scope. This mechanism will develop local capacity to conduct high quality independent evaluations and other studies for USAID/BiH and other donors. By doing so, the local capacity to use tools and to execute the processes of conducting politically independent analysis of public policy initiatives and the challenges facing BiH will be strengthened. This mechanism and the partnerships it facilitates will help foster a culture of data-driven, results-oriented decision- making.

It is impossible, of course, for the full capacity to undertake the breadth of these activities to solely be in the domain of one institution. However, this institution will have the ability to administratively and financially manage each of the activities outlined under Components 1 – 4 above. It will have the capacity to contract for services, manage workload, and oversee consultants and contractors. And, it will be seen as a regional leader in conducting research, analyzing data, drawing conclusions, and, in ways that are politically independent and edifying to the public, presenting choices for courses of action to address BiH’s most intractable public policy and development challenges.

By the end of this activity, USAID will have created a ‘center of excellence’ for policy analysis in

BiH. This activity will facilitate one or more partnerships between US and BiH universities.

Knowledge, skills, and methods will be transferred to a center of higher learning in BiH. An independent public institution of higher learning is best poised to be the beneficiary of this transfer of knowledge and skills in order to provide the most benefit to the public. This is the best way to ensure that the next generation of Bosnians and a new cadre of political leaders can take advantage of these tools. This will also help promote a public culture of independent and data-driven analysis to take root.

V. MANAGEMENT AND STAFFING PLAN

This contract will be managed by the USAID/BiH Program Office with technical input from the

USAID/BiH DO teams. The Contractor will discuss IP performance, IP M&E systems, and IP targets only with USAID, unless otherwise directed by the COR. The responsibilities of the DO and project teams are not to be delegated to the M&E Contractor. The Contractor will be fully supportive of the work of the DO and project teams but will not serve as their representative in the field. The Contractor will make recommendations and observations for USAID program improvements, but solely to

USAID and not to IPs. The Contractor will not make directives or recommendations to IPs;

Contractor work with IPs will consist only of technical assistance, monitoring, verification, analysis, and selected training.

Pursuant to the requirements specified in the Tasks described above, the Contractor will establish an in‐country presence (office) for all work performed under this contract. That office will manage the staffing, logistics, administration, financial, and reporting tasks that this contract will require. In‐ country management will be provided by Key Personnel, who will ensure the requirements articulated in this Statement of Work are fully met. Project management is expected to include cost and quality control of all tasks and assignment undertaken to achieve the objectives of this contract. Key personnel and/or other technical staff proposed should reflect proven experience and success in dealing with the activity identified by this statement of work, such as monitoring, performance management, evaluation design, analysis, and so on.

A. Key Personnel

The key personnel whom the contractor will furnish for the performance of this contract are:

(1) Chief of Party

The position will require a minimum of experience in leading an organization, in both management and technical fields, with a proven track record of programmatic accomplishment, professional achievement, management competence, strong written and oral communication skills, and excellent interpersonal skills.

a) Educational Requirements/Preferred Qualifications:

A master’s or higher university degree in relevant subject areas, preferably in International

Development, Economics, Public Policy, or Business Administrations

b) Desirable Experience:

The COP should have overseas development experience, familiarity with database development and a thorough understanding of monitoring and evaluation.

The proposed COP should have at least ten years of experience working in project management in developing countries, preferably including experience in the Balkans.

The proposed COP should have a proven record of excellent management, leadership, decision‐making, communication and interpersonal skills, including being an articulate speaker and sympathetic listener.

The COP should have demonstrated ability: to assemble, quickly and effectively, and to manage a highly‐skilled, interdisciplinary team; to interact with and lead a broad range of governmental and non‐governmental actors and institutions across sectors; and to implement detailed, organized, timely, accurate M&E programs.

c) Anticipated Roles, but not limited to the following:

Coordinate and be ultimately responsible for all data collection, monitoring, report generation, evaluation and related tasks.

Primary liaison with USAID/Bosnia COR and Contracting Officer.

Ensure quality control and the overall responsiveness of technical assistance provided under the contract, including necessary adjustments to operations to fit the requirements of USAID.

(2) Deputy COP/Senior Evaluation Specialist

This individual will be responsible for leading and managing the technical work related to performance monitoring under this contract. Among other relevant tasks, the Senior M&E Specialist will lead/oversee the review of project level M&E plans, and support such technical work as reviewing reporting methods, carrying out training needs, designing reporting tools and advising DO and project teams on their M&E work. The individual will oversee, recruit for, and participate in evaluations and assessments throughout the life of the contract. S/he will ensure the short‐term technical experts and evaluation team members recruited under this contract are of a high caliber and possess the required technical expertise. The Senior M&E Specialist may also serve as the evaluation team leader of impact evaluations conducted under this contract to be identified.

a) Educational Requirements/Preferred Qualifications:

A master’s or higher university degree in, economics, public policy, statistics, political science, public policy, or other relevant field.

b) Desirable Experience:

Indicator development, data collection and analysis, data quality assessments, and/or performance monitoring and reporting.

Providing guidance to technical teams in developing and updating performance monitoring plans.

Creating and managing performance monitoring plans.

Leveraging performance monitoring data for improving ongoing project management

Designing and implementing performance and impact evaluations of development interventions.

Designing and field testing surveys and other data collection instruments.

Designing, working on and preferably leading an evaluation team.

Project management skills are desirable.

Familiarity with USAID policies related to evaluation and performance management highly desirable.

Strong written and oral communication, and excellent interpersonal skills

c) Anticipated Roles, but not limited to the following:

The individual will contribute substantively to the design and field testing of the evaluation methodology, participatory data collection methods and protocols, data verification techniques, and other technical evaluation and analytical tasks conducted under this contract.

S/he will be responsible for the design, quality, development and completion of all assessments, analytical reports and evaluations.

The key personnel specified in this solicitation are considered to be essential to the work being performed. USAID reserves the right to adjust the level of key personnel during the performance of this contract. Prior to replacing any of the specified individuals, the Contractor will immediately notify both the Contracting Officer and COR reasonably in advance and will submit written justification (including proposed substitutions) in sufficient detail to permit evaluation of the impact on the contract. No replacement of key personnel will be made by the Contractor without the written consent of the Contracting Officer. In recruiting these Key Personnel, the

Contractor is encouraged to identify candidates who exhibit unique combinations of technical, managerial, and interpersonal skills and expertise.

ANNEX A: DEFINITIONS

To ensure consistency in the use of key concepts, the terms and classifications described below will be used by USAID in this contract:

Activity: An activity is a sub-component of a project that contributes to a project purpose. It typically refers to an award (such as a contract or cooperative agreement), or a component of a project such as policy dialogue that may be undertaken directly by Mission staff.

Assessment: A comprehensive review of activities (conducted by USAID, host governments or other institution) in a given development sector of a specific country or region and the institutional, environmental, and policy context in which these activities occur. Assessments highlight issues prevailing in the sector and identify possible approaches to address these issues. Assessments inform the strategy design and revision process of donors and host country partners.

Baseline study: An analysis describing the situation prior to a development intervention, against which progress can be assessed or comparisons made.

Case study design: A non-experimental design that provides an in-depth comprehensive description and understanding of an intervention as a whole and in its context.

Counterfactual: A hypothetical statement of what would have happened (or not) had the program not been implemented.

CDCS: Country Development Cooperation Strategy.

Evaluation: Systematic collection and analysis of information about the characteristics and outcomes of programs and projects as a basis for judgments, to improve effectiveness, and/or inform decisions about current and future programming. Evaluation is distinct from assessment, which may be designed to examine country or sector context to inform project design, or an informal review of projects. There are two types of evaluations in USAID:

(1) Performance Evaluation: Focuses on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporated before-after comparisons, but generally lack a rigorously defined counterfactual. Mixed qualitative and quantitative methods are appropriate

(USAID Evaluation Policy).

(2) Impact Evaluation: Measures the change in a development outcome that is attributable to a defined intervention. Impact evaluations are based on models of cause and effect.

Requires a credible counterfactual, or comparison group, to control for factors other than the intervention that might account for the observed change. Experimental methods are preferred, but quasi-experimental methods may be used (USAID Evaluation Policy).

Experimental design: A methodology in which research subjects are randomly assigned to either a treatment or control group, data is collected both before and after the intervention, and results for the treatment group are benchmarked against a counterfactual established by results from the control group.

Ex-post: (after the fact) refers to the time period after a given intervention has been administered.

Usually used in outcome and impact evaluations to determine the effect of a given intervention on the outcomes measured.

Meta-evaluation: A systematic and objective assessment that aggregates findings and recommendations from a series of evaluations.

Performance management: (Managing for Results) is the systematic process of monitoring the achievements of program activities; collecting and analyzing performance information to track progress toward planned results; using performance information and evaluations to influence decision-making and resource allocation; and communicating results to advance organizational learning and communicate results to stakeholders (USAID Evaluation Policy).

Project: A project is a set of executed interventions or activities, over an established timeline and budget intended to achieve a discrete development result (i.e. the project purpose) through resolving an associated problem. It is explicitly linked to the CDCS Results Framework

Quasi-experimental Design: A methodology in which research subjects are assigned to treatment and comparison groups typically through some sort of matching strategy that attempts to minimize the differences between the two groups in order to approximate random assignment.