MCC-15-RFQ-0148_-_Solicitation.pdf

PAGE 1 OF 1. REQUISITION NO.

2. CONTRACT NO. 3. AWARD/EFFECTIVE DATE 4. ORDER NO. 5. SOLICITATION NUMBER 6. SOLICITATION ISSUE DATE

a. NAME b. TELEPHONE NO. (No Collect Calls) 8. OFFER DUE DATE/LOCAL

TIME

9. ISSUED BY CODE 10. THIS ACQUISITION IS UNRESTRICTED OR SET ASIDE: % FOR:

SMALL BUSINESS

HUBZONE SMALL

BUSINESS

SERVICE-DISABLED

VETERAN-OWNED

SMALL BUSINESS

WOMEN-OWNED SMALL BUSINESS

(WOSB) ELIGIBLE UNDER THE WOMEN-OWNED

SMALL BUSINESS PROGRAM

EDWOSB

8(A)

NAICS:

SIZE STANDARD:

11. DELIVERY FOR FOB DESTINA-

TION UNLESS BLOCK IS

MARKED

SEE SCHEDULE

12. DISCOUNT TERMS

13a. THIS CONTRACT IS A

RATED ORDER UNDER

DPAS (15 CFR 700)

13b. RATING

14. METHOD OF SOLICITATION

RFQ IFB RFP

15. DELIVER TO CODE 16. ADMINISTERED BY CODE

17a. CONTRACTOR/OFFEROR CODE FACILITY CODE 18a. PAYMENT WILL BE MADE BY CODE

TELEPHONE NO. DUNS: DUNS+4:

PHONE: FAX:

17b. CHECK IF REMITTANCE IS DIFFERENT AND PUT SUCH ADDRESS IN OFFER 18b. SUBMIT INVOICES TO ADDRESS SHOWN IN BLOCK 18a UNLESS BLOCK BELOW IS CHECKED

SEE ADDENDUM

19. 20. 21. 22. 23. 24.

ITEM NO. SCHEDULE OF SUPPLIES/SERVICES QUANTITY UNIT UNIT PRICE AMOUNT

(Use Reverse and/or Attach Additional Sheets as Necessary)

25. ACCOUNTING AND APPROPRIATION DATA 26. TOTAL AWARD AMOUNT (For Govt. Use Only)

27a. SOLICITATION INCORPORATES BY REFERENCE FAR 52.212-1, 52.212-4. FAR 52.212-3 AND 52.212-5 ARE ATTACHED. ADDENDA ARE ARE NOT ATTACHED.

27b. CONTRACT/PURCHASE ORDER INCORPORATES BY REFERENCE FAR 52.212-4. FAR 52.212-5 IS ATTACHED. ADDENDA ARE ARE NOT ATTACHED

28. CONTRACTOR IS REQUIRED TO SIGN THIS DOCUMENT AND RETURN _______________ 29. AWARD OF CONTRACT: REF. ___________________________________ OFFER COPIES TO ISSUING OFFICE. CONTRACTOR AGREES TO FURNISH AND DATED ________________________________. YOU’RE OFFER ON SOLICITATION DELIVER ALL ITEMS SET FORTH OR OTHERWISE IDENTIFIED ABOVE AND ON ANY (BLOCK 5), INCLUDING ANY ADDITIONS OR CHANGES WHICH ARE ADDITIONAL SHEETS SUBJECT TO THE TERMS AND CONDITIONS SPECIFIED SET FORTH HEREIN IS ACCEPTED AS TO ITEMS:

30a. SIGNATURE OF OFFEROR/CONTRACTOR 31a. UNITED STATES OF AMERICA (SIGNATURE OF CONTRACTING OFFICER)

30b. NAME AND TITLE OF SIGNER (TYPE OR PRINT) 30c. DATE SIGNED 31b. NAME OF CONTRACTING OFFICER (TYPE OR PRINT) 31c. DATE SIGNED

AUTHORIZED FOR LOCAL REPRODUCTION (REV. 2/2012)

PREVIOUS EDITION IS NOT USABLE Prescribed by GSA - FAR (48 CFR) 53.212

7. FOR SOLICITATION

INFORMATION CALL:

STANDARD FORM 1449

OFFEROR TO COMPLETE BLOCKS 12, 17, 23, 24, & 30

SOLICITATION/CONTRACT/ORDER FOR COMMERCIAL ITEMS 70

MCC-15-RFQ-0148 07/15/2015

Raymond Staniewski (202) 521-3656 8/18/2015 1:00 PM EST

Millennium Challenge Corporation Contracts and Grants Management Division 875 15th St, NW Suite 200 Washington, DC 20005-2227

X 0

541690

N/A

X

N/A

X

Millennium Challenge Corporation

CGM

875 Fifteenth St., NW Washington, DC 20005-2227

Millennium Challenge Corporation Contracts and Grants Management Division 875 15th St., NW, Suite 200 Washington, DC 20005-2227 Ray Staniewski, Contract Specialist (KTR) (202) 521-3656

Interior Business Center M/S D-2773 7301 West Mansfield Avenue Lakewood CO 80235-2230

MCC-15-RFQ-0148 under the BPA for DPE/M&E Independent Evaluation Services

Ex-post evaluation of the Lesotho Health Sector Project

X X

X ONE

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 2 of 70

TABLE OF CONTENTS

SUPPLIES OR SERVICES AND PRICES/COSTS

Schedule Accounting and Appropriation Data Limitation of Funds – Incrementally Funded Contracts Time and Materials No-Year Funding

DESCRIPTION / SPECIFICATIONS / STATEMENT OF WORK

Scope Background and Objectives Ex Post Evaluation Roles and Responsibilities:

Tasks

PACKAGING AND MARKING

Packaging and Marking

INSPECTION AND ACCEPTANCE

DELIVERIES OR PERFORMANCE

Deliverables Reporting Travel Requirements Deliverable Instructions Acceptance Criteria Period of Performance Place of Performance

CONTRACT ADMINISTRATION DATA

Contract Administration

SPECIAL CONTRACT REQUIREMENTS

MCC 52.232-72 - Limitation of Funds – Incrementally Funded Contracts (NOV 2006) MCC 52.232-73 Travel Reimbursement (MAY 2013) MCC 52.232-74 ODC Reimbursement (JAN 2007) MCC 52.243-70 Increase In Services (OCT 2006) KeyPersonnel Provision on Peer Review and Posting of Reports Provision on Insurance Maintenance Disclaimer for Deliverables Provision on Stakeholder Input for Evaluation Products Provision on Human Subjects Protection Personally Identifiable Information and Confidentiality Provision on Delivered Work Product Free From 3rd Party Rights Special Contract Requirements

CONTRACT CLAUSES

FAR 52.212-5 – CONTRACT TERMS AND CONDITIONS REQUIRED TO IMPLEMENT STATUTES OR EXECUTIVE ORDERS – COMMERCIAL

ITEMS (May 2015) FAR 52.217-8 Option to Extend Services (NOV 1999) FAR 52.217-9 Option to Extend the Term of the Contract (MAR 2000) FAR 52.232-18 Availability of Funds (APR 1984) FAR 52.252-2 Clauses Incorporated by Reference (FEB 1998) Contract Clauses

LIST OF ATTACHMENTS

Annex 1 – Updated program logic diagram Annex 2 – Evaluability Assessment Tool Annex 3 – Detailed evaluation questions Annex 4 – Evaluation Services Travel SOW template Annex 5 – Evaluation Design Report Outline Annex 6 – MCC Informed Consent Statement Templates (for qualitative and quantitative data collection) Annex 7 – MCC Data Documentation and Anonymization Requirements Annex 8 – Interim/Final Report Outline Annex 9 – Evaluator Progress Report and Work Plan

REPRESENTATIONS, CERTIFICATIONS AND OTHER STATEMENTS OF OFFERORS OR RESPONDENTS

Representations, Certifications and Other Statements

INSTRUCTIONS, CONDITIONS AND NOTICES TO OFFERORS OR RESPONDENTS

FAR 52.214-34 -- Submission of Offers in the English Language (Apr 1991) FAR 52.214-35 -- Submission of Offers in U.S. Currency (Apr 1991) FAR 52.216-1 Type of Contract Inquiries / Questions Concerning the Solicitation Quotation Due Date Quotation Volumes

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 3 of 70

Quotation Format PageLimits Preparation of Volume 1 – Technical Quotation Preparation of Volume II – Cost Quotation Staffing and Required Expertise Offeror Interview

EVALUATION FACTORS FOR AWARD

FAR 52.212-2 Evaluation-Commercial Items (Oct 2014)

J.3. ANNEX 3: Detailed Evaluation Questions

(End of Table of Contents)

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 4 of 70

SUPPLIES OR SERVICES AND PRICES/COSTS

Schedule

B.1.1. BPA Consultant to Conduct an Ex-Post Performance Evaluation of the Lesotho Health Sector Project.

CLIN Description of Supplies/Services Qty U/I Unit Price Amount

0001 Phase 1 Tasks (Evaluability Assessment and Evaluation Design Report) Time & Materials (T&M)

1 LOT $___________ $___________

0002 Phase 2 Tasks (Evaluation Materials/Data collection and Final report/dissemination) Time & Materials (T&M)

1 LOT $___________ $___________

0003 Other Direct Cost 1 LOT $___________ $___________

0004 Data Collection Costs 1 LOT $ 75,000.00 $ 75,000.00 Optional Task

Note: MCC anticipates out-year data collection costs for which the requirements are unknown at this time. To be fair to all vendors providing a quotation, a plug number is utilized for this CLIN only. However, if you believe this plug number is inadequate, please advise during the Question and Answer period of the RFQ. More than one data collection requirement may be required. Once requirement(s) become known, a quotation will be requested from the Contractor and a cost will be negotiated and memorialized in a modification to this call order.

Not-To-Exceed Grand Total: $

B.1.2. Labor Rates – All labor rates will be Fixed-Price for the life of this requirement.

Accounting and Appropriation Data

[RESERVED]

Limitation of Funds – Incrementally Funded Contracts

This will be an incrementally funded contract. Phase 1 Tasks (Evaluability Assessment and Evaluation Design Report) will be fully funded, with Phase 2 Tasks (Evaluation Materials/Data collection and Final report/dissemination) dependent on approval of the Evaluation Design Report and associated budget.

In accordance with clause MCC52.232-72, “Limitation of Funds – Incrementally Funded Contracts,” funds in the amount of [TBD] have been allotted to this contract. It is contemplated that funds now obligated under this contract will cover the work to be performed until [TBD].

The Contractor shall not perform work on the contract which exceeds the total amount actually allotted by the Government to the contract. The Government is not obligated to reimburse the Contractor for costs incurred in excess of the total amount allotted by the Government to this contract. The contractor assumes the risk for any increased costs beyond what the Government obligates.

Time and Materials No-Year Funding

The call order is funded with No-Year Due Diligence funding. As such, funding obligated to this call order may be used during the life of the order.

(End of SECTION B)

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 5 of 70

DESCRIPTION / SPECIFICATIONS / STATEMENT OF WORK

Scope This contract is for a consultant to provide Independent Evaluation Services for an ex-post evaluation of the Lesotho Health Sector Project. Within this Statement of Work the contractor is also referred to as the “evaluator”.

Background and Objectives

C.2.1. The MCC M&E Policy states that every Project must undergo an independent evaluation. The Lesotho Compact included a Health Sector Project that requires an ex-post evaluation.

C.2.2. An impact evaluation is defined in the MCC M&E Policy as “A study that measures the changes in income and/or other aspects of well-being that are attributable to a defined intervention. Impact evaluations require a credible and rigorously defined counterfactual, which estimates what would have happened to the beneficiaries absent the project.”

C.2.3. A performance evaluation is defined in the MCC M&E Policy as “a study that seeks to answer descriptive questions, such as: what were the objectives of a particular project or program; what the project or program has achieved; how it has been implemented; how it is perceived and valued;

whether expected results are occurring and are sustainable; and other questions that are pertinent to program design, management and operational decision making.”

C.2.4. This consultancy is expected to cover an ex-post evaluation of the Lesotho Health Sector Project, which comprised several Activities. As described in the Tasks section below, the evaluation will have two phases, with funding for Phase II dependent on design decisions made in Phase I.

C.2.5. For the purposes of the technical and financial proposal, the bidders should provide their estimate for level of effort to complete Phase I and Phase II tasks. MCC has estimated a total cost of $28,960 for travel, lodging, and per diem related expenses for a maximum of two trips over the period of performance. Actual travel costs will depend on the agreed evaluation plan and work plan.

C.2.6. Health Sector Project Background (2008-2013).

C.2.6.1. According to the 2004 Lesotho DHS, 24 percent of adults age 15-49 in Lesotho were infected with Human Immunodeficiency Virus (HIV) prior to the Lesotho Compact. The Lesotho Compact, which was implemented from 2008-2013, invested approximately $140 million in Lesotho’s health sector in order to strengthen the country’s health care system and thereby mitigate negative economic impacts of HIV/Acquired Immune Deficiency Syndrome (AIDS), poor maternal health, tuberculosis (TB) and other diseases. In particular, infrastructure funded under the Lesotho Compact was expected to provide a sustainable platform for the delivery of anti-retroviral therapy (ART) and other essential health services throughout the country; such services were expected to result in a measurable extension of productive life-years for people living with HIV/AIDS, TB and other debilitating diseases. The Compact investments were complemented by investments from other donors in the sector.

C.2.6.2. The primary activities funded under the Health Sector Project include:

C.2.6.3. Health Center (HC) Activity. HCs play a primary role in the provision of HIV/AIDS prevention, TB treatment and maternal and child health services. The HC Activity renovated, rehabilitated, furnished, and equipped 138 of the country’s 188Cs to bring the national stock

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 6 of 70 of HCs up to a common standard. The renovation of associated staff housing and improvement of water, power and communications were considered necessary to improve retention of health personnel at these facilities and improve the quality of care provided. The HCs renovated under the compact were owned by Lesotho’s Ministry of Health (MoH), the Christian Health Association of Lesotho (CHAL), the Lesotho Red Cross Society, and Maseru City Council.

Renovations coincided with the devolution of responsibility for these centers to districts and communities.

C.2.6.4. ART Clinics Activity. This Activity rehabilitated 14 out of 16 public hospital Out-Patient Departments (OPDs), provided management training to support extension of ART, and was expected to complete national coverage of ART at the hospital level. Designs for reconfiguring the OPDs were to be tailored to each hospital, and include adequate space and appropriate patient flow for incorporating ART services along with facilities for managing TB co-infection.

The Activity also provided needed power, water and telecommunications equipment, medical equipment and instruments, clinical furniture, and office equipment and furniture as required to meet MoH ART standards.

C.2.6.5. Central Laboratory Activity. A new central laboratory was considered critical to the national HIV/AIDS prevention, ART and TB programs given that the central laboratory in existence before the Lesotho Compact was too small to house the existing level of testing and had no room for expansion. This Central Laboratory Activity funded the design, construction, and equipping of a new central laboratory. It also provided for in-country and short-term external training for laboratory staff on equipment, service expansion and quality assurance associated with the new facility.

C.2.6.6. Blood Transfusion Service Activity. Safe blood for transfusions is an important element of a health system, and the demand for blood transfusions has been known to increase in countries with expanded ART services. To address the country’s need for a more formal system of blood supplies, this Activity funded construction of a dedicated central facility for collecting and processing blood, two mobile collection units, and blood storage equipment for two Government-owned and operated regional centers.

C.2.6.7. National Health Training College Activity. Prior to the Lesotho Compact, Lesotho undertook a number of initiatives to reverse the trend of deteriorating human resources. This Activity supported efforts to increase the number of graduates from nursing and allied health (e.g., pharmacy and lab technician) programs by designing, constructing, and furnishing 120 dormitory spaces and 6 apartment residences at Lesotho’s National Health Training College (NHTC). In addition to the infrastructure improvements described here, other inputs that are described in the Health Systems Strengthening Activity section below, aimed to strengthen teaching and mentoring capacities.

C.2.6.8. Health Systems Strengthening (HSS) Activity. The overall success of the Health Sector Project was believed to hinge upon improved delivery of the national essential health services package, including HIV/AIDS prevention and treatment. In the context of Lesotho, the poor quality of infrastructure was identified by the Government as a constraint to improved and expanded service delivery. In addition, the following health systems areas were considered key to service delivery and identified for Compact support in order to improve service provision at the community level.

C.2.6.8.1. Training. This Sub-activity sought to strengthen pre-service and in-service training

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 7 of 70 capacities. Prior to the Lesotho Compact, a large number of qualified applicants were rejected by the training institutions due to an insufficient number of tutors, which, in turn, resulted in too few graduates to fill MoH and CHAL vacancies. This Sub-activity aimed to develop qualified tutors, mentoring programs, and improved curricula for preparing a consistently larger cadre of nurses to provide health services at the community level. The Compact also aimed to develop an in-service training plan within the context of devolved services, and assist in implementing the most critical elements of the plan, such as health center level training of health center staff, local administrations, and communities regarding the new roles and responsibilities of each.

The HSS contractor conducted a needs assessment to identify existing needs and, thus, inform the specific training that was offered.

C.2.6.8.2. Decentralization. At the inception of the Lesotho Compact, Lesotho was in the process of devolving service delivery to the district level. A well-managed decentralization process was considered critical for health status and health services in Lesotho. And the MoH was one of the first ministries to implement the new plan. Before the Compact began, three districts had piloted decentralized health services. This was expected to expand to the remaining seven districts between 2007 and 2010. This Sub-activity aimed to support this process.

C.2.6.8.3. Research and Development Unit within MoH. Medicine is strongly evidence-based and research plays a vital role in developing interventions and assessing practices. At the inception of the Lesotho Compact, the MoH had a two person health research unit with a mandate to provide oversight to research activities in the health sector. The Lesotho Compact sought to increase this team’s capacity to coordinate research activities and to share vital lessons learned from Lesotho-specific health research.

C.2.6.8.4. Informatics. Although not specifically described in the Compact document, the Lesotho Compact established an electronic Health Management Information System at the District Health Management Teams (DHMTs) and MoH in an effort to inform policy and service delivery. An Electronic Medical Record System was also introduced as a pilot in one OPD to speed patient flow and improve patient treatment. However, rather than roll this out to all OPDs, it was ultimately implemented as a pilot at one facility.

C.2.6.8.5. Medical Waste Management Activity. In 2005, in conjunction with a World Bank health sector reform project, Lesotho conducted a comprehensive assessment of its medical waste management practices and prepared a National Health Care Waste Management Plan (WMP). However, little progress had been made in its implementation prior to the Lesotho Compact. MCC funding aimed to improve occupational health and safety and medical waste management practices, which posed significant environmental, health, and safety hazards and did not comply with MCC Environmental Guidelines prior to the Lesotho Compact. This Activity focused on updating and implementing the WMP, strengthening Government agencies that were responsible for implementing, monitoring, and enforcing the policies and standards established under the new WMP, and supporting public outreach and awareness campaigns. During the Compact period, a pilot effort was introduced at two hospitals and 15 HCs, to equip provide personal protective equipment, containers, and transport, in order to manage medical waste; the plan was to complete the rollout during the Compact, pending the completion of infrastructure.

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 8 of 70

C.2.6.8.6. Project Implementation.

The Health Sector Project’s implementation structure primarily consist of:

MCA-Lesotho. MCA-Lesotho included a full-time health sector project manager who reported to the Head of Infrastructure. This person was responsible for (i) overseeing Project implementation across all Activities; (ii) ensuring that critical milestones were met and activities progressed as planned;

and (iii) ensuring that implementation problems were resolved and alerting the Government of any constraints. This person also served as MCA-Lesotho’s liaison to the Health Sector PIU (as defined below) and to health donors’ forums.

MoH had overall responsibility for oversight of implementation of the Health Sector Project. MoH was also responsible for technical quality, timeliness of implementation, and integration of Compact interventions into MoH. MoH coordinated activities in relation to the Ministry of Local Government, and with CHAL and the Lesotho Red Cross Society. A dedicated Health Sector Project Implementation Unit (Health Sector PIU) was established within MoH to administer the Project directly.

C.2.6.8.7. Beneficiaries.

The Health Sector Project was expected to increase incomes for approximately 752,000 Basotho. This estimate was derived using 50% of health clinic catchment area populations, which was defined by the Ministry of Health using the 2004 Department of Health Survey. Benefit streams include improved health outcomes, particularly in terms of maternal and child health, HIV/AIDS, and TB, along with time savings, and ultimately improved productivity. Health personnel are also seen as benefitting from the Project as they will gain skills and have access to improved equipment to lower the risk of infection in the work place.1

C.2.6.8.8. Original independent evaluation.

MCC originally contracted evaluations for the HCs and OPDs (i.e., the infrastructure components of the Health Sector Project) in 2008 and continued those evaluations under a follow-on contract that was executed in 2011. (The 2011 contract expanded the evaluations’ focus slightly to include HSS, though the focus was much smaller than that for the infrastructure components and the evaluator acknowledged that the initial evaluation design did not account for the HSS Activity.)

The original HC evaluation design was a rigorous impact evaluation that used a randomized rollout strategy to define a control group. Experimental methods were infeasible for the OPD evaluation. However, the evaluator explored the possibility of identifying a comparison group of OPDs not being renovated under the Compact, but ultimately recommended a dose-response model-based approach instead.

1 MCC revised its approach to calculating beneficiaries of MCC investments after the Lesotho Compact was signed; this description reflects the revised approach.

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 9 of 70

Construction delays rendered the HC impact evaluation infeasible. MCC, MCA-Lesotho, and representatives from the Health PIU revisited the Project’s program logic in 2013 in an attempt to better understand and document how the program was expected to work and what outcomes could reasonably be expected from infrastructure alone (per the original evaluation designs), and thereby guide the consideration of alternatives. (See Annex 1 for an updated program logic diagram that attempts to account for all Project Activities.) The teams quickly began discussing whether and how the HC and OPD evaluations should adjust to better account for the substantial investments in non-infrastructure-related Health Sector Activities. Those discussions resulted in a decision to re-procure the Health evaluation and pursue an efficient approach that treats the Health Sector Project as a package of interventions, as appropriate.

Existing data. The following data were collected in connection with the earlier evaluation contracts or monitoring efforts and might contribute to an ex post evaluation of the Health Sector Project:

Survey Description Year Continuous Multipurpose Survey (CMS)

Initially, the health module of CMS, which is administered by the Lesotho Bureau of Statistics, was to serve as the primary household survey for the evaluation. It included questions about patterns of health care use (preventive and ambulatory care), barriers to access, perceptions of the quality of health centers infrastructure, morbidity in the last four weeks, and related symptoms or injuries. However, it was replaced by a survey that targeted MCC intervention sites rather than a nationally representative sample.

Impact Evaluation Multipurpose Survey

(IEMS)

Household survey that targeted MCC intervention sites and collected similar content as the CMS health module described above.

Exit Survey HC and OPD patients were interviewed about their perceptions of the infrastructure, wait times, and perceived quality of services.

Staff Satisfaction Survey HC and OPD staff were interviewed about working and living conditions, the perceived quality of services offered, and the physical state of the health facilities.

ART Patient Cohort Study The evaluator collected administrative data from patient records in order to measure changes in health service utilization and health outcomes like patient retention rates, disease progression and mortality in a cohort of HIV-positive adults visiting HCs and OPDs.

2010-11

Health Facility Survey Survey of HCs and OPDs that collected data on a number of variables for monitoring purposes, including physical infrastructure, human resources, equipment and supplies, operationality, service provision, and quality of care.

Ex Post Evaluation

C.3.1. The guiding questions for this ex post evaluation are:

C.3.1.1. Was the program evaluable?2

C.3.1.2. Was the program implemented according to plan?

C.3.1.3. What were program results on key outcomes?

C.3.1.4. What lessons can MCC or the Government of Lesotho apply in future programs related to program design, implementation, and sustaining results?

2 See Task 1, below, and the Evaluability Assessment Tool in Annex 2 for context on this evaluation question.

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 10 of 70

C.3.2. See Annex 3 for a detailed list of subsidiary questions to guide the evaluation.

C.3.3. Site visits. Site visits are expected to a sample of HCs, OPDs, and surrounding communities for verification of Compact outputs, assessment of the use, functionality, and maintenance of these outputs, and to investigate the adoption and continuation of behaviors and practices promoted by the Compact. In selecting the sample it will be important to select HCs and OPDs from remote areas as well as the more accessible areas.

Roles and Responsibilities:

C.4.1. The MCC, country counterparts, along with the independent evaluator, play critical roles in designing, implementing, and disseminating the independent evaluations. The MCC and country counterpart roles are as follows:

C.4.2. Millennium Challenge Corporation (MCC): MCC is responsible for oversight of the independent evaluator and quality control of evaluation activities, including the following specific responsibilities:

C.4.2.1. Guide the evaluation priorities and evaluation questions. MCC will guide the evaluation priorities and evaluation questions based on the learning priorities for the sector and agency.

C.4.2.2. Build buy-in and ownership of the evaluation. MCC will work with the independent evaluator and country counterparts to ensure there is local support for the independent evaluator.

C.4.2.3. Manage the independent evaluators. MCC will develop Terms of Reference for the independent evaluators and manage the contracting process, including Request for Proposals, Proposal Review, selection of the independent evaluator, development of a work plan, and supervision of the independent evaluator in line with the work plan and agreed deliverables.

Quality review of evaluation materials. Using the program logic as a foundation, MCC is responsible for reviewing evaluation materials, including the design report and survey materials, for feasibility, technical and factual accuracy, and adherence with international standards (when applicable).

C.4.2.4. Quality review of evaluation reports. MCC is responsible for reviewing independent evaluators’ reports to ensure technical and factual accuracy. While MCC will provide feedback, the independent evaluator is ultimately the author of the evaluation reports and may or may not take into consideration MCC’s comments. MCC therefore develops an Official Statement for each evaluation report to confirm the technical and factual accuracy of the report, or clarify any differences on technical or factual issues between MCC and the independent evaluator.

C.4.2.5. Lead initial public dissemination efforts. MCC is responsible for leading initial public dissemination of results, in particular ensuring a feedback loop internally in order to inform future decision making on similar program investment, design, and implementation. This requires ensuring the evaluation results are disseminated on the MCC website, and results are disseminated to appropriate stakeholders in a timely fashion.

C.4.2.6. Country Counterparts: During implementation, the Millennium Challenge Account (MCA) is the accountable entity responsible for building local ownership and commitment to the evaluation, oversight of the data collection firm, and quality control of evaluation activities.

After implementation ends, partner countries may also have a Designated Post-Compact M&E

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 11 of 70

Counterpart that will serve in this capacity and carry out the following specific responsibilities:

C.4.2.6.1. Build buy-in and ownership of the evaluation. The Designated Post-Compact M&E Counterpart will work with the independent evaluator and MCC to ensure the evaluation design follows the rules of program design and implementation.

C.4.2.6.2. Quality review of evaluation materials. Using the program logic as a foundation, the Designated Post-Compact M&E Counterpart is responsible for reviewing evaluation materials, including the design report and survey materials, for feasibility, technical and factual accuracy, and adherence with international standards (when applicable).

C.4.2.6.3. Quality review of evaluation reports. The Designated Post-Compact M&E Counterpart is responsible for reviewing independent evaluators’ reports to ensure technical and factual accuracy. While the Designated Post-Compact M&E Counterpart will provide feedback, the independent evaluator is ultimately the author of the evaluation reports and may or may not take into consideration the comments provided. MCC partner countries may therefore develop an Official Statement for each evaluation report to confirm the technical and factual accuracy of the report, or clarify any differences on technical or factual issues between them and the independent evaluator.

C.4.2.6.4. Facilitate public dissemination efforts. The Designated Post-Compact M&E Counterpart is responsible for facilitating public dissemination of results, in particular ensuring a feedback loop in-country in order to inform future decision making on similar program investment, design, and implementation.

Tasks

C.5.1. This will be an incrementally funded contract. Phase 1 Tasks (Evaluability Assessment and Evaluation Design Report) will be fully funded, with Phase 2 Tasks (Evaluation Materials/Data collection and Final report/dissemination) dependent on approval of the Evaluation Design Report and associated budget.

C.5.2. Independent Evaluator Roles and Responsibilities: The independent evaluator is responsible for the overall design, implementation, and dissemination of the evaluation, including the following specific responsibilities:

C.5.2.1. Develop an evaluation design given rules of implementation and feasibility of options.

Independent evaluators must assess the program design and implementation to develop the most rigorous evaluation design feasible, whether it is a performance or impact evaluation, and identify the most appropriate evaluation methodology feasible given the context. The independent evaluator is responsible for developing an evaluation design report, in consultation with MCC and the Designated Post-Compact M&E Counterpart, which summarizes the program logic, evaluation design, sampling, analysis plan, and other critical design elements of the evaluation.

C.5.2.2. Develop evaluation materials that are held to international standards. Using the program logic as a foundation, the independent evaluators define how key outcomes will be measured and develop the survey instruments or other data collection tools in order to answer critical evaluation questions. Note that “survey” here refers primarily to quantitative surveys, but may also refer to qualitative instruments where appropriate.

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 12 of 70

C.5.2.3. Ensure appropriate review of evaluation materials and research protocols. Independent evaluators are responsible for ensuring all evaluation materials, including the evaluation design, survey instruments, sampling strategy, data collection and entry protocols are reviewed by an institutional review board (IRB). IRB review and approval is required before any data collection commences. The independent evaluators will document all approvals and informed consent procedures throughout the evaluation time period.

C.5.2.4. Supervise data collection. If any primary data collection is required, the independent evaluator is responsible for quality control over data collection activities, including interviewer training, pre-testing, and implementing data quality review checks during data collection.

C.5.2.5. Lead data cleaning, analysis, interpretation of results. The independent evaluator is responsible for leading any necessary data cleaning and consultation with the data collection firm to produce final analysis files. The independent evaluator is responsible for leading all analysis in line with the agreed analysis plan developed with MCC, and consulting with MCC should the analysis plan need to be revised or adapted.

C.5.2.6. Produce evaluation reports. The independent evaluator is responsible for sharing initial evaluation reports with local stakeholders and MCC for review and feedback. However, given the independent evaluator is the author of the evaluation reports, local stakeholder and MCC feedback may or may not be taken into consideration for the final report. The independent evaluator is responsible for documenting all feedback and evaluator responses.

C.5.2.7. Lead public dissemination efforts. The independent evaluator is responsible for leading public dissemination efforts facilitated by the Designated Post-Compact M&E Counterpart and MCC (such as local workshops and conferences), but also additional conferences or other opportunities to publicly disseminate the results of the evaluation. The evaluator will advise MCC on any public dissemination opportunities and collaborate as appropriate.

C.5.3. Task 1: Evaluability Assessment

C.5.3.1. The Evaluator is responsible for conducting an evaluability assessment (using the Evaluability Assessment Tool in Annex 2) on existing program documentation, including the M&E Plan, ERRs, project description document, implementer reports.

C.5.3.2. The Evaluator will provide an Evaluability Assessment Report to MCC. The result of the Evaluability Assessment Report will determine how to proceed with the Evaluation Design Report.

C.5.4. Task 2: Develop Evaluation Design Report

C.5.4.1. Develop a Work Plan with expected deadlines for deliverables. Any revisions to the Work Plan will be submitted in the Evaluator’s Progress Reports.

C.5.4.2. The Evaluator will develop a travel SOW (see Evaluation Services Travel SOW template in Annex 4) and conduct a field missions as needed to discuss program design and implementation issues to inform the evaluation design, as well as engage local stakeholders (Designated Post-Compact M&E Counterpart, former MCA staff, and Implementing Entities, as well as local government) on the evaluation.

C.5.4.3. The Evaluator will develop the Evaluation Design Report, based on an outline agreed in

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 13 of 70 consultation with the MCC M&E lead. Please reference Annex 5 (Evaluation Design Report Outline) as an example.

C.5.4.4. As per MCC’s Gender Policy and Gender Integration Guidelines, the Evaluator will ensure that evaluation design, data collection requirements, and analysis address gender and other social differences as appropriate and feasible. This involves including research questions that address social and gender dimensions of the project logic, assumptions, and project design;

disaggregating impacts by gender, age, income, and other social difference (e.g., ethnicity) as appropriate; including special modules or studies as needed. If the evaluation includes primary data collection, the gender analysis will include, at the minimum, analysis of sex-disaggregated data for respondents. If the evaluation focuses on sector reforms, the evaluation will be required to examine the impact of a gender focus on institutional policies, practices and outcomes. Social and Gender Integration Plans or other related studies and reports completed during the compact term should be consulted for additional key topics of interest. The evaluation will disaggregate and analyze data for other key groups that may be deemed vulnerable in Lesotho, such as ethnic or religious minorities.

C.5.4.5. The Evaluator will document feedback from local stakeholders and the response to that feedback.

C.5.4.6. The Evaluator will submit the Evaluation Design Report to MCC and may be asked to present to the Evaluation Management Committee3. All feedback and response to feedback will be documented by the Evaluator. The Evaluator will continue to maintain the Evaluation Design Report as the evaluation develops. The Evaluation Design Report will remain a living document that is updated by the Evaluator as necessary and resubmitted to MCC as necessary.

C.5.5. Task 3: Obtain and/or Develop Evaluation Data

C.5.5.1. The Evaluator will develop a travel SOW (see Evaluation Services Travel SOW template in Annex 4) as per the agreed evaluation plan for any necessary travel. The Evaluator should expect that this ex-post evaluation will rely heavily on (i) existing data, and/or (ii) low-cost data collection, such as mini-surveys or focus groups that can be managed directly by the evaluator.

C.5.5.2. The Evaluator will lead development of any necessary survey instruments for data extraction from existing sources or data collection for new sources.

C.5.5.3. The Evaluator will develop survey materials in English, unless otherwise agreed with MCC.

When necessary, the Evaluator will utilize existing instruments to improve coordination across MCC independent evaluations for meta-analysis. The Evaluator is responsible for any necessary pilot testing of survey instruments and review of back-translations conducted by data collection firms prior to data collection start.

C.5.5.4. The Evaluator will document feedback from local stakeholders and response to the feedback.

C.5.5.5. The Evaluator will submit the Evaluation Materials to MCC and may be asked to present to the Evaluation Management Committee. All feedback and response to feedback will be documented by the Evaluator.

3 The Evaluation Management Committee is chaired by the MCC M&E Managing Director and is comprised of each Compact’s M&E lead, Sector lead, Economist, Country Representative, and other staff (SGA, ESP) as necessary.

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 14 of 70

C.5.5.5.1. The Evaluator is responsible for the quality of the English and local language survey instruments.

C.5.5.6. The Evaluator is responsible for developing and overseeing any data extraction and collection and entry protocols and implementation. This requires ensuring appropriate Institutional Review Board (IRB) review and approval of research protocols, survey instruments, and informed consent procedures (please reference the MCC Informed Consent Statement Templates in Annex 6).

C.5.5.6.1. In accordance with Section H, the Contractor should assess the local IRB requirements and determine whether or not additional, international-level IRB approval is required in order to ensure adherence to international standards.

C.5.5.7. The Evaluator is responsible for conducting and documenting quality control checks of data as it is collected and entered in order to reduce data collection and entry error.

C.5.6. Task 4: Develop Final Evaluation Report and (any corresponding) Data Files

C.5.6.1. Using the raw dataset, the Evaluator will produce a cleaned analysis dataset that follows MCC’s guidelines for public use data, see Annex 7 (MCC Data Documentation and Anonymization Requirements). For preparation of qualitative data files, the consultant will work with MCC to determine the most effective and appropriate method for assessing the files for public dissemination, and preparing them for dissemination as appropriate.

C.5.6.1.1. This will include STATA do files and final merged analysis files for documentation purposes.

C.5.6.2. In addition, the Evaluator will produce a full set of metadata for each survey that follows MCC’s guidelines for public use data, see Annex 7 (MCC Data Documentation and Anonymization Requirements).

C.5.6.3. The Evaluator is responsible for data analysis and developing the Final Report. The Final Report outline will be developed in consultation with the MCC M&E lead. Please see Annex 8 (Interim/Final Report Outline) as a reference.

C.5.6.4. The Evaluator will document feedback from local stakeholders and response to feedback. In order to obtain feedback, the Evaluator may lead local workshops and/or virtual reviews of Final Report by the Designated Post-Compact M&E Counterpart, former MCA staff, and Implementing Entities.

C.5.6.4.1. In accordance with Section H, the Evaluator is responsible for allowing any key local stakeholders the opportunity to submit a public Statement of Difference or Support, as agreed with MCC and the Designated Post-Compact M&E Counterpart.

C.5.6.5. The Evaluator will submit the Report to MCC and present to the Evaluation Management Committee. All feedback and response to feedback will be documented by the Evaluator. Any revisions to the report deemed necessary by the Evaluator will be incorporated into the final draft.

C.5.6.6. As per Section H, MCC may: (i) elect to obtain external peer review; and (ii) make the Report and peer reviews public on the MCC website.

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 15 of 70

C.5.6.7. The Evaluator will also be responsible for reviewing any materials developed by MCC public relations for dissemination on the MCC website for quality assurance.

C.5.6.8. The Evaluator may also be asked to participate in other MCC-financed dissemination and training events, such as MCC M&E College and the Annual MCC Impact Evaluation Workshop.

(End of SECTION C)

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 16 of 70

PACKAGING AND MARKING

Packaging and Marking Packaging and Marking shall be in accordance with the Contractor’s Blanket Purchase Agreement.

(End of SECTION D)

(Continued on Next Page)

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 17 of 70

INSPECTION AND ACCEPTANCE

[RESERVED]

(End of SECTION E)

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 18 of 70

DELIVERIES OR PERFORMANCE

Deliverables

F.1.1. The Evaluator should be aware that all final products (Design Report, Baseline Report, Survey Materials, Datasets, and Final Reports) of the evaluation that will be made publicly available on the MCC website as indicated below. Each of these final products are subject to MCC-facilitated peer review in order to assess the quality of the final product. In addition, all deliverables intended for public use must be Section 508 compliant - http://www.section508.gov.

F.1.2. As per Section H, the independent evaluator is responsible for ensuring that all public release documentation (Evaluation Design Report, Interim/Final Reports) are reviewed to minimize any risk to respondent confidentiality. This may require submission of internal use only and public release reports.

F.1.3. In addition to the required reporting detailed above, the Evaluator is responsible for the following deliverables:

Reporting

F.2.1. The Evaluator is required to submit a Progress Report (please see Annex 9 Evaluator Progress Report and Work Plan as a reference) and any draft or final deliverables with each invoice.

F.2.2. If invoices are submitted less frequently than quarterly, the Evaluator is required to submit a quarterly report with a summary of activities, status of deliverables, and any draft or final deliverables by April 1 (for January-March quarter); July 1 (for April-June quarter); October 1 (for July-September 1); and January 1 (for October-December quarter).

Phase Task Deliverables Publicly Available?

Phase

1 Evaluability Assessment

Written assessment of problem diagnostic, program logic, program participants, risks and assumptions, and metrics

TBD

2 Develop Evaluation Design Report

Work Plan with expected deliverables deadlines No Draft Evaluation Design Report No Local Stakeholder feedback with response; Confirmation of commitment to evaluation design

Yes

MCC feedback with response Yes Final Evaluation Design Report (updated as needed) Yes

Phase

3 Obtain and/or Develop Evaluation Data

Final English and translated (as required) questionnaires, training manuals

Yes

IRB package (research protocol, other documentation, approval/clearances, informed consent statement(s)) if required

No

Documentation of local stakeholder and MCC feedback and response

No

4 Develop Final Report and Disseminate Results

Draft Evaluation Report No Local Stakeholder feedback with response Yes MCC feedback with response Yes Final raw and analysis files, anonymized following MCC guidelines; STATA do files (As per Annex 4- MCC Data Documentation and Anonymization Requirements.)

Yes

Final Evaluation Report; Public Statement of Difference/Support Yes PPTs for presentations TBD http://www.section508.gov/

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 19 of 70

Travel Requirements

F.3.1. Travel will be required as part of this Call Order. Two (2) international trips to Lesotho will be required as follows, however, these may be amended based on the Contractor’s evaluation design and Program implementation plan:

Estimated

International Trips Estimated Total Days Each Trip

Estimated Total Days in Country

2 45 90

Deliverable Instructions

F.4.1. Due dates for all deliverables will be discussed with MCC and then included in the work plan and deliverable schedule.

F.4.2. The work plan and deliverable schedule will be updated as needed and agreed with MCC.

F.4.3. All technical deliverables shall be submitted to the designated Contracting Officer’s Representative (COR) or Project Monitor (PM) as specified.

F.4.4. Deliverables will be reviewed and accepted or concerns raised/comments provided within 21 calendar days of receipt.

F.4.5. The Contractor shall appropriately address any concerns raised by MCC and provide final deliverables within 14 calendar days.

F.4.6. Deliverable due dates to be confirmed with MCC prior to the start of the task.

F.4.7. Unless otherwise stated, for written deliverables, reports shall be considered “draft” upon initial receipt.

F.4.8. The outline, format (including length) and content shall be agreed upon with MCC at least 14 calendar days prior to the due date for each deliverable.

F.4.9. Unless otherwise noted, all deliverables shall be submitted in English in electronic format using WINDOWS based products including WORD for text, and data tables in EXCEL.

F.4.9.1. WORD documents shall have single-sided pages, be in Times New Roman 12-font size, single-spaced paragraphs with one-inch margins.

F.4.9.2. Digital photos files shall be submitted in JPG format.

F.4.9.3. Geographic Information System (GIS) data shall be submitted both in ArcView files, and PDF printable files.

F.4.9.4. Presentations shall be prepared with POWERPOINT.

F.4.9.5. Raw data shall be submitted in electronic formats in English and readable by standard database or statistical software, such as Excel, STATA, or SPSS; format must be agreed upon by MCC.

F.4.10. All deliverables shall be shared with relevant stakeholders.

MCC-15-RFQ-0148 LESOTHO HEALTH EVALUATION Page 20 of 70

F.4.11. All delivery dates shall coincide with US Federal Government workdays and be received by MCC by 5:00 PM, Washington DC time.

F.4.12. All deliverables shall be delivered to the assigned COR with a copy of the transmittal letter sent via e-mail to the Contracting Officer.

F.4.13. The contractor primary reporting relationships are with the Contracting Officer’s Representative (COR), and the Project Manager (PM).

F.4.14. All formal communication with MCC, including reports, shall be done through the COR and/or the

PM.

F.4.15. The COR and/or the PM will review and evaluate the contractor’s performance, and also will help coordinate work with technical counterparts.

F.4.16. Upon award of the call order, a detailed timeline and reporting schedule shall be determined between the Contractor and MCC.

Acceptance Criteria

F.5.1. The Independent Evaluator must have an approved Evaluation Design Report, updated and approved as necessary, before commencing work on data collection, analysis, and dissemination activities.

F.5.2. The decision of MCC to pursue the implementation of Evaluations will be closely tied to the ability of each activity’s evaluation to produce accountability and learning goals, specifically as they relate to convincingly answering the listed evaluation questions.

F.5.3. Work plans should include a timeline and staffing plan. The latter must describe the division of responsibilities and estimated LOE for each staff member according to deliverable.

F.5.4. When provided, the Contractor must follow MCC provided templates and guidelines for specific deliverables, such as Trip SOWs, Design Reports, and data files.

F.5.5. A detailed work plan for task completion and deliverables shall be established when the Contractor begins work.

F.5.6. The deliverables will be evaluated according to the following criteria:

F.5.6.1. Timeliness in complying with all of the elements laid…