Draft JOFOC_Zambia LWSSD Evaluation.pdf

Justification for Other Than Full and Open Competition

95332418F0336 | JOFOC

1. Requesting Activity

In accordance with Federal Acquisition Regulation (FAR) Subpart 6.303-2, the Millennium Challenge Corporation (MCC) Department of Policy and Evaluation (DPE) requests a modification to contract No. 95332418F0336 (formerly MCC-13-BPA-0040, MCC-17-CL-0013) to increase the contract ceiling and extend the period of performance.

The contracting activity is MCC’s Contracts and Grants Management (CGM) Division.

This is a Justification for Other Than Full and Open Competition.

2. Nature of the action Being Approved

The action being approved by this JOFOC will be a modification to contract no.

95332418F0336 between Mathematica Policy Research and MCC for an Independent Evaluation of the Zambia Compact’s Lusaka Water Supply Sanitation and Drainage (LWSSD) Project.

Specifically, CGM requests an overall increase of $1,357,313 to the ceiling price of the contract (from $1,198,069 to $2,555,382) and an extension of one (1) year in the period of performance (from an end date of 09/28/2022 to 09/28/2023). The ceiling increase represents addition of scope (new tasks with associated deliverables) leftover from an incomplete evaluation (with CDC through an Interagency Agreement) that was functionally terminated due to delays and other management challenges.

3. Description of the Services Required The Zambia Compact was signed in May 2012, entered into force on November 15, 2013 and closed on November 15, 2018; details about the compact can be found on MCC’s website (https://www.mcc.gov/resources/doc/star-report-zambia).

The LWSSD Project is a single sector project in water and sanitation with one project logic, which includes two interlinked activities: The Infrastructure Activity ($284 million) and the Institutional Strengthening Activity ($27 million).

The Infrastructure Activity was expected to be evaluated by the Centers for Disease Prevention and Control (CDC) through an Interagency Agreement signed in 2015 (IAA # 95332418M0053; Order # 953324F0395). In 2017, MCC awarded a contract to Mathematica Policy Research to evaluate the “the effectiveness of the LWSSD Project as a whole, with a particular emphasis on the Institutional Strengthening Activity” (Section C.2, pg. 21, contract No. 95332418F0336) due to the interlinked nature of the outcomes from the two project activities. Mathematica was expected to incorporate data inputs from the CDC evaluation to fulfill the scope requirement of evaluating the entire LWSSD project.

In 2019, the CDC and MCC agreed to end the evaluation inter-agency agreement due to a delay in the completion of the Infrastructure Activity and other challenges in the management of the Interagency Agreement. Evaluations generally require at least two rounds of data collection, one that occurs prior to the start of project implementation, and one or more after completion of the project. By 2019, CDC had completed the first round of household level surveys (baseline) but had not started the second and final round (endline) required to gather results of Infrastructure Activity. Without the endline round of Infrastructure surveys, the Mathematica evaluation team will be unable to complete the required scope or assess the results of the project as a whole. This justification describes the necessity of adding the scope leftover from CDC’s evaluation to Mathematica’s contract to ensure MCC commitment to conducting independent evaluations of its investments, specifically to measure MCC’s impact on targeted outcomes.

Modifying the existing Mathematica evaluation contract to incorporate the incomplete evaluation tasks from the prior CDC interagency agreement would avoid duplication of effort and increased costs and is in the best interest of the Government. This approach is both economical and efficient and would allow the evaluation work to continue. As a result, time, resources, tasks, and deliverables must be added to the evaluation contract because the original evaluation contract was not designed to directly measure and evaluate the Infrastructure Activity, and would not allow sufficient time to complete the evaluation, based on delays in infrastructure completion.

The Mathematica evaluation scope has been modified to include the following additional tasks to incorporate the scope to directly evaluate the Infrastructure Activity of the LWSSD, rather than relying on completed evaluation deliverables from CDC, as originally posited in Mathematica’s scope.

A. Update Evaluation Design Report – Infrastructure Activity Module (Task 8): This task is required to develop a new evaluation design module to assess the Infrastructure Activity, based on updated additional Infrastructure Activity Evaluation Questions (within section C.3 in the expanded scope). Mathematica has already prepared and submitted an Evaluation Design Report; this task would add a new module to address the expanded scope. This task would include reviewing baseline data collected by the previous evaluator and proposing an evaluation design to maximize the potential for attribution to the compact project.

B. Develop Endline Evaluation Materials – Infrastructure Surveys (Task 10):

The current scope of Mathematica’s evaluation relies expressly on the quantitative household level surveys expected to be completed by the previous evaluator. This new task includes development of the endline household survey materials and sub-contracting a local survey firm to collect the data. As part of this task Mathematica would also obtain approval for the household survey from ethical review boards in the US and Zambia. This is a new task focused specifically on the quantitative household infrastructure surveys. This task is distinct from the final qualitative data collection round already expected in the contract scope which is described in an existing task (Task 9 - Revise Endline Evaluation Materials – Qualitative component)

a. A data collection sub-contract for quantitative household

Infrastructure Surveys would require an estimated plug value of $600,000 for this task.

C. Supervise Endline data collection – Infrastructure Surveys (Task 12): This new task includes the necessary oversight related to the quantitative household infrastructure surveys. The Evaluator is responsible for developing and managing all data collection and entry protocols, supervising implementation, and ensuring data quality control. This is a new task focused specifically on the quantitative household infrastructure surveys. This task is distinct from the final qualitative data collection round already expected in the contract scope which is described in an existing task that has remained unchanged (Task 11 – Undertake Endline Data Collection – Qualitative component).

The tasks below include existing tasks in the Mathematica contract that have been modified to accommodate the expanded scope of evaluating the Infrastructure Activity of the LWSSD

D. Develop Final Report and Data Documentation Package (Task 13): The additional scope of evaluating the Infrastructure activity through quantitative data will require additional analysis and restructuring of the final report. Additionally, the quantitative survey data will need to be anonymized per the requirements of the MCC Disclosure Review Board and prepared into a data documentation package. While this requirement exists in the current task, it previously was not required because MCC does not typically expect evaluators to prepare data packages for qualitative data only. While this task remains largely unchanged in the contract, the level of effort required to complete this task will increase due to the addition of the new Infrastructure survey tasks described above (Task 8 and Task 10).

E. Disseminate Final Results (Task 14) This is an existing task that would be augmented to include the findings in response to the additional evaluation questions. As part of this task, the Evaluator would help draft the evaluation brief and present the findings from the more comprehensive endline—including the original performance evaluation of the institutional strengthening activities as well as the household survey findings—both at MCC and in Zambia.

In order to complete the additional tasks, additional funds and time beyond the allotted 4 years in the current MPR contract. The period of performance is also being extended by 12 months with the addition of Option Year 5 from October 1, 2022 to September 30, 2023. Mathematica’s contract currently ends on September 30, 2022.

To preserve consistency in the evaluation, avoid duplication of efforts, and additional cost from bringing in a new evaluator, it is advisable to increase the ceiling and extend the period of performance for the Mathematica contract.

The contract ceiling would need to be raised by a maximum of $1,357,313 from $1,198,069 to $2,555,382 and an extension of one (1) year in the period of performance (from an end date of 09/28/2022 to 09/28/2023). The increase reflects data collection costs and labor costs for endline studies and have been independently assessed by MCC through an IGCE. The endline round was originally expected to be conducted by another evaluator (the Centers for Disease Control and Prevention), who has since functionally ended the agreement with MCC.

3. Identify the statutory authority permitting other than full and open competition.

Pursuant to FAR 6.302-1, only one responsible source and no other supplies or services will satisfy agency requirements without substantial duplication of cost to the Government that is not expected to be recovered through competition and unacceptable delays in fulfilling the agency’s requirements.

4. Further describe the proposed contractor’s unique qualifications or the nature of the acquisition (i.e. emergency, statutory requirement to use one source) that meets the use of the authority cited.

There are several factors that make Mathematica the logical choice to take on the expanded scope of the evaluation of the Infrastructure Activity of the Zambia LWSSD Project in the interest of economy and efficiency. They are outlined below.

Ending of the agreement with previous evaluator (CDC) without completing work is the primary driver for this procurement action. The Infrastructure Activity of the LWSSD Project within the Zambia compact was expected to be evaluated by the Centers for Disease Prevention and Control (CDC) through an interagency agreement awarded in 2015. This Activity included the majority of the compact investment (> 65%) and included benefits to an estimated 1.2 million beneficiaries through improvements of water supply, sanitation, and drainage. The CDC evaluation design included quantitative data collection from three separate populations of beneficiaries (households expected to receive water and sanitation improvements; households along improved drainage canals to reduce flooding;

and businesses along improved drainage). CDC completed the baseline round of these three infrastructure surveys by 2018, which also coincided with the closing of the compact. At the time of compact close, major portions of the water and sanitation network improvements in key beneficiary areas were incomplete. As the evaluation design required completion of the infrastructure before the final

(endline) round of data could be collected, CDC did not complete the endline round.

The endline infrastructure surveys are required to assess the impact of the LWSSD Project. If the CDC had continued on and finished the endline work, there would be no need for this procurement action or a justification for sole source to Mathematica. The option of choosing Mathematica rather than a new contractor provides the most efficient and economical path to completing the Infrastructure endline evaluation, without additional ramp up costs of bringing a new contractor on board.

Mathematica’s scope already covers evaluation of the entire LWSSD Project through data input from the previous evaluator (CDC): In 2017, MCC awarded a contract to Mathematica Policy Research to evaluate the “the effectiveness of the LWSSD Project as a whole, with a particular emphasis on the Institutional Strengthening Activity” (Section C.2, pg. 21, contract No. 95332418F0336). The LWSSD Project has only two activities: the Infrastructure Activity and the Institutional Strengthening Activity. Mathematica was expected to incorporate data inputs from the CDC evaluation focused on the Infrastructure Activity to fulfill the Mathematica scope requirement of evaluating the entire LWSSD project. Mathematica’s evaluation design relies on the infrastructure surveys and analysis that was expected to be completed by the CDC. With the ending of the agreement between CDC and MCC; Mathematica will be unable to fulfill the expectation of assessing the entire LWSSD Project. The additional Infrastructure surveys scope now being added to Mathematica’s primary scope was already within the purview of the existing contract. Therefore, the Government will benefit from adding the additional scope to Mathematica’s contract.

The Zambia LWSSD Project is integrated with one interconnected theory of change such that evaluation scopes cannot be naturally segmented. The LWSSD Project is a single sector project in water and sanitation with one theory of change, which includes two interlinked activities: The Infrastructure Activity ($284 million) and the Institutional Strengthening Activity ($27 million). The theory of change underpins the evaluation design and identifies how different activities within a project contribute together to the same outcomes of interest.

The interlinked nature of outcomes from both the Infrastructure and Institutional Strengthening activities means that there cannot be a natural technical segmentation of the evaluation scopes. The single interconnected theory of change underpinned Mathematica’s original scope which required data inputs from the CDC evaluation tasked with assessing the Infrastructure Activity only.

Mathematica is under an existing competitively awarded contract to perform the Zambia LWSSD Project Evaluation which cannot be logically severed from evaluation planning, data collection, and analysis without incurring additional time, cost, and a risk to the project. Now that the CDC scope must be tasked to another evaluator, bringing in a brand-new contractor could result in a disjoined evaluation product and additional coordination and oversight costs to the Government, which would not be offset by competition.

There is a high probability of success of the full LWSSD Project evaluation due to Mathematica’s deep familiarity with the additional scope. Throughout the life of the Mathematica contract, the research team has gained extensive working knowledge of the project, its stakeholders, and the evaluation itself; award to any other source would result in substantial duplication of costs to the Government that is not expected to be recovered through competition, along with unacceptable delays in fulfilling the agency’s requirements in conjunction with the related Compact and evaluation timelines. Mathematica has thus far completed one round of data collection and reporting where the research team demonstrated a sound understanding of the entire LWSSD Project, including the Infrastructure Activity and the CDC scope. Due to the requirement that Mathematica incorporate data from the other evaluators into their analysis, Mathematica has reviewed, provided feedback, and engaged with CDC on technical aspects of the quantitative Infrastructure survey for the baseline round, and preparation of the CDC’s baseline report since 2018. Mathematica researchers are deeply familiar with the components of the CDC data that would need to be incorporated as part of the new tasks described above and this familiarity would save the Government time and money through minimizing mobilization costs.

Government would incur an estimated delay up to 10 months and mobilization and duplicated costs estimated up to $204,000 if this scope was awarded competitively to new contractor. In the interest of economy and efficiency, Mathematica is the best choice to take over the additional evaluation scope to minimize mobilization delays and duplication of effort; as a new contractor who would be less familiar with the scope.

The estimated timeframes and financial costs for a new firm taking over the work are outlined below and are based on MCC’s M&E team’s experience working with a variety of evaluation firms on several different contracts. The team tracks the rough costs and timelines for the standard tasks and deliverables of MCC’s evaluation contracts.

a. In order for a new evaluation firm to complete the planned exploratory analysis, they will need to spend time becoming familiar with the LWSSD Project and its stakeholders at MCC in Washington, DC and relevant stakeholders in the water and sanitation sector in Zambia. On average, this start-up phase for evaluators takes 6-8 months and includes document review, meetings, and trips to the country. The historical cost to MCC for this phase ranges from $50,000 to $100,000 for desk review and country visits alone, but it is challenging to estimate precisely because these costs are linked to the costs of other evaluation tasks such as the design of the evaluation.

b. Additionally, a new evaluation firm would have to assess the design and data collection conducted to date by both the terminated CDC evaluation

(covering the Infrastructure Activity) and the existing Mathematica evaluation (covering Institutional Strengthening Activity), and ensure that they fully integrate their new plan with both evaluation designs. This would add additional coordination burden on both the new evaluator and Mathematica and would result in new risks for the existing Mathematica evaluation, as well as the new evaluator. The cost to develop an evaluation design historically has ranged from $80,000 to $170,000 on other evaluations. It generally takes 5-8 months to develop a design report and obtain approval by the Evaluation Management Committee (EMC), without the added coordination effort required. We estimate the additional level of effort required would result in an increase in both the stated costs and timeline by about 20% (6 – 10 months; $96,000 – $204,000).

The factors described in (a) and (b) are often overlapping and not additive. A delay between 6- 10 months and an increase in cost between $96,000 and $204,000 is estimated if a new evaluator were to take over the new scope. At this particular stage of the evaluation, the additional time to design the endline approach could be accommodated, but the additional cost could be significant and it is not clear that there would be additional value produced.

5. Describe efforts made to ensure that offers were solicited from as many potential sources as practical, including whether the action was synopsized as required by FAR Subpart 5.2.

The original solicitation for this requirement was sent to seven (7) DPE/DCO Mission Support BPA holders (via email transmission) on 07/27/2017that responded Notice of Interest Request that was forwarded to all 48 BPA holders under Solicitation No. MCC- RFQ-0129 in accordance with the procedures at FAR Subpart 13.5. The requirement was unrestricted; quotations were received from four (4) vendors.

6. Describe the market research conducted (reference FAR Part 10) or state the reason market research was not conducted.

Market research was conducted by comparing labor rates to those of similar MCC evaluation contracts. Also, proposed data collection costs were vetted by comparing with prior MCC experiences conducting surveys in Zambia. As noted previously, the MCC M&E team tracks and assesses costs across the various evaluation contracts and currently manages about 98 such evaluations. As for the estimated ODCs to cover the data collection firm sub-contracts, these costs, too, are in line with the costs seen for other, similar, data collection efforts. As a form of market research, this particular action (i.e., the contract modification to increase the ceiling and extend the period of performance of this call order) is also being posted on beta.sam.

7. Cite any other facts supporting other than full and open competition for this acquisition.

The evaluation as currently designed and managed poses low risk to the government, with expectations for high value for tax-payer dollar given the expected learning. Endline data collection is the largest driver of the increased cost for these contracts. However, as the current evaluator, Mathematica is the entity best suited to conduct additional data analysis, and manage data collection for the Zambia LWSSD Project evaluation now that the previous evaluator (CDC) is no longer in agreement with MCC to complete the work.

This is because they have built great deal of knowledge of the project and local context in their 2.5 years working on this contract. They have also developed relationships with key stakeholders at MCC and in Lusaka, such that they understand the various parties’ needs and interests and can appropriately tailor their work to these needs.

In considering the justification, it is important to understand the nature of evaluation work. The design and timing of evaluations depends entirely on the design and timing of the project being evaluated. The ideal scenario for an evaluation occurs when a project design and its underlying rationale are clearly articulated upfront and adhered to throughout project implementation, and when implementation takes place on time. This allows for evaluators to design the evaluation in advance, collect baseline data before project interventions begin, and collect endline data at a pre-determined date. However, this is not the reality for most development projects, and the evaluations have to adapt to changes to the project along the way.

For example, the decision to delay the endline infrastructure survey data collection originally part of CDC’s scope was based on the delay of implementation of the project itself (construction of water and sewer infrastructure in survey population neighborhoods was delayed by about 1 year). This is necessary to accurately and comprehensively measure the project’s results. The delay by CDC in implementation was unacceptable whichresulted in a functional end to the interagency agreement; and that endline scope must now be completed by another contractor. These types of implementation timeline changes almost always have cost implications. It would be simplest for evaluators to wait until design and implementation are complete before making decisions about the evaluation approach and content of surveys, but this is impossible because baseline studies must be conducted before project implementation begins. Evaluations have to be nimble in order to rigorously measure impacts, and it is not possible for them to predict every project design or implementation change at the time of contract award.

8. List all sources, if any, who expressed in writing an interest in the acquisition.

9. State the actions, if any, the agency will take to remove or overcome any barriers to competition before any subsequent acquisition for the supplies or services required.

At this time, MCC does not anticipate any future non-competitive actions in relation to this project.

10. Certification of the Managing Director of the Program Office:

I certify that the supporting technical data, which form the rationale for a sole source acquisition, is complete and accurate to the best of my knowledge and belief.

Certified by: ___________________________________ Date: _______________

11. Contracts & Grant Division Contracting Officer Certification:

Attached is a price determination that the anticipated cost to the Government is fair and reasonable. I certify that this justification is accurate and complete to the best of my knowledge and belief.

Certified by: ____________________________________ Date: _______________

12. Managing Director, Contracts & Grants Division:

Certified by: ____________________________________ Date: ________________ (Actions exceeding $1,000,000)

13. Office of General Counsel/Competition Advocate Approval:

Approved by: ___________________________________ Date: ________________ (All actions exceeding micropurchase threshold)