MCC-17-RFQ-0073_BPA_Attachment_J.2_Mongolia_Roads_RFQ_(FINAL).pdf

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 1

Attachment J.2.

Economic Analysis and Independent Evaluation Services in Support of the Mongolia North-South Road Project

NOTE: This call order will be issued in accordance with the terms and conditions of the basic BPA RFQ, along with any additional terms and conditions referenced herein.

SECTION B – SUPPLIES OR SERVICES AND PRICE/COSTS

B.1. Price/Cost Schedule

CLIN DESCRIPTION OF SUPPLIES/SERVICES QTY U/I UNIT PRICE AMOUNT

0001 BASE PERIOD

Phase 1 – Research Design

Time and Materials (T&M)

Period of Performance: TBD

(Estimated 6 MO)

1.00 LO US$__________ US$__________

1001 OPTION PERIOD I: Data Collection

Not Separately Priced (NSP)

1001A OPTION PERIOD IA

Phase 2 – Data Collection (All Questions)

Time and Materials (T&M)

Period of Performance: TBD

(Estimated 6-18 MO, depending on data collection needs)

1.00 LO US$__________ US$__________

1001B OPTION PERIOD IB

Phase 2 – Data Collection (Core Research Questions Only)

Time and Materials (T&M)

Period of Performance: TBD

(Estimated 6-18 MO, depending on data collection needs)

1.00 LO US$__________ US$__________

2001 OPTION PERIOD II

Phase 3 – Final Evaluation Report

Not Separately Priced (NSP)

2001A OPTION PERIOD IIA

Phase 3 – Final Evaluation Report

*Note: Follows Option Period IA. Final Report consists of

Core Research AND Supplemental Research.

Time & Materials (T&M)

Period of Performance: TBD

(Estimated 6-12 MO)

1.00 LO US$__________ US$__________

2001B OPTION PERIOD IIB

Phase 3 – Final Evaluation Report

*Note: Follows Option Period IB. Final Report consists of Core Research only.

Time & Materials (T&M)

Period of Performance: TBD

(Estimated 6-12 MO)

1.00 LO US$__________ US$__________

TOTAL ESTIMATED PRICE:* US$_____________

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 2

*MCC reserves the right to set the contract ceiling above the total estimated price to account for potential deviations due to unforeseen circumstances. However, the Contractor will be expected to perform in good faith in accordance with their proposed cost estimates.

NOTES:

This RFQ describes the requirements for two different countries (Cabo Verde – Attachment 1 and Mongolia –

Attachment 2) and is expected to result in two separate BPA call orders. Offerors must submit a separate quotation for each country they choose to bid on. The PoP may be different for each country, within the ranges specified above, to accommodate country-specific requirements. The final PoP will be determined prior to contract signing based on the successful offeror’s proposed work plan. Each call order and additional option periods are subject to the availability of funds.

Offerors will need to price each option period separately. Additionally, they will need to price the final option period (OP II) twice – once for a scenario where Option IA SLIN 1001A is exercised, and once for a scenario where Option IB SLIN 1001B is exercised. Either OP I SLIN 1001A or 1001B and OP II SLIN 2001A or 2001B will be exercised, but not both.

Because the periods are based on deliverables, subsequent periods may be exercised early, as decided in the work plan (agreed between MCC and the Evaluator in Task 1).

B.2. FAR 52.216-1 -- Type of Contract (APR 1984)

The Government contemplates award of a time-and-materials (T&M) call order resulting from this solicitation.

B.3. Accounting and Appropriation Data

[RESERVED]

B.4. Funding Type

This call order will be funded, at time of award, with no-year Due Diligence funding. As such, funding obligated to this call order may be used during the life of the call order.

B.5. Limitation of Funds – Incrementally Funded Contracts

In accordance with clause MCC 52.232-72, “Limitation of Funds – Incrementally Funded Contracts,” funds in the amount of [TBD] have been allotted to this contract. It is contemplated that funds now obligated under this contract will cover the work to be performed until [TBD]. If the contractor considers the funds obligated under this contract to be insufficient to cover the work to be performed until the date specified herein, or another date agreed to by the parties, the contractor shall notify the contracting officer in writing and indicate the date on which it expects expended funds to approximate 75 percent of the total amount obligated. The notice shall state the estimated amount of additional funds required to continue performance through the date for which incremental funds are provided.

The Contractor shall not perform work on the contract which exceeds the total amount actually allotted by the

Government to the contract. The Government is not obligated to reimburse the Contractor for costs incurred in excess of the total amount allotted by the Government to this contract. The contractor assumes the risk for any increased costs beyond what the Government obligates.

B.6. Labor Rates & Other Direct Costs (ODCs)

Labor rates at the inception of a contract period will carry through as “fixed-price” without adjustment for the life of that contract period.

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 3

Hours by labor category and pricing under ODCs are estimates only and may be increased/decreased as necessary for successful completion of the contract requirements and as approved by the COR/PM. However, the contractor’s expenditures at any given point shall not exceed the total obligated amount or the total ceiling price for this contract as specified below.

Using the template below, Offerors must include all proposed labor categories, fully-burdened hourly rates and Other

Direct Costs (ODCs) for each contract period identified in B.1. Proposed labor rates must be less than or equal to the

Offeror’s proposed BPA rates.

PRICING SUMMARY PER CONTRACT PERIOD

Base

CLIN 0001

Option I

SLIN 1001A

Option I

SLIN 1001B

Option II

SLIN 2001A

Option II

SLIN 2001B

TOTAL

Total Labor Hours

Total Labor Costs $ - $ - $ - $ - $ - $ -

Total ODCs $ - $ - $ - $ - $ - $ -

Total Est. Cost $ - $ - $ - $ - $ - $ -

Base Period – Period of Performance TBD (est. 6 months)

Item # Description(2) U/I Qty Rate Total

LABOR

0001 hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

ODCs(1)

ODC Sub-Total

Total Estimated Value (Base Period CLIN 0001)

Option Period IA (All Questions) – Period of Performance TBD (est. 6 – 18 months)

LABOR

1001A hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

1001A

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 4

Total Estimated Value (Option Period IA CLIN 1001A)

Option Period IB (Core Questions Only) – Period of Performance TBD (est. 6 – 18 months)

LABOR

1001B hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

1001B

Total Estimated Value (Option Period IB CLIN 1001B)

Option Period IIA (All Questions) – Period of Performance TBD (est. 6 – 12 months)

LABOR

2001A hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

2001A

Total Estimated Value (Option Period IIA CLIN 2001A)

Option Period IIB (Core Questions Only) – Period of Performance TBD (est. 6 – 12 months)

LABOR

2001B hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

2001B

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 5

Total Estimated Value (Option Period IIB CLIN 2001B)

TOTAL COST SUMMARY (All Questions) – CLIN/SLINs 0001, 1001A, 2001A

Description(2) U/I Qty Rate Total

LABOR

hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

Total Estimated Cost

TOTAL COST SUMMARY (Core Questions Only) – CLIN/SLINs 0001, 1001B, 2001B

Description(2) U/I Qty Rate Total

LABOR

hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

Total Estimated Cost

NOTE (1): Subcontracting costs for data collection are subject to the requirements of FAR 52.244-

2, Subcontracts. Breakdown data collection costs by type of data collection and round (for example, if multiple rounds of traffic counts are proposed, price each round of separately).

NOTE (2): A job description shall be included for each labor category proposed. The job description must identify essential and non-essential tasks that are assigned to a specific position.

It must also identify reporting relationships and shall also describe required qualifications, minimum requirements, working conditions and desirable qualifications.

(End of SECTION B)

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 6

SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK

C.1. Introduction and Background

The Millennium Challenge Corporation (MCC) is a U.S. Government corporation whose mission is to provide assistance that will support economic growth and poverty reduction in carefully selected countries that demonstrate a commitment to just and democratic governance, economic freedom, and investments in their citizenry. For more information on the

MCC, please visit www.mcc.gov.

The MCC model is centered around four core principles: good governance, country ownership, a focus on results, and transparency. MCC Monitoring and Evaluation (M&E) has two objectives, stemming from the MCC core principles:

Accountability, which refers to MCC’s obligation to report on their activities and attributable outcomes, accept responsibility for them, and disclose these findings in a public and transparent manner and Learning, which refers to

MCC’s efforts to better understand the causal links between its interventions and changes in poverty and income.

As part of fulfilling these objectives, every project MCC invests in must undergo a comprehensive, independent evaluation. This Call Order will support such an evaluation for the Mongolia North-South Road Project.

As background on the Mongolia Compact, on October 23, 2007, the Government of Mongolia (GoM) and the Millennium

Challenge Corporation (MCC), a United States Government agency, signed a five-year, $285 million Compact to reduce poverty in Mongolia through economic growth. The five year Compact entered into force on September 17, 2008 and ended on September 17, 2013. The Compact was re-scoped in 2009. At this time, the $79.7 million North-South Road

Project was identified through close coordination with the Government of Mongolia and other international financial institutions in Mongolia. The project filled an important gap in the key north-south economic corridor of the country, through the construction of an important link in Mongolia’s first all-weather paved road running from its southern border with China to its northern border with Russia.

The North-South Road Project consisted of two activities: Construction of the Choir-Sainshand 176.4 kilometer road segment and Technical Assistance. The objective of the Project was more efficient transport for trade and access to services, specifically through increased traffic, decreased travel times, decreased vehicle operating costs and decreased road roughness1.

The scope of the evaluation described in this Call Order builds off the lessons learned on transport evaluations described in MCC’s “A Cross-Portfolio Analysis of Roads Rehabilitations Financed by the Millennium Challenge Corporation in

Developing Countries” (See Section J).

C.2. Scope

Provide Economic Analysis and Independent Evaluation Services in support of the Mongolia North-South Road Project.

C.3. Evaluation Questions and Guidance on Methodology

This evaluation must include an economic analysis (HDM-4/RED) and a performance evaluation (maintenance/transport sector work) of the MCC Mongolia Compact I North-South Road Project.

The economic analysis is critical to MCC’s work. MCC calculates the likely impact on economic growth of its programs by analyzing whether the proposed programs are consistent with international evidence on drivers of economic growth

1 More information on the Mongolia I Compact, and the North-South Road Project, can be found here: https://www.mcc.gov/where-we-work/program/mongolia-compact#program-docs http://www.mcc.gov/ https://www.mcc.gov/where-we-work/program/mongolia-compact%23program-docs https://www.mcc.gov/where-we-work/program/mongolia-compact%23program-docs

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 7 and by use of cost-benefit analysis. The essence of such an analysis is a straightforward comparison of costs and benefits, where the costs are the MCC-funded grants and the benefits are increases in incomes in recipient countries.2 For roads projects, these benefits are typically proxied with a consumer surplus approach by estimating the decline in vehicle operating costs and time savings for road users using HDM-4 or RED, as determined by the characteristics of the road investment.

The performance evaluation components of this requirement seek to complement the economic analysis by answering descriptive questions that provide context to and enhance the knowledge gained through the economic analysis.

Each of the research areas described below is intended to inform and provide depth and nuance to the others.

Consequently bidders should be aware that all research to be conducted should be integrated by the Evaluator, with each part informing the others in the Final Report. For instance, the political economy of road maintenance analysis should inform the maintenance assumption used in the economic analysis. The market structure analysis should inform a discussion of how the benefits are distributed across the population. And the road user and traffic growth studies should both directly inform the economic analyses as well as inform a broader discussion of the changes occurring as a result of the project investments.

It is important to note that MCC’s choice to conduct all of the research components outlined below will be informed by the capabilities of the chosen Evaluator, and MCC’s assessment of the value of the research plan developed in the

Evaluation Design Report deliverable.

This Call Order focuses on four research areas, divided further into seven research questions in total. Note that some questions are Core, and fundamental to the MCC transportation evaluation approach. Other questions are Supplemental, and will be explored at the discretion of MCC, on a country-specific basis, based on the design proposed in the Evaluation

Design Report and a careful weighing of the learning to be gained against its costs.

The distinction between the Core and Supplemental Questions is reflected in the Tasks, Phases and Option Periods. After the Evaluation Design Report, if Option Period IA is exercised, work to answer all questions will be completed. If Option

Period IB is exercised, only work to answer the Core Questions will be completed.

Work in all research areas exercised, whichever those may be, should interact with the results of each other. The goal is to inform, to the greatest extent possible, a broader discussion of the changes occurring as a result of the project investments in the Final Report. See Table 1 for a summary of all research areas, research questions, and the Core/Supplemental distinctions.

Finally, note that the final research report, before jumping into Research Question 1, must address Research Question 0:

Was the project implemented according to plan? Were there any deviations from the original design? If so, these should be documented to the greatest extent possible.

Research Area 1: Engineering Analysis and Economic Model

Research Question 1: What is the economic return – calculated in terms of vehicle operating cost (VOC) savings and travel time (TT) savings – of the road investment? [Core]

To answer Research Question 1, the Evaluator should conduct an HDM-4 or RED data collection and analysis on each road that is part of the MCC road investment. The Evaluator shall first assess the quality of pre-existing data (as available) and then collect the updated data required for modeling. Final costs are now known and prior assumptions about Post-Compact traffic counts and road conditions should be updated with actuals

2 For more information about MCC’s economic analysis methods, visit http://www.mcc.gov/pages/docs/doc/guidelines-for-economic-and-beneficiary-analysis.

http://www.mcc.gov/pages/docs/doc/guidelines-for-economic-and-beneficiary-analysis http://www.mcc.gov/pages/docs/doc/guidelines-for-economic-and-beneficiary-analysis

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 8 to get a more realistic estimate of the road’s returns. The Evaluator must review the previous models and may make changes to parameters, calibration, or assumptions if deemed necessary, but these should be discussed with MCC in advance and clearly documented in the Evaluation Design Report (EDR), described below.

Standardized collection of HDM-4/RED data is especially important. It allows MCC to compare models consistently across countries, and over time within a given road investment. See Section J – Data Collection

Quality Protocols for detailed data collection requirements and standards. The cost and value of all data collection should be considered by the Evaluator. If possible, the Evaluator should propose more cost effective methods that meet at least the standards outlined in Section J. As Mongolia I is an older MCC Compact, data collection that occurred during due diligence or during the Compact may not have used current MCC standards. Any necessary alterations from previous HDM-4/RED data collection methods on the road investment should be documented in the Evaluation Design Report, and the implication for results discussed.

Finally, the evaluation should document the factors driving changes in the ERR for the project (e.g. cost overruns, poor maintenance, inaccurate traffic projections, changes in scope or requirements during implementation, etc.), and include any recommendations for future road ERR models at MCC. This discussion should incorporate the data and analyses completed as part of the other research areas (below) MCC chooses to investigate. For example, assumptions about road maintenance (Research Area 2) should feed into this review of the ERR model.

Research Area 2: Maintenance

Research Question 2A: What are the relevant road authority’s current maintenance practices and what is the likelihood that MCC’s investment will remain adequately maintained for the life of the investment? Based on this assessment, what set of maintenance assumptions should be used in the HDM-4 model to yield the best estimate of the costs and benefits of the road investment? [Core]

The purpose of this question is to understand the sustainability of MCC’s road investment and to update assumptions about road maintenance practices into the ERR model in Research Question 1. This analysis should include a descriptive review as well as a technical assessment of the frequency and quality of the road authority’s emergency, routine and periodic maintenance practices. These should review administrative records and reports of maintenance resources and activities targeting relevant roads in the country, supplemented by actual field studies to verify or adjust the administrative evidence.

Since periodic maintenance is not generally required until five or more years after improvement and many

MCC roads were more recently improved, the study should include maintenance of otherwise comparable, but previously improved roads as well. Unlike emergency and routine maintenance, appropriate periodic maintenance evidence may be limited to that from previously improved roads.

The Evaluator shall assess the maintenance financial needs relative to the actual maintenance expenditures -and the actual condition of the roads – and compare these to assumptions made in the previous ERR models of the investment. The accuracy of MCC’s maintenance assumptions has not yet been formally tested and assessed Post-Compact. This analysis should include a discussion on how to improve and update these maintenance assumptions ex-ante for future MCC projects, and should take into account how maintenance needs might change over time (for example, traffic growth).

Research Question 2B: In cases where MCC invested in improving maintenance practices or included a maintenance

Conditions Precedent in the Compact (applicable to Mongolia), what were the effects of those efforts and why? [Core]

The goal of this question is to understand whether MCC’s investments or incentives related to road maintenance have been effective.

Research Question 2C: What political, and economic incentives are shaping road maintenance decisions in the country?

And what other key factors are influencing actual maintenance practices? [Supplemental]

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 9

This research question is meant to evaluate MCC’s assumptions about the maintenance and sustainability of improvements in infrastructure even deeper, based on a rigorous political economy analysis that incorporates appropriate and available data.3 Beyond understanding the accuracy of the ERR assumptions, the Evaluator shall also provide political economy analysis of the formal and informal decision-making that determines road maintenance of MCC’s investments and similar roads. The goal of this analysis is to understand the political and economic factors driving decisions around a country’s road maintenance, as well as any other significant constraints to a well-managed system of transportation assets, in order to better inform future road investment decisions. The Evaluator shall analyze how maintenance does or does not occur including planning, budgeting, implementation, and oversight relative to what is written or documented. The Evaluator shall assess how maintenance funding is allocated and in what priority vis-à-vis capital expenditures. This political economy analysis should be used to supplement the answers to Research Questions 2A and 2B by furthering the discussion on the sustainability of MCC’s investment, the assessment of assumptions that go into the ERR model, and the impact of MCC investments in maintenance activities or use of Conditions Precedent.

Research Area 3: Road Usage Patterns

Research Question 3A: Who is traveling on the road, why, what they are transporting, what they are paying for transport, and how long does it take to move along key routes? [Core]

This question is meant to address the next step in the program logic for any road investment: the impact on the road user. The key linkage between the road improvements and economic benefits is the reduction in VOCs and TT and how they influence the demand for transportation services. But the economic benefits of reduced transport costs must work through traffic composition and volume. Whether the expectation is for farmer, household, or trader transportation behavior to change, the change would manifest in greater or more productive usage of the road. While the HDM-4/RED models estimate VOC reductions and travel time savings based on the specifications of the road (i.e. less wear-and-tear and faster travel speeds on smoother roads) as well as generated and diverted traffic, these are estimates rather than actuals; and those models are agnostic about who experiences the benefits and why. In addition, it is not guaranteed that reductions in VOCs will result in reductions in transport prices (see Research Question 4), therefore it is not safe to assume that VOC reductions translate into reduced prices of transport services. Therefore the evaluator must propose an analytic approach to estimating traffic composition along improved roads, and road-user representative estimates of changes in travel times and prices of transportation services.

As part of Research Area 1, the Evaluator shall consider the costs and value of gathering origin-destination (O-

D) evidence through O-D vehicle intercept surveys, or other appropriate methods on relevant roads segment(s). In order to meet the needs of Research Area 3A, bidders should consider whether an intercept survey or alternate O-D methodology warrants exploratory enhancements in content, sampling, and methods.

The cost and value of gathering this expanded O-D evidence may be proposed by the Evaluator. Depending on the questions and logic motivating the investment, this expanded O-D evidence may include expanded identification of road users, the purposes of trips, their regularity/ frequency, origin/destination of the vehicles and passengers, travel times, fares for passenger and goods, cargo types, and estimated value for goods transported.

Research Question 3B: Have road usage patterns changed, in terms of who is traveling on the road, why, what they are transporting, what they are paying for transport, and how long it takes to move along key routes? [Supplemental]

This question will explore changes in the structure of transportation demands, possibly addressing whether a change may have occurred as a result of MCC road investments or other complementary investments, or due to unrelated factors. As this data was not collected ex-ante, additional retrospective quantitative or qualitative data collection may be required to assess any possible changes and to explore their likely causes.

3 A political economy analysis should explore some aspects of the way incentives shape behavior as well as the power, interests, agency, ideas, the subtleties of building and sustaining coalitions, and the role of contingency in the road sector.

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 10

The exact scoping and sampling for Question 3A and 3B will depend on the program logic for the particular

MCC transportation investment, and will be agreed upon with MCC before finalizing the evaluation design.

Research Area 4: Transportation Market Structure

Research Question 4: How is the transportation market structured and what is the likelihood that VOC savings will be passed on to consumers of transportation services? [Supplemental]

Like Question 3, this question is meant to assess the actual distribution of benefits of the MCC road investment for the road users. This should include an evaluation of the transportation market structure and the formal and informal institutions that regulate and govern the transportation market, including possible non-competitive market behavior (e.g., trucking cartels, inefficient regulatory practices, routine unauthorized highway tolls, fines, extortion, etc.). MCC would like to explore the structure and competitiveness of the transportation sector to understand how likely it is that vehicle operating cost savings that result from road improvements will be passed on to transport consumers, such as public transport users or producers transporting their inputs and products to or from markets.

The Evaluator shall work with MCC to decide what methods will best answer this question. The Evaluator should consider whether an analysis of the efficiency of commercial road-transport industries (trucking, buses, taxis, etc.) should be conducted to address the costs and purchaser pricing in the sector and how these costs and prices were actually affected by MCC financed improvements in road quality, and/or whether a political economy analysis would provide additional insight into the impact of the market structure on consumer pricing and the sustainability of the investment. This research question will also be informed by the data collected as part of the HDM-4 analysis and the expanded O-D surveys to understand whether cost savings for vehicle owners are passed on to passengers in the form of lower fares for people and goods. Separate survey-based data collection is not expected as part of this task, but data collection in the form of key informant interviews with relevant stakeholders may be warranted (such as transport or shipping companies).

Finally, note that additional research questions may be developed in discussions between MCC and the Evaluator, and agreed upon between MCC and the Evaluator as part of the Evaluation Design Report.

Table 1: Summary of Research Areas, Research Questions, and Classification

Research Area Research Question Core Supple-mental

Engineering Analysis and Economic Model

#1: What is the economic return – calculated in terms of vehicle operating cost (VOC) savings and travel time (TT) savings– of the road investment?

X

Maintenance

#2A: What are the relevant road authority’s current maintenance practices and what is the likelihood that MCC’s investment will remain adequately maintained for the life of the investment?

X

#2B: In cases where MCC invested in improving maintenance practices or included a maintenance Conditions Precedent in the

Compact, what were the effects of those efforts and why?

X

#2C: What political and economic incentives are shaping road maintenance decisions in the country?

X

Road Usage Patterns

#3A: Who is traveling on the road, why, what they are transporting, what they are paying for transport, and how long it takes to move along key routes?

X

#3B: Have road usage patterns changed, in terms of who is traveling on the road, why, what they are transporting, what they are paying for transport, and how long it takes to move along key routes?

X

Transportation #4: How is the transportation market structured and what is the X

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 11

Market Structure likelihood that VOC savings will be passed on to consumers of transportation services?

C.4. Roles and Responsibilities

A. Independence:

MCC’s Policy on Independent Evaluations. Based on the 1991 DAC Principles for Evaluation of Development

Assistance, MCC’s Evaluation Policy requires independent evaluations of each MCC project. MCC’s implementation of this independence policy adheres to these principles, especially the following: “Impartiality and independence will best be achieved by separating the evaluation function from the line management responsible for planning and managing development assistance.” (P.6.)

Organizational Conflicts of Interest. MCC-M&E is responsible for adhering to our policy-driven requirements for independence. These requirements are distinct from Organizational Conflicts of Interest (OCI) proscribed in the Federal

Acquisition Regulations. Interpretations and enforcement of the OCI statutes are the remit of our Office of General

Counsel.

Independence of Evaluation Team. Evaluation team members must be impartial and therefore should not have contributed to the design or implementation of the project being evaluated. In cases where individuals or firms that have worked on the project are proposed to be part of the evaluation team, a justification and independence risk mitigation plan must be provided to MCC for consideration.

The Independent Evaluator’s Role. To remain consistent with the principles of independent evaluations during project planning and implementation, the independent evaluator’s feedback is restricted to reporting evidence and findings.

Recommendations shall not be made before completing a final evaluation as they can influence project planning or implementation, thereby jeopardizing the independence an evaluation. Prescriptions and recommendations may be included in final evaluations.

Real-time feedback to project implementation. MCC supports monitoring and evaluation activities intended to provide feedback on how to improve program planning and implementation. But this support shall be separate from the independent evaluator. Any contribution of the Independent Evaluator to this feedback shall be limited to reporting evidence and findings. Any exceptions to this rule must be approved in advance by the EMC.

B. MCC, MCA/Post-Compact Government Responsibilities

The MCC, MCA (or in this case, as the evaluation is beginning post-Compact, the Government of Mongolia’s Post-

Compact M&E POC), along with the Independent Evaluator, play critical roles in designing, implementing, and disseminating the independent evaluations. The MCC and Government of Mongolia roles are as follows (For further detail see Section J: Evaluation Management and Review Process):

1. Millennium Challenge Corporation (MCC): MCC is responsible for oversight of the Independent Evaluator and quality control of evaluation activities, including the following specific responsibilities:

a. Assess when a program is ready for evaluation planning through evaluability assessment;

b. Determine what program components (Project(s), Activities, Sub-Activities) will be covered by the evaluation;

c. Set the evaluation questions to achieve accountability and learning objectives;

d. Build buy-in and ownership of the evaluation;

http://www.oecd.org/dac/evaluation/dcdndep/45438179.pdf http://www.oecd.org/dac/evaluation/dcdndep/45438179.pdf

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 12

e. Contract and Supervise the Independent Evaluator;

f. Conduct quality reviews of all evaluation products (reports, questionnaires, etc);

g. Facilitate public dissemination efforts to inform decision-makers on learning generated by the evaluation;

h. Oversee preparation of public-use data that ensures appropriate balance of enabling verification of analysis and broad use of the data with adherence to promises of confidentiality to survey respondents;

and

i. Determine the evaluation budgets.

2. Government of Mongolia: After the end of a Compact, the MCA (the entity responsible for implementation of the Compact) is often disbanded, but a Post-Compact government representative responsible for evaluation activities is appointed. Throughout the remainder of this document, this person will be referred to as the “Post-

Compact POC” for the Government of Mongolia. This person is responsible for building local ownership and commitment to the evaluation, oversight of the data collection firm, and quality control of evaluation activities, including the following specific responsibilities. However, it is important to note that, as several years have passed since the end of the Compact, the Post-Compact POC may or may not be very familiar with the Project

Activities.

a. Build buy-in and ownership of the evaluation;

b. Conduct quality reviews of all evaluation products (reports, questionnaires, etc.); and

c. Facilitate public dissemination efforts.

C.5. Tasks

Independent Evaluator: The Independent Evaluator is responsible for the overall design, implementation, and dissemination of the evaluation, including the following specific responsibilities:

a. Develop the most rigorous evaluation design feasible given rules of program implementation (Tasks 1-2)

b. Support MCC and the Post-Compact POC to build buy-in and ownership of evaluation. (Task 2)

c. Develop evaluation materials that are held to international standards (Task 3 or Task 5)

d. Ensure Institutional Review Board clearance of survey protocols (Task 3 or Task 5)

e. Manage the data collection firms (Post-Compact) (Task 4 or Task 6)

f. Ensure data quality during collection and entry through supervisions and management (Task 4 or Task 6)

g. Lead data cleaning, analysis, and interpretation of results (Task 7)

h. Produce high quality, credible, transparent evaluation reports (Task 7)

i. Lead preparation of public-use data that ensures appropriate balance of enabling verification of analysis and broad use of the data with adherence to promises of confidentiality to survey respondents (Task 7)

j. Lead public dissemination efforts (Task 8)

Base Period – Research Design

Task 1: Assess Evaluation Plan, ERR Model and Existing Data & Documentation

The Evaluator is responsible for conducting an Evaluability Assessment on all existing documentation related to the program and proposed evaluation plan provided by MCC using the MCC Evaluability Assessment Tool (See

Section J) as a guide. The documentation reviewed should include, but is not limited to:

o A review of project files pertaining to road investments in each country: These will be provided by MCC as available. Project files may consist of the following: signed contracts for the work, all

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 13 amendments/variation orders, and the as-built drawings including geotechnical reports and results. The

Evaluator shall also be provided with any previously collected IRI, deflection, traffic data collected at the end of the work by MCC or the country itself. The Evaluator shall notify MCC of any omissions or problems in a timely manner in order for MCC to assist the Evaluator in completing this review.

o A review of the before-project Economic Rate of Return (ERR) models used (and any subsequent models, if changes occurred) and assumptions including the appropriate documentation and justification of the data inputs that served to establish the project baseline ERR and NPV. Suggested corrections or adjustments to the existing economic model should be discussed with MCC as part of the development of the Evaluation Design in Task 2. Any agreed-upon changes must be documented in the Evaluation Design

Report.

o A review of project costs. MCC will provide the Evaluator with all available final costs including the

Design and Supervision Costs, Variation Orders and Claims Costs, Resettlement Costs, Environmental and Social costs not included in Design and Construction Costs, receiving country contributions, Project

Management and Technical Support Costs. The estimated Maintenance and Rehabilitation (M&R) costs shall be updated by the Evaluator. Where feasible, all costs should be reviewed alongside other relevant local costs.

o A review of all road maintenance and technical assistance documents provided by the country with the objective of analyzing and understanding how supplemental road maintenance activities (institutional reform, policy reform, etc.) were planned, carried out and completed during the Compact, and following the completion of works. Where IRI data is available at a network level, this should include a comparison over time to see how the overall network is doing. This can be part of the final Atlas file deliverable.

Where historical technical and financial road maintenance audit information is available, this should include a comparison before, during and after the compact, in a graphical format. This can be part of the final Atlas file.

o A review of both the availability and quality of all data pertaining to roads that have been rehabilitated under MCC’s compacts in the relevant country.

MCC and the Evaluator should identify all MCC and MCA self-evaluation activities and determine how the independent evaluation will (i) validate findings of self-evaluation(s), (ii) use self-evaluation(s) to inform the independent evaluation, particularly around understanding what was designed, what was implemented, and understanding any necessary course corrections, (iii) complement the self-evaluation(s) by expanding on self-evaluation activities.

The Evaluator will produce an Evaluability Assessment Report that provides an independent assessment of the program being evaluated. This assessment should be guided by the Evaluability Assessment Tool and should assess the quality, completeness, and/or appropriateness of the Problem Diagnostic, Program Logic, Cost-Benefit

Analysis, Risks & Assumptions, Monitoring Indicators, and Proposed Evaluation Questions. The report should aim to convey the Evaluator’s overarching assessment of the coherence of the program design, the underlying economic analysis, and the M&E approach.

o This deliverable must include a summary assessment of the cost-benefit models, existing data, and field data requirements.

o This deliverable must answer the following questions: What data (including any past HDM-4/RED models) is available vs what is completely missing? For the available data, what is the level of quality?

For both of the above, what are the implications for the evaluation?

Develop a Work Plan with expected deadlines for deliverables. Any revisions to the Work Plan will be submitted in the Evaluator’s Progress Reports. At this time, the Evaluator shall also begin the planning and logistics required to complete the assignment, including the mobilization of all equipment and personnel.

Note that the North-South Road Project was completed in September 2013, and MCC generally expects to collect

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 14 ex-post data within 5 years of completion of works. MCC will consider well-justified recommendations by the

Independent Evaluator for the time frame of data collection, but the timing should be carefully considered.

Task 2: Develop Evaluation Design Report

The Evaluator will conduct missions to the field as needed to discuss program design and implementation issues to inform the evaluation design, as well as engage local stakeholders (including the post-compact POC and any previous implementing entities that still exist, as well as local government) on the proposed evaluation. This includes hosting local stakeholder workshops to develop/refine research questions, assess appropriate and feasible quantitative and qualitative evaluation methodologies, develop the evaluation timeline, and build local stakeholder commitment to the evaluation design.

The Evaluator will develop the Evaluation Design Report, including all technical analysis required for assessing and confirming (as necessary) internal validity, power calculations, and required sample sizes. The Evaluation

Design Report outline will be developed in consultation with the MCC M&E lead. Please reference the

Evaluation Design Report Outline (See Section J) as a guide. In developing the Evaluation Design Report, the

Evaluator shall do the following:

o Determine and cite any appropriate sources of in-country data to establish current values for economic parameters in HDM-4 pertaining to vehicle fleets that are relevant to the modeling exercise. This might include cost of tires, average vehicle life by type, cost of a tune up, etc.

o The Evaluator shall perform a level 1 HDM-4 calibration specific to the country context for the modeling of roads in the select country. The calibration procedure including all documentation, references and sources in documented format shall be provided to MCC for review and approval, as a deliverable separate to the Evaluation Design Report, before use in running the economic model. This should be submitted in Excel, with explanations for how the data was collected or determined to be satisfactory.

o The Evaluator shall conduct, and summarize in the Evaluation Design Report, a comprehensive literature review of (at least): Studies that have used vehicle intercept surveys and what type of questions have been answered using this methodology; the political analysis literature on maintenance regimes to inform the discussion of what methodology they recommend to address Research Question 2C; the literature on transportation market structure including, but not limited to, literature on trucking cartels, to address

Research Question 4.

The methodology discussion in the Evaluation Design Report shall include but is not limited to: an explanation of the proposed research questions to be addressed through the evaluation as well as the proposed methods (surveys, focus groups, interviews, etc.) to address those research questions, some of which are suggested in this RFQ. The specifics of those methods (timing, location, sample sizes, etc.) shall also be explained in the Evaluation Design

Report. For example, the Evaluator shall identify how the expanded vehicle intercept survey will be designed and implemented, including a rigorous sampling strategy.

The Evaluation Design Report should include a methodology, developed by the Evaluator, for determining road maintenance needs based on the actual constructed pavement to establish a realistic case and compare this to the actual case. The methodology should be based on a justified and documented analytical engineering approach.

o The Evaluator shall review, as part of the evaluation, each country’s road maintenance and rehabilitation

(M&R) files for each road to understand how much money has been spent and is planned on being spent on routine and periodic maintenance per road financed by MCC funds.

o Since MCC road investments are only 3-5 years old, field observations and data collection in-country, on older roads comparable to the MCC investment (e.g., road roughness measures, deflection measures, rutting, cracking, and potholes) should be included, as necessary and cost-effective, to assess the road maintenance regime. While some countries may already have reliable data that can be used in this exercise, in countries that do not, the evaluator is encouraged to propose the most cost-effective

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 15 methodology for making this comparison. The Evaluator shall determine if the maintenance performed to date is appropriate and identify any potential areas of concern.

In the Final Report, the Evaluator shall need to provide a separate page showing the detailed structural analysis and calculation for remaining structural life and the traffic information including all assumptions.

The required data collection for the HDM-4 modeling is detailed in Annex J, and should be reflected in the

Evaluation Design Report. The plan for implementing this data collection should be included in the Evaluation

Design Report.

The Political Economist4 should propose a methodology to assess MCC's impact on the maintenance regime as well as future prospects (sustainability) of MCC’s investments in road maintenance. The approach should be rigorous and consider recent methodologies in order to address a variety of questions identified in close cooperation with MCC, and documented in the Evaluation Design Report.

The Evaluator shall review and summarize in the Evaluation Design Report all documents on the institutional structure of the transportation sector in Mongolia, provided by Mongolia, with the objective of analyzing the transportation market structure and the regulatory environment that governs the transportation market in country.

As per MCC’s Gender Policy and Gender Integration Guidelines, the Evaluator will ensure that evaluation design, data collection requirements, and analysis address gender and other social differences as appropriate. This involves including research questions that address social and gender dimensions of the project logic, assumptions, and project design; disaggregating impacts by gender, age, income, and other social difference (e.g. ethnicity) as appropriate; including special modules or studies as needed.

The Evaluator will include in the Design Report how they plan to review the MCC cost-benefit analysis and report on the relevant key parameters.

The Evaluator will document feedback from local stakeholders and the response to that feedback in an annex to the final report. In order to obtain feedback, the Evaluator may lead local workshops and/or virtual reviews of the

Design Report by the Post-Compact POC and Implementing Entities. Note that the draft Evaluation Design

Report must be translated into Mongolian before beginning this step.

o In accordance with Section H, the Evaluator is responsible for allowing any key local stakeholders the opportunity to submit a public Statement of Difference or Support, as agreed with MCC and the Post-

Compact POC.

The Evaluator will submit the Evaluation Design Report to MCC and may be asked to present to the Evaluation

Management Committee (See Section J). All feedback and response to feedback will be documented by the

Evaluator in an annex to the final version of the report.

o In accordance with Section H, MCC may: (i) select to obtain external peer review; and (ii) make the

Report and peer reviews public on the MCC website.

The Evaluator will produce a Nesstar Metadata file (See Section J) summarizing the final Evaluation Design

Report for dissemination on the MCC Evaluation Catalog. Prior to publication, the Evaluator should review the current version of MCC’s Evaluation Microdata Guidelines (See Section J) for guidance on preparation of public documentation that links to microdata to reduce re-identification risk.

The Evaluator will continue to maintain the Evaluation Design Report as the evaluation develops. The Evaluation

Design Report will remain a living document that is updated by the Evaluator as necessary and resubmitted to

MCC as necessary.

Alongside the elaboration of the Evaluation Design Report, the Evaluator shall also finalize the budget for the

Option Period I and Option Period II work, documenting and justifying any changes from the Evaluator’s original

4 While this political economy analysis of road maintenance should not be seen as an endorsement for any one approach or methodology, Ramalignam (2011), Booth and Golooba-Mutebi (2009), Fritz, Levy and Ort (2014), Wales and Wild (2012), Ramalingam, Laric, and Primrose (2014), Duncan and Williams (2010), Pettit (2013), and Haider and Rao (2010) provide a discussion of some of these analyses that are employed by and for other donors).

MCC-17-RFQ-0073/Attachment 2 (Mongolia Roads Evaluation) Page 16 budget, which, upon negotiation and approval by MCC, will be incorporated into the contract.

At the conclusion of Task 2, the Evaluator shall provide the Evaluation Design Report to MCC as described in

Section G below. At this point, MCC will decide whether to exercise Option Period IA or Option Period IB of the contract. If necessary, the Evaluation Design Report will be updated accordingly. The Evaluator shall not proceed to Task 3 or Task 5 until the Evaluation Management Committee approves the Evaluation Design Report.

Option Period IA – All Research Questions (Core and Supplemental Questions)

Task 3: Develop Evaluation Materials (Core and Supplemental Questions)

Before beginning work on Option Period I tasks, and upon MCC’s formal approval of the Evaluation Design

Report, MCC will provide the Evaluator with a contract modification, from the Contracting Officer, to begin work on Option Period I/Task 3. The Evaluator shall not begin on any Option Period I work until the contract modification, from the Contracting Officer, is received.

Upon receipt of the contract modification to continue, the Evaluator shall proceed according the approved work plan to prepare all materials necessary to collect all data required for answering the Core and Supplemental

Questions, as agreed in the…