95332418Q0080_-_QAs_IE_Georgia_II_STEM_Project_Final.pdf

Solicitation Number: 95332418Q0080

Independent Evaluation of the Georgia II STEM Higher Education Project

Questions and Answers

1 | P a g e

Question 1: Please confirm whether an incumbent exists for this potential requirement. If so, please provide the following information?

a) Vendor name

b) Contract number

c) Value

d) Date of award

e) Expiration date

MCC Response: No incumbent exists for this requirement.

Question 2: Please confirm the anticipated ceiling for this award. It wasn't entirely clear from the solicitation document whether there was a figure in mind given the scope of the tasks.

MCC Response: MCC will not disclose the anticipated ceiling. Offerors should propose based on the information provided in the solicitation and their professional experience with similar requirements.

Question 3: Section C.3 (Page 9) What monitoring data is being collected by MCC or SDSU with respect to the ABET accreditation criteria listed in the RFP, and which were targeted by the proposed curricula?

MCC Response: MCC is tracking the application process for each partner program for ABET. In addition, the ABET foundation did a preliminary study, contracted by MCA-G, to assess each program’s readiness to apply for accreditation, including curricula and equipment.

Question 4: Section C.4 (Page 13): The RFP references a case study methodology but does not identify the subject of the case studies to be developed. In terms of the subject of the case study approach, there are several options MCC may wish to consider, i.e.: Case studies focused on 1) individual beneficiaries, 2) the Georgian universities, 3) the program as a whole. Feedback on the focus of the case studies would be appreciated.

MCC Response: The case study should be on the SDSU-Georgia Program, as an example of standing up a university program, specifically in STEM fields.

Question 5: Section C.4 (Page 14-15) Many of the Evaluation Questions are complex in nature, containing multiple lines of inquiry and/or sub-questions. At what stage would it be appropriate for contractors to suggest alternative questions that may be more easily answered? Would MCC be interested in proposed questions at the proposal stage or would it be more appropriate to have this discussion as part of the

Evaluability Assessment?

MCC Response: MCC will work closely with the Evaluator during the Evaluability Assessment and

Evaluation Design Report phase to review the list of evaluation questions and make modifications if necessary. Bidders can propose modifications to the list in their technical quotation, but should provide a suitable explanation for the change.

Question 6: Section C.4 (Page 16) It’s mentioned that the qualitative data should be collected in regular points of time, so it seems is a longitudinal cohort study. Is this assumption correct?

MCC Response: Yes.

Questions and Answers

2 | P a g e

Question 7: On pg. 8, Section B.6, MCC states that job descriptions for each labor category proposed should “identify reporting relationships and shall also describe required qualifications, minimum requirements, working conditions, and desirable qualifications.” Can MCC clarify what information should be identified in describing “work conditions”?

MCC Response: Working conditions refers to the working environment in which an individual/staff works, which may include amenities, physical environment, stress and degree of safety or danger.

Question 8: On pg. 13, Section C.4, under “Desk Review of Existing Data Sources,” MCC lists some important sources of data and information. Will MCC provide assistance in obtaining some of the sources listed, including sources that may be held by the Ministry of Education or National Statistics Office (e.g.

GeoStat)?

MCC Response: Yes, MCC and MCA-G will facilitate access to all data sources listed in the solicitation, and other documentation that is determined necessary for the evaluation.

Question 9: On pg. 15, Section C.4, under “Evaluation Questions (Process Related),” MCC states: “What are the differences between how each set of stakeholders view the project and what determines the success of the project? Is there a consensus on what success looks like? (see…” Can MCC provide the rest of the text for the parenthetical sentence that is incomplete at the end?

MCC Response: That is a typo. The “(see…” should be deleted.

Question 10: On pg. 18, Section C.5, MCC describes Task 1 on assessing the evaluation plan. Does

MCC expect the contractor to conduct the Evaluability Assessment as a desk review exclusively or will the contractor need to conduct some in-country assessment as well?

MCC Response: It will be primarily a desk review. One trip to Tbilisi may be necessary to gather additional documentation, but no additional data collection is necessary for the evaluability assessment.

Question 11: Section C.5 (Page 20) MCC references the use of tracer studies to collect data on all graduates.

Is it envisaged the tracer study will be a one-time event, collecting data on each graduates from the SDSU

US bachelor degree programs in Georgia? If this is the case, we understand that only one graduate class will have completed the program at the end of the Compact (i.e. July 2019). Alternatively, will the tracer study data be longitudinal in nature, i.e. following specific cohorts/individuals over time? If so, over what duration would data be collected?

MCC Response: The bidders should propose the methodology of the tracer studies based on the overall framework they are proposing. MCC is first interested in answering the evaluation questions listed. The bidders should propose a strategy for tracer studies that will answer those questions.

Question 12: On page 20, under Task 3, the RFP states, “Most likely MCA-G will contract and fund the data collection firm for the baseline, and the Evaluator will contract and fund any necessary follow up data collection.” Accordingly, the template provided in section B.6 includes references to “Subcontract(s)

– Data Collection (optional)” in Option Period 1 and in Option Period 4 and a plug figure is provided for

Option Period 4 but none is provided for Option Period 1. Given that the Compact will terminate in July

2019 and Option Period 1 will likely occur after closeout, should offerors assume baseline data collection will occur in the base year or will MCC include a plug figure for data collection in Option Period 1?

Questions and Answers

3 | P a g e

MCC Response: The first round of data collection will happen during the base period and will be contracted by MCA-G using MCA funds. The reference to “Subcontract(s) – Data Collection (optional)” in Option Period 1 has been removed in the updated solicitation posted on 6/8/2018.

Question 13: Section C.5 (Page 22) Task 7 identifies monthly progress reports related to program monitoring, however, the length of the program monitoring has not been specified. What is the expected duration of Task 7?

MCC Response: The monthly progress reports referred to in Task 7 are progress reports of the evaluator’s work that should be submitted to MCC for approval of invoices. These progress reports are meant for project management purposes. The duration is the duration of the evaluation contract.

Question 14: Section C.5 (Page 22) Task 8 references data collection following the closure of the Compact in July 2019. Is data collection expected to take place immediately following closing of the Compact? If this is the case, it’s unlikely the tracer study will be able to provide much information on the outcomes of the first graduate cohort, i.e. as it relates to increased employment opportunities and higher salaries.

MCC Response: The data collection schedule will be determined by the evaluator and MCC during the evaluation design report phase. This will be based on input from SDSU, MCA-G on when impacts will be realized and the correct timing for data collection.

Question 15: Is there a budget in place to establish substitution procedures to reduce attrition and get a big enough representative sample? e.g. extra budget allocated to track students who moved from original geographical locations.

MCC Response: There is sufficient budget for this purpose in the MCA data collection budget, as well in the plug number in the evaluation contract.

Question 16: Is there a sampling framework available where we can get quick access to the beneficiaries?

MCC Response: The sampling framework is simply the student register from SDSU-Georgia and the partner universities.

Question 17: What’s the level of sampling representativeness expected from the tracer study?

MCC Response: Sampling should be done at the student level, and stratified across programs and majors.

Question 18: Page 27 of the solicitation states that:

“The overall anticipated period of performance of this contract is as follows:

- Base Period: July 2018 – July 2019

- Option Period 1: July 2019 – July 2020

- Option Period 2: July 2020 – July 2021

- Option Period 3: July 2021 – July 2022

- Option Period 4: July 2022 – July 2023

While the Contract will be awarded for the Base and Option Periods, the exercise of the option periods is at the sole discretion of MCC.” We understand that the methodology including tasks 1-11 will be developed for the Base Period July 2018 – July 2019 and that the financial offer will thus provide

Questions and Answers

4 | P a g e resources for the Base Period only. Could you please confirm that this understanding is correct and that

Option Periods (1-4) will be discussed after the award and on new basis?

MCC Response: That is not correct. The quote submission should include all five years (base period plus option periods). MCC anticipates that the tasks outlined in the SOW will be completed within five years.

As such, the RFQ allows for a base period (one year) and four option periods (one year each).

Question 19: Page 44 of the solicitation states that for key personnel (Senior Analyst Tertiary education, Senior Analyst Qualitative methods, Statistician) a “PHD or equivalent” is needed. Can you please clarify what is to be considered as “equivalent” to a PHD?

MCC Response: “PhD equivalent” is defined as 10 additional years of work experience in the tertiary education field, economics, social sciences, and/or qualitative research (i.e., additional to what is already required for the position).

Question 20: Please confirm the target cohort of the tracer study is the one enrolled in July 2014. Can you provide the expected date of graduation for this cohort? Should the tracer study try to survey students that were enrolled in July 2014 but dropped out from the program?

MCC Response: The tracer study should include students from all cohorts that graduate from the SDSU-

Georgia program, and potentially from ABET track partner university programs, in order to have a comprehensive understanding of the impacts on employment and salary of the MCC project.

Question 21: Please confirm the consultant will have access to all administrative data (e.g. students’ application process, sociodemographic data collected administratively).

MCC Response: Yes.

Question 22: Can you provide the total number of students enrolled in the program, by cohort?

MCC Response:

2015-

(1 st cohort)

2016-

(2 nd

2017-

(3 rd

2018-2019

(4 th cohort)

TOTAL

RECRUITED

STUDENTS

Total 84 126 206 230 645

Georgians 81 115 183 230 609

International 3 11 23 0 37

Male 65 70 145 115

Female 19 56 58 115 3 rd

133 (33%) th:

248 (40%)

Question 23: Please describe the selection process of participant students. Were there more applicants than accepted students? Is contact information on rejected students available?

MCC Response: The selection process is a combination of a SDSU specific early admission process, described on their website, and the Georgian process, mostly dependent on achieving a high enough score on the national exams. Information for some rejected students is possible to get, but not readily available.

Questions and Answers

5 | P a g e

Question 24: Do participant students pay tuition or are they granted scholarships? If the latter, what do the scholarships cover (e.g. partial/full tuition, stipends)? How do these benefits compare to what students in other programs in the same universities have to pay/receive?

MCC Response: It’s a mix of self-pay and scholarships.

Question 25: One of the research questions in p 14 says “Do graduates of MCC-supported US Bachelor’s program have better levels of income than Georgian students graduating from the non-US STEM

Bachelor’s programs?” Is there an expected magnitude of income increase the evaluator should consider?

MCC Response: The wage premium assumed in the CBA, based on a survey of businesses undertaken by

Hans Gutbrod during compact development, was 44%.

Question 26: One of the research questions in p 14 says “Does the project help alleviate a skill mismatch in the Georgian labor market?” Is there any documentation showing this is the case?

MCC Response: Those are internal documents that can only be shared after the contract is signed and

NDAs are submitted. However, the World Bank Systematic Country Diagnostic for 2018 provides useful background documentation and similar findings to the MCC due diligence studies.

http://documents.worldbank.org/curated/en/496731525097717444/Georgia-Systematic-Country-

Diagnostic-from-reformer-to-performer

Question 27: Page 16 says “The tracer studies should cover all graduates in any cohort of students that is assessed”. What is the graduation rate of these programs?

MCC Response: The first cohort will graduate in June 2019.

Question 28: Page 20 says “It will not be feasible to collect clean baseline data for the Georgia II Higher

Education Evaluation however”. Can you say a few words why this is the case, or what does baseline mean in this context?

MCC Response: This sentence refers to the fact that it is not possible to survey SDSU students on certain outcomes before they were exposed to the SDSU-Georgia program. As such, the first round of data collection will not be pre-intervention. A true baseline would be before they enrolled in the program.

Question 29: Page 20 says “it is critical that the Evaluator have the contact and basic demographic information from all students who will be part of the tracer study sample population.” Will this be provided by MCC or how is the consultant supposed to get this data?

MCC Response: MCC and MCA-G will provide this data.

Question 30: Baseline report: Please clarify what does baseline means in this context. When students were accepted to the program? When the first cohort graduated? If the former, can you say a few words about the quantities data the evaluator will have at its disposal in terms of the characteristics of the students (e.g. age, gender, results of application to the program). If the latter, can you confirm the evaluator is supposed to survey these students around time they are graduating?

MCC Response: The first cohort enrolled in 2015 and will graduate in 2019. SDSU has all demographic data on students. The evaluator should propose a data collection time that fits the evaluation questions listed in the RFQ.

http://documents.worldbank.org/curated/en/496731525097717444/Georgia-Systematic-Country-Diagnostic-from-reformer-to-performer http://documents.worldbank.org/curated/en/496731525097717444/Georgia-Systematic-Country-Diagnostic-from-reformer-to-performer

Questions and Answers

6 | P a g e

Question 31: Endline data: Please confirm after how many years of graduation the endline data is supposed to be collected, and if we should focus on the cohort that started in June 2014 or consider other cohorts as well.

MCC Response: All cohorts should be considered.

Question 32: Please confirm only one round of data collection after graduation should be collected.

MCC Response: This is up to the evaluator to propose based on their overall framework, ensuring they are answering the evaluation questions.

Question 33: Have SDSU and the participating Georgian Universities already agreed to share student data with an external evaluator or will obtaining this agreement be part of the evaluation activities? If the latter, in what ways will MCC assist with this task?

MCC Response: Yes, MCC and MCA-G will assist the evaluator in gathering all of the necessary administrative data from implementing partners.

Question 34: Does MCC expect the evaluation to establish causality and thereby require experimental or quasi-experimental methods?

MCC Response: No. MCC is not expecting the evaluator to implement an experimental or quasi-experimental.

Question 35: Can MCC provide a cost range for the submission?

MCC Response: See Question 2 above.

Question 36: On pg. 71, Section L.10.5, MCC notes that the Cost Quotation should include a copy of the Offeror’s Negotiated Indirect Cost Rate Agreement, if applicable. Can MCC confirm that contractors will be allowed to charge indirect costs on “materials” (ODCs and subcontracts) if their NICRA includes rates for those categories of costs?

MCC Response: Yes. If an Offeror’s NICRA includes rates for ODCs and/or subcontracts, they may charge indirect costs in accordance with their NICRA.

Question 37: Is there a target range or maximum budget that is allocated for the evaluation?

MCC Response: See Question 2 above.

Question 38: Can you share a copy of the original project design and partnership?

MCC Response: This is an MCC internal document that can only be shared with the Evaluator once the contract and NDAs are signed.

Question 39: Was there preliminary work done to determine the relevance of the targeted disciplines to

Georgian industry and economy? Can you share that work?

MCC Response: Those are internal documents that can only be shared after the contract is signed and NDAs are submitted. However, the World Bank Systematic Country Diagnostic for 2018 provides useful

Questions and Answers

7 | P a g e background documentation and similar findings to the MCC due diligence studies.

http://documents.worldbank.org/curated/en/496731525097717444/Georgia-Systematic-Country- Diagnostic-from-reformer-to-performer

Question 40: One of the outcome questions asks: To what extent do any changes observed in awareness seem directly linked to this project? Can you elaborate on what is meant by “awareness”?

MCC Response: This reference is to awareness of the SDSU program in Georgia.

Question 41: What type of student level data do the participating colleges collect?

MCC Response: The university programs collect student demographic data, admission test scores, and limited financial data.

http://documents.worldbank.org/curated/en/496731525097717444/Georgia-Systematic-Country-Diagnostic-from-reformer-to-performer http://documents.worldbank.org/curated/en/496731525097717444/Georgia-Systematic-Country-Diagnostic-from-reformer-to-performer