95332418Q0080_-_Georgia_II_STEM_Evaluation.pdf

SOLICITATION/CONTRACT/ORDER FOR COMMERCIAL ITEMS

OFFEROR TO COMPLETE BLOCKS 12, 17, 23, 24, & 30

1. REQUISITION NUMBER PAGE 1 OF

2. CONTRACT NO. 3. AWARD/EFFECTIVE

DATE

4. ORDER NUMBER 5. SOLICITATION NUMBER 6. SOLICITATION ISSUE

DATE

7. FOR SOLICITATION

INFORMATION CALL:

a. NAME b. TELEPHONE NUMBER (No collect calls)

8. OFFER DUE DATE/

LOCAL TIME

9. ISSUED BY

13b. RATING

14. METHOD OF SOLICITATION

CODE

15. DELIVER TO 16. ADMINISTERED BY CODE

18a. PAYMENT WILL BE MADE BY CODE 17a. CONTRACTOR/

OFFEROR

CODE

FACILITY

CODE

CODE

TELEPHONE NO.

17b. CHECK IF REMITTANCE IS DIFFERENT AND PUT SUCH ADDRESS IN

OFFER

18b. SUBMIT INVOICES TO ADDRESS SHOWN IN BLOCK 18a UNLESS BLOCK

BELOW IS CHECKED

RFQ IFB RFP

SEE ADDENDUM

19.

ITEM NO.

20.

SCHEDULE OF SUPPLIES/SERVICES

21.

QUANTITY

22.

UNIT

23.

UNIT PRICE

24.

AMOUNT

(Use Reverse and/or Attach Additional Sheets as Necessary)

25. ACCOUNTING AND APPROPRIATION DATA 26. TOTAL AWARD AMOUNT (For Govt. Use Only)

28. CONTRACTOR IS REQUIRED TO SIGN THIS DOCUMENT AND RETURN

DELIVER ALL ITEMS SET FORTH OR OTHERWISE IDENTIFIED ABOVE AND ON ANY

ADDITIONAL SHEETS SUBJECT TO THE TERMS AND CONDITIONS SPECIFIED

29. AWARD OF CONTRACT: REF.

DATED . YOUR OFFER ON SOLICITATION

(BLOCK 5), INCLUDING ANY ADDITIONS OR CHANGES WHICH ARE

SET FORTH HEREIN, IS ACCEPTED AS TO ITEMS:

30a. SIGNATURE OF OFFEROR/CONTRACTOR

30b. NAME AND TITLE OF SIGNER (Type or print) 30c. DATE SIGNED

31a. UNITED STATES OF AMERICA (SIGNATURE OF CONTRACTING OFFICER)

31b. NAME OF CONTRACTING OFFICER (Type or print)

COPIES TO ISSUING OFFICE. CONTRACTOR AGREES TO FURNISH AND

OFFER

13a. THIS CONTRACT IS A

RATED ORDER UNDER

DPAS (15 CFR 700)

SEE SCHEDULE

12. DISCOUNT TERMS

ARE ARE NOT ATTACHED

ARE ARE NOT ATTACHED

27a. SOLICITATION INCORPORATES BY REFERENCE FAR 52.212-1, 52.212-4. FAR 52.212-3 AND 52.212-5 ARE ATTACHED. ADDENDA

27b. CONTRACT/PURCHASE ORDER INCORPORATES BY REFERENCE FAR 52.212-4. FAR 52.212-5 IS ATTACHED. ADDENDA

11. DELIVERY FOR FOB DESTINA-

TION UNLESS BLOCK IS

MARKED

10. THIS ACQUISITION IS UNRESTRICTED OR SET ASIDE: % FOR:

SMALL BUSINESS

HUBZONE SMALL

BUSINESS

SERVICE-DISABLED

VETERAN-OWNED

SMALL BUSINESS

WOMEN-OWNED SMALL

BUSINESS (WOSB)

ECONOMICALLY

DISADVANTAGED

WOMEN-OWNED SMALL

BUSINESS (EDWOSB)

8 (A)

NAICS:

SIZE STANDARD:

STANDARD FORM 1449 (REV. 3/2005) BACK

19.

ITEM NO.

20.

SCHEDULE OF SUPPLIES/SERVICES

21.

QUANTITY

22.

UNIT

23.

UNIT PRICE

24.

AMOUNT

32a. QUANTITY IN COLUMN 21 HAS BEEN

RECEIVED INSPECTED ACCEPTED, AND CONFORMS TO THE CONTRACT, EXCEPT AS NOTED:

41a. I CERTIFY THIS ACCOUNT IS CORRECT AND PROPER FOR PAYMENT

32b. SIGNATURE OF AUTHORIZED GOVERNMENT

REPRESENTATIVE

32c. DATE

41b. SIGNATURE AND TITLE OF CERTIFYING OFFICER 41c. DATE

42a. RECEIVED BY (Print)

42b. RECEIVED AT (Location)

42c. DATE REC'D (YY/MM/DD) 42d. TOTAL CONTAINERS

40. PAID BY

32d. PRINTED NAME AND TITLE OF AUTHORIZED GOVERNMENT

REPRESENTATIVE

32e. MAILING ADDRESS OF AUTHORIZED GOVERNMENT REPRESENTATIVE 32f. TELPHONE NUMBER OF AUTHORZED GOVERNMENT REPRESENTATIVE

32g. E-MAIL OF AUTHORIZED GOVERNMENT REPRESENTATIVE

33. SHIP NUMBER 34. VOUCHER NUMBER 35. AMOUNT VERIFIED

CORRECT FOR

PARTIAL FINAL

37. CHECK NUMBER

38. S/R ACCOUNT NO. 39. S/R VOUCHER NUMBER

36. PAYMENT

COMPLETE PARTIAL FINAL

95332418Q0080

TABLE OF CONTENTS:

Listing of Incorporated Purchase Requisitions Section B - Supplies or Services and Prices/Costs Section C - Description/Specifications/Work Statement Section D - Packaging and Marking Section E - Inspection and Acceptance Terms Section F - Delivery or Performance Section G - Contract Administration Data Section H - Special Contract Requirements Section I - Contract Clauses Section J - List of Documents, Exhibits, and other Attachments Section K - Representations, Certifications, and Other Statements of Offerors Section L - Instructions, Conditions and Notices to Offerors Section M - Evaluation Factors for Award

Listing of Incorporated Purchase Requisitions

Incorporated Purchase Requisition Numbers:

DPE-PR-18-0039

Section B - Supplies or Services and Prices/Costs

Item Number Base Item Number

Supplies/Services Quantity Unit

0001 Base Period - Independent Evaluation of the Georgia II STEM

Higher Education Project

Contract Type:Time and Materials Unit Price

Other Direct Costs Extended Price

Description:

Purchase Requisitions DPE-PR-18-0039

IDC Type: Not Applicable

Item Number Base Item Number

Supplies/Services Quantity Unit

1001 Option Period I - Independent Evaluation of the Georgia II STEM

Higher Education Project

Contract Type:Time and Materials Unit Price

Other Direct Costs Extended Price

Description:

Purchase Requisitions

IDC Type: Not Applicable

Item Number Base Item Number

Supplies/Services Quantity Unit

2001 Option Period II - Independent Evaluation of the Georgia II STEM

Higher Education Project

Contract Type:Time and Materials Unit Price

Other Direct Costs Extended Price

Description:

Purchase Requisitions

IDC Type: Not Applicable

Item Number Base Item Number

Supplies/Services Quantity Unit

3001 Option Period III - Independent Evaluation of the Georgia II STEM

Higher Education Project

Contract Type:Time and Materials Unit Price

Other Direct Costs Extended Price

Description:

Purchase Requisitions

IDC Type: Not Applicable

Item Number Base Item Number

Supplies/Services Quantity Unit

4001 Option Period IV - Independent Evaluation of the Georgia II STEM

Higher Education Project

Contract Type:Time and Materials Unit Price

Other Direct Costs Extended Price

Description:

Purchase Requisitions

IDC Type: Not Applicable

Clauses incorporated by reference

None

Clauses incorporated by full text

Section B - SUPPLIES OR SERVICES AND PRICE/COSTS

B.2 FAR 52.216-1 -- Type of Contract (APR 1984)

The Government contemplates award of a Time-and-Materials (T&M) contract resulting from this solicitation.

(End of Provision)

B.3 Accounting and Appropriation Data

[RESERVED]

B.4 Funding Type

This contract will be funded, at time of award, with no-year Due Diligence funding. As such, funding obligated to this contract may be used during the life of the contract.

B.5 Limitation of Funds – Incrementally Funded Contracts

In accordance with clause MCC 52.232-72, "Limitation of Funds – Incrementally Funded Contracts," funds in the amount of [TBD] have been allotted to this contract. It is contemplated that funds now obligated under this contract will cover the work to be performed until [TBD]. If the contractor considers the funds obligated under this contract to be insufficient to cover the work to be performed until the date specified herein, or another date agreed to by the parties, the contractor shall notify the contracting officer in writing and indicate the date on which it expects expended funds to approximate 75 percent of the total amount obligated. The notice shall state the estimated amount of additional funds required to continue performance through the date for which incremental funds are provided.

The Contractor shall not perform work on the contract which exceeds the total amount actually allotted by the Government to the contract. The Government is not obligated to reimburse the Contractor for costs incurred in excess of the total amount allotted by the Government to this contract. The contractor assumes the risk for any increased costs beyond what the Government obligates.

B.6 Labor Rates & Other Direct Costs (ODCs)

Labor rates at the inception of a contract period will carry through as "fixed-price" without adjustment for the life of that contract period.

Hours by labor category and pricing under ODCs are estimates only and may be increased/decreased as necessary for successful completion of the contract requirements and as approved by the COR/PM. However, the contractor's expenditures at any given point shall not exceed the total obligated amount or the total ceiling price for this contract as specified below.

Using the template below, Offerors must include all proposed labor categories, fully-burdened hourly rates and Other Direct Costs (ODCs) for each contract period identified in the CLIN structure.

PRICING SUMMARY PER CONTRACT PERIOD

Base Option 1 Option 2 Option 3 Option 4 TOTAL

Total Labor Hours

Total Labor Costs

Total ODCs $ - $ - $ - $ - $ - $ -

Total Est. Cost $ - $ - $ - $ - $ - $ -

Base Period – Period of Performance TBD (est. 12 months)

Item # Description(2) U/I Qty Rate Total

LABOR

0001 hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

ODCs(1)

ODC Sub-Total

Total Estimated Value (Base Period)

Option Period I – Period of Performance TBD (est. 12 months)

Item # Description(2) U/I Qty Rate Total

LABOR

1001 hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

ODCs(1)

ODC Sub-Total

Total Estimated Value (Option Period I)

Option Period II – Period of Performance TBD (est. 12 months)

Item # Description(2) U/I Qty Rate Total

LABOR

2001 hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

ODCs(1)

ODC Sub-Total

Total Estimated Value (Option Period II)

Option Period III – Period of Performance TBD (est. 12 months)

Item # Description(2) U/I Qty Rate Total

LABOR

3001 hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

ODCs(1)

ODC Sub-Total

Total Estimated Value (Option Period III)

Option Period IV – Period of Performance TBD (est. 12 months)

Item # Description(2) U/I Qty Rate Total

LABOR

4001 hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

ODCs(1)

4001 Subcontract(s) – Data Collection (

Endline)

Lot 1 $150,000.00 $150,000.00

ODC Sub-Total

Total Estimated Value (Option Period IV)

TOTAL COST SUMMARY

Description(2) U/I Qty Rate Total

LABOR

hr.

hr.

hr.

hr.

Total Labor Hours - -

Labor Sub-Total

ODCs(1)

Subcontract(s) – Data Collection (Endline)

Lot 1 $150,000.00 $150,000.00

ODC Sub-Total

Total Estimated Cost

NOTE (1):MCC anticipates approximately $150,000.00 worth of endline data collection subcontracting costs for which the requirements are unknown at this time. To be fair to all vendors providing a quotation, a plug number is utilized for data collection subcontracting costs in the respective year(s) in which data collection is expected to occur. However, if you believe this plug number is inadequate, please advise during the Question and Answer period of the solicitation. "Data Collection Subcontract(s)," as used in the tables above, refers only to the cost of subcontracting a data collection firm and any associated indirect costs (e.g., subcontract handling fees). It does not include any prime contractor labor associated with overseeing/managing the data collection effort; this time should be incorporated into the level of effort proposed under the respective direct labor CLIN. Subcontracting costs for data collection are subject to the requirements of FAR 52.244-2, Subcontracts.

NOTE (2): A job description shall be included for each labor category proposed. The job description must identify essential and non-essential tasks that are assigned to a specific position. It must also identify reporting relationships and shall also describe required qualifications, minimum requirements, working conditions and desirable qualifications.

(End of SECTION B)

Section C - Description/Specifications/Work Statement

Clauses incorporated by reference

None

Clauses incorporated by full text

Section C

SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK

C.1Introduction

The Millennium Challenge Corporation (MCC) is a U.S. Government corporation whose mission is to provide assistance that will support economic growth and poverty reduction in carefully selected countries that demonstrate a commitment to just and democratic governance, economic freedom, and investments in their citizenry. For more information on the MCC, please visit www.mcc.gov

C.2 Scope

Undertake an independent evaluation of the Georgia II STEM Higher Education Project to assess the progress toward the original objectives and outcomes stated in the program logic. TheContractor shall provide support to MCC for Independent Evaluation Services in support of the Georgia II STEM Higher Education Project objectives of the $30M STEM Higher Education Project are:

1. To bring a U.S. University to Georgia to partner with Georgian public universities to offer U.S. Bachelor's degree programs in the STEM academic disciplines.

2. To provide capacity enhancement for Georgian Public Universities with the goal of Georgian University programs reaching international standards and acquiring international program accreditation

C.3 Background

To achieve the above objectives, MCA-Georgia signed a 15-month "pre-enrollment" agreement under which San Diego State University (SDSU) undertook the necessary actions to enroll students starting July 2014, followed by a 45-month collaborative agreement to complete the remainder of the project activities through July 2019. Per this agreement, SDSU:

1) administer and offer academic programs that are professionally (Accreditation Board for Engineering and Technology (ABET), Accrediting Commission for Schools, Western Association of Schools and Colleges (ACS WASC), and regionally (WASC) "ABET" is the Accreditation Board for Engineering and Technology; "ACS" is the American Chemical Society, and "WASC" is the Accrediting Commission for Schools, Western Association of Schools and Colleges (ACS WASC), one of the six regional accrediting agencies in the United States and the one that accredits SDSU.accredited and internationally recognized;

2) assist partner universities to achieve internationally-recognized accreditation for target degrees;

3) develop curricula and train Georgian faculty;

4) develop facilities that facilitate the SDSU programs and;

5) develop partnerships with industry.SDSU is responsible for design, development, and delivery of the academic programs, as well as, required infrastructure improvements.

SDSU is in the process of implementing US and Georgian bachelor's dual degree programs in six disciplines (chemistry, computer science, computer engineering, electrical engineering, civil engineering, construction engineering). Students who graduate from these university programs will receive two diplomas, from both San Diego State University and from the partner university indicated below:

Partner University U.S. Degree Programs

Tbilisi State University Chemistry (Biochemistry focus)

Computer Engineering

Electrical Engineering

Computer Science

Ilia State University Computer Engineering

Electrical Engineering

Georgian Technical University Chemistry

Computer Engineering

Electrical Engineering

Civil Engineering

Construction Engineering

Project Outputs and Outcomes

In the short term the STEM Higher Education Project is expected to provide the following outputs:

§ Upgraded infrastructure and equipment

§ Faculty development

§ Improved curricula

§ Inclusive outreach program

§ Introduction of professors, curricula and frameworks

§ Distance learning programs for Georgian students.

In the long run the STEM Higher Education Project is expected to increase the earnings of students who graduate from US degree programs and from internationally accredited programs in Georgia, and improve their employment opportunities. The project is also expected to reduce imports of education (study abroad) and reduce imports of human capital (expatriate scientists and engineers).

Project Logic

A visual description of the logic underlying all Compact projects, including the STEM Higher Education Project, is included in Figures 1 and 2 as follows:

Figure 1. Compact-wide Program Logic (1 of 2)

Figure 2. Compact-wide Program Logic (2 of 2)

Accreditation of Georgian Programs

A critical component of the STEM Higher Education Project, and core element of the collaborative agreement with SDSU is ABET accreditation of the engineering and technology programs at partner universities, and ACS accreditation for chemistry programs. This relates directly to the second objective of the project, and is central to the sustainability of MCC's investment.

ABET accreditation requires consideration of the programs according to several criteria, which have been specifically targeted by the proposed curricula, including:

1. Students – the qualifications of the students that are accepted and the monitoring of their performance against graduation requirements.

2. Program Educational Objectives – the mission of the program and its consistency with the institution's mission, and the achievement of these objectives. This criterion requires a working relationship with industry and an Advisory Board in order to establish objectives and to assess the degree to which graduates achieve them in practice after graduation.

3. Student Outcomes – student outcomes that prepare graduates to attain the program educational objectives must be established, and performance must be assessed against them. This assessment is made via metrics that are established with both direct and indirect assessments. Culminating experiences are also incorporated in the assessment strategy.

4. Continuous Improvement –regular use of appropriate, documented processes for assessing and evaluating attainment of student outcomes.

5. Curriculum – program curriculum is defined to achieve student outcomes, and the details of this connection must be established and maintained.

6. Faculty – the, qualifications, size and the composition of the faculty to meet the needs of the curriculum.

7. Facilities – the physical resources (classrooms, labs, offices) available to support the needs of the program.

8. Institutional Support – financial resources to allow faculty development and support services of the Departments providing these programs.

The MCC Compact-funded Collaborative Agreement between SDSU – Georgia and MCA-Georgia describes a path to accreditation for programs at Georgian partner universities in which the SDSU curriculum is transposed to these institutions.

As the faculty at the Georgian partner universities build competence in teaching the SDSU, ABET-accredited curriculum, e.g.

they would adapt SDSU's educational tools and techniques in the classroom, develop accreditation self-studies, and submit those for accreditation as Georgian students graduate from their programs. This is referred to as the first "track" for ABET accreditation.

An alternative model that is considered a second "track" towards ABET accreditation, involves SDSU-Georgia supporting partner universities in obtaining ABET accreditation for a few of their existing Georgian language engineering programs, in which they already have a number of graduates working in industry.

Fundraising and Partnerships

Although it was not explicitly incorporated into the original project design, partnership promotion and fundraising have become critical achieving the project's two main objectives stated above. To date, MCC's $31M investment in the SDSU-Georgia program has been leveraged to attract nearly $3M scholarships for Georgian students from a dozen different donors.

SDSU-Georgia has also recently hired a returned Georgia Peace Corps volunteer to assist with recruitment, fundraising, student-life activities, and English language development.

In addition to the private sector engagement, MCA-G and MCC have working closely with the Government of Georgia (GoG) to ensure additional financial support for the project. GoG is also contributing $1.1M in scholarship support to the first three SDSU classes, has committed to providing just over $11M to support cohorts 4 and 5, and is prominently featuring SDSU in the Ministry of Education's "Study in Georgia" campaign to attract more international students to pursue higher education degrees in Georgia.

Student Loans

MCC and MCA-G are also currently exploring the possibility of a student loan program with the European Development Bank (ERBD). The structure and size of the program has not yet been determined.

Further Contributions from GoG

Finally, re-flows from a Georgia Compact I investment fund, totaling over $13M, are financing student scholarships and capital investment.

C.4 Evaluation Questions and Guidance on Methodology

The Evaluator should propose a case study methodology that assesses the assumptions in the project logic, provides a rich analysis of project implementation, and distills out lessons that can be valuable to the Georgian government in its objectives to improve STEM higher education quality, to MCC in future investments in tertiary education partnerships, to U.S. higher education institutions and associations, and to the international donor community more broadly. The evaluation should be outcome/impact oriented and focus on uncovering concrete evidence on what worked, challenges, and opportunities for improvement.

The Evaluator will, in dialogue with the Millennium Challenge Account-Georgia (MCA-G) and MCC, and in liaison with various stakeholders and partners, document the project development, process, successes, and lessons learned focusing on key outcomes of interest laid out in the project logic. The final evaluation report should highlight the following:

• The context of the STEM sector in Georgia;

• The roles of different organizations and stakeholders involved in the project;

• The role of MCC as the donor;

• Actors (internal and external) involved in the processes, interventions;

• Process, approach, and outcome(s). This should include any other factors, both internal and external, as well as those that may have been outside the control of MCC and MCA-g that may have contributed to the outcomes;

• The significance of relevant policy drivers/framework, which might exist and influence the project and/or outcomes;

• Outcomes related to the success of SDSU graduates in the labor market

Evaluation Questions (Outcome Related)

• Was the original program logic sound? The original program logic was based on the assumption that private investment in Georgia is constrained by a lack of qualified scientists and engineers. Were/are Georgian companies unable to employ expatriate scientists and engineers (for example from former Soviet countries, Eastern Europe, Turkey or South Asia) due to legal impediments such as work visa requirements? Or were/are expatriate scientists and engineers viewed as too costly by Georgian employers, implying that the constraint was an insufficient supply of inexpensive well trained scientists and engineers rather than an insufficient supply of well-trained scientists and engineers?

• Was the project successful in achieving the objectives articulated in the Compact, in the winning proposal from SDSU, and in the M&E plan program logic? (see logic visual below)

• What employment opportunities do graduates of MCC-supported US Bachelor's program have and how do those opportunities differ from Georgian students graduating from the non-US STEM Bachelor's programs?

• Do graduates of MCC-supported US Bachelor's program have better levels of income than Georgian students graduating from the non-US STEM Bachelor's programs?

• Does the project contribute towards reduction of number of Georgian students seeking education abroad?

• How does the project contribute to the reduction of foreign professional workers (and increase in Georgian nationals) hired in the STEM areas?

• We can assume that accreditation generally equals a high quality degree program, but to what extent are Georgian' s aware of the significance of the accreditation/to what extent is that a draw for the program?

• To what extent has SDSU achieved name recognition and a positive reputation in Georgia? How much of a role might this play in influencing future sustainability of the program?

• Does the project help alleviate a skill mismatch in the Georgian labor market? Does it produce skills which are in high demand of the Georgian economy?

• To what extent do any changes observed in awareness seem directly linked to this project?

• What lessons are applicable to other Higher Education policies and interventions in Georgia?

Evaluation Questions (Process Related)

• How has the Higher Ed/STEM landscape in Georgia changed since this project was conceived? Are there more/fewer donors working in the sector? How have GoG commitment and vision changed since the project was originally conceived? How have SDSU's vision changed since the project has was originally conceived, both in San Diego and in Tbilisi? What about general Georgian awareness of the STEM field and opportunities both in education and employment?

• How should tuition rates be set in order to make high quality STEM higher education programs (U.S. and/or ABET-, ACS-accredited) sustainable in Georgia given (i) family willingness and ability to pay, (ii) availability of Georgian government funding to support high quality STEM higher education, (iii) other financial resources that might be available to sustain high quality STEM higher education? What should MCC take into account given the opportunity to design a project such as this one again?

• Have all the stakeholders in this project been identified and incorporated?

• What are the differences between how each set of stakeholders view the project and what determines the success of the project? Is there a consensus on what success looks like? (see

• Given that the pool of stakeholders is large and varied in their interests, what lessons are there for establishing the incentives so that future projects such as this one can be successful?

The following principles should underline the overall approach and narrative based on the STEM Higher Education Project Design:

• Value of existing knowledge (e.g., of institutions, mechanisms, processes, etc.);

• Sustainability;

• Scaling up potential for replication and/or adaptation;

• Documentation of existing STEM initiatives in tertiary education.

The narrative should draw from the following:

Desk Review of Existing Data Sources

Over the past seven years, several consultancies have produced a body of work that will be critical to this evaluation. The evaluation should include a document review, including review of the following documents and should seek to build off the existing information and data sources, fill in any critical gaps, and produce a comprehensive final evaluation report. These resources include, but are not limited to:

• All MCC investment risk committee documents and all MCC investment management committee documents produced during compact development that relate to the Georgia II compact.

• All concept papers submitted by the Government of Georgia during compact development.

• 2011 Constraints Analysis

• Original SDSU financial and technical proposals;

• SDSU 20-year MoU with the GoG and amendments;

• SDSU 15-month, 12-month, and 9-month agreements with MCA-Georgia;

• All deliverables produced by SDSU under these agreements and MCC/MCA comments on deliverables as applicable; GRDF agreement and amendment; GRDF drawing requests;

• SDSU- Georgian partner university agreements (with TSU, ISU, GTU);

• MCC M&E Plan section on STEM Higher Education Project;

• SDSU public-private partnership MoU;

• ABET and WASC accreditation assessments;

• Dalberg analysis of SDSU financial sustainability (2016);

• Final SDSU proposal to Cartu Foundation for scholarship support;

• SDSU student satisfaction survey results (2017)

• Marts and Lundy Report on SDSU-Georgia fundraising (2017)

• SDSU Georgia strategic growth plan (2017)

• SDSU Sustainability Plan/report (2017)

• ABET Foundation studies (2017)

Key Informant Interviews and Focus Groups

An initial round of focus groups may be necessary to the process related evaluation questions, such as determining the definition of success, and how different stakeholders view success.

US-based:

• Katerina Ntep, Jonathan Brooks, Peter Rosner, Sarah Olmstead, Rodney Chow/Chris Ackerman, Nora Dudwick, Marcel Ricou, Jenner Edelman, Sonia Shahrigian, Ryan Moore, Van Crowder, Jim McNicholas, Earl Kellogg, Joe Dougherty/Dalberg (current/former MCC staff or consultants);

• Stanley Maloy, Nancy Marlin, Alan Sweedler/Joe Shapiro, Bill Tong, Chukuka Enwemeka, Tom McCarron/Agnes Wong-Dickerson, Michele Goetz (SDSU)

• Peter McPherson (APLU), Joe Dougherty/Ian Warthin/Virginia Woolworth (Dalberg), Kenneth Walker (Emory University nursing program), Peter Moock (former HCD education consultant)

Georgia-based:

• Magda Magradze, Nodar Surguladze, Elene Jibladze, Giorgi Kopaleishvili, Giorgi Kaladze, Keti Maminaishvili (MCA-Georgia and current/former Ministry of Education)

• Ken Walsh, Halil Guven, Elene Aladashvili, Nino Chubinidze, Steve Armington, Nino Chinchaladze (SDSU and partners)

• Rector Gia Sharveshidze, Vice Rector Misha Chkhenkeli, Head of Scientific Research and Development Giorgi Gvedashvili, ABET Committee members (TSU)

• Rector Giga Zedania, ABET Committee Chair (ISU)

• Rector Archil Prangishvili, Head of International Relations Otar Zumburidze, ABET Committee members (GTU)

• SDSU-Georgia donors

• SDSU-Georgia Advisory members (~5 members/not full Board)

• Georgia Ministry of Education stakeholders: Deputy Minister Murgulia, Tamar Sanikidze/head of EQE and former Minister of Education, likely others

Site visits

• A selection of SDSU classroom and lab spaces at each partner university (both renovated, not renovated with Compact funds)

• SDSU-Georgia administrative offices

Tracer Studies

• Because increased employment opportunities and higher salaries are key outcomes of the project, the evaluation should involve tracer studies of graduates from (i) the SDSU US bachelor degree programs in Georgia, (ii) comparable degree programs at Georgian universities that are not internationally accredited, and (iii) comparable internationally accredited degree programs at Georgian public universities if available.

• The tracer studies should cover all graduates in any cohort of students that is assessed, and should collect at least the following information: (i) current employment status, (ii) current field of employment, (iii) current employer,

(iv) job position or title, (v) annual salary or monthly wage.

C.4. Roles and Responsibilities

A. Independence:

MCC's Policy on Independent Evaluations. Based on the 1991 DAC Principles for Evaluation of Development Assistance, MCC's Evaluation Policy requires independent evaluations of each MCC project. MCC's implementation of this independence policy adheres to these principles, especially the following: "Impartiality and independence will best be achieved by separating the evaluation function from the line management responsible for planning and managing development assistance." (P.6.)

Organizational Conflicts of Interest. MCC-M&E is responsible for adhering to our policy-driven requirements for independence.

These requirements are distinct from Organizational Conflicts of Interest (OCI) proscribed in the Federal Acquisition Regulations. Interpretations and enforcement of the OCI statutes are the remit of our Office of General Counsel.

Independence of Evaluation Team. Evaluation team members must be impartial and therefore should not have contributed to the design or implementation of the project being evaluated. In cases where individuals or firms that have worked on the project are proposed to be part of the evaluation team, a justification and independence risk mitigation plan must be provided to MCC for consideration.

The Independent Evaluator's Role. To remain consistent with the principles of independent evaluations during project planning and implementation, the independent evaluator's feedback is restricted to reporting evidence and findings. Recommendations shall not be made before completing a final evaluation as they can influence project planning or implementation, thereby jeopardizing the independence an evaluation. Prescriptions and recommendations may be included in final evaluations.

Real-time feedback to project implementation. MCC supports monitoring and evaluation activities intended to provide http://www.oecd.org/dac/evaluation/dcdndep/45438179.pdf feedback on how to improve program planning and implementation. But this support shall be separate from the independent evaluator. Any contribution of the Independent Evaluator to this feedback shall be limited to reporting evidence and findings.

Any exceptions to this rule must be approved in advance by the Evaluation Management Committee (EMC).

B. MCC & MCA Responsibilities: The MCC, MCA, along with the Independent Evaluator, play critical roles in designing, implementing, and disseminating the independent evaluations. The MCC and MCA roles are as follows (For further detail see Section J: Evaluation Management and Review Process):

1. Millennium Challenge Corporation (MCC): MCC is responsible for oversight of the Independent Evaluator and quality control of evaluation activities, including the following specific responsibilities:

a. Assess when a program is ready for evaluation planning through evaluability assessment;

b. Determine what program components (Project(s), Activities, Sub-Activities) will be covered by the evaluation;

c. Set the evaluation questions to achieve accountability and learning objectives;

d. Build buy-in and ownership of the evaluation;

e. Contract and Supervise the Independent Evaluator;

f. Conduct quality reviews of all evaluation products (reports, questionnaires, etc.);

g. Facilitate public dissemination efforts to inform decision-makers on learning generated by the evaluation;

h. Oversee preparation of public-use data that ensures appropriate balance of enabling verification of analysis and broad use of the data with adherence to promises of confidentiality to survey respondents;

and

i. Determine the evaluation budgets.

2. Millennium Challenge Account (MCA)-Georgia and country partners: MCA-G and country partners are responsible for building local ownership and commitment to the evaluation, oversight of the data collection firm, and quality control of evaluation activities, including the following specific responsibilities:

a. Assess when a program is ready for evaluation planning through evaluability assessment;

b. Set the evaluation questions to achieve accountability and learning objectives;

c. Build buy-in and ownership of the evaluation;

d. Manage the data collection firms;

e. Conduct quality reviews of all evaluation products (reports, questionnaires, etc); and

f. Facilitate public dissemination efforts.

C.5. Tasks

Independent Evaluator: The Independent Evaluator is responsible for the overall design, implementation, and dissemination of the evaluation, including the following specific responsibilities:

a. Develop the most rigorous evaluation design feasible given rules of program implementation (Tasks 1-2)

b. Support MCC and MCA to build buy-in and ownership of evaluation. (Task 2 and Task 7)

c. Develop evaluation materials that are held to international standards (Task 3 and Task 8)

d. Ensure Institutional Review Board clearance of survey protocols (Task 4 and Task 9)

e. Manage the data collection firms (Post-Compact) (Task 4 and Task 9)

f. Ensure data quality during collection and entry through supervisions and management (Task 4 and Task

9)

g. Lead data cleaning, analysis, and interpretation of results (Task 5 and Task 10)

h. Produce high quality, credible, transparent evaluation reports (Task 5 and Task 10)

i. Lead preparation of public-use data that ensures appropriate balance of enabling verification of analysis and broad use of the data with adherence to promises of confidentiality to survey respondents (Task 10)

j. Lead public dissemination efforts (Task 11)

Task 1: Assess Evaluation Plan

• The Evaluator is responsible for conducting an Evaluability Assessment on all existing documentation related to the program and proposed evaluation plan provided by MCC and MCA using the MCC Evaluability Assessment Tool (See Section J) as a guide.

• MCC and the Evaluator should identify all MCC and MCA self-evaluation activities and determine how the independent evaluation will (i) validate findings of self-evaluation(s), (ii) use self-evaluation(s) to inform the independent evaluation, particularly around understanding what was designed, what was implemented, and understanding any necessary course corrections, (iii) complement the self-evaluation(s) by expanding on self-evaluation activities.

• The Evaluator will produce an Evaluability Assessment Report that provides an independent assessment of the program being evaluated. This assessment should be guided by the Evaluability Assessment Tool and should assess the quality, completeness, and/or appropriateness of the Problem Diagnostic, Program Logic, Cost-Benefit Analysis, Risks & Assumptions, Monitoring Indicators, and Proposed Evaluation Questions. The report should aim to convey the Evaluator's overarching assessment of the coherence of the program design, the underlying economic analysis, and the M&E approach.

• Develop a Work Plan with expected deadlines for deliverables. Any revisions to the Work Plan will be submitted in the Evaluator's Progress Reports.

Task 2: Develop Evaluation Design Report

• The Evaluator will conduct missions to the field as needed to discuss program design and implementation issues to inform the evaluation design, as well as engage local stakeholders (MCA and Implementing Entities, as well as local government) on the proposed evaluation. This includes hosting local stakeholder workshops to develop/refine research questions, assess appropriate and feasible quantitative and qualitative evaluation methodologies, develop evaluation timeline, and build local stakeholder commitment to the evaluation design.

• The Evaluator will develop the Evaluation Design Report, including all technical analysis required for assessing and confirming (as necessary) internal validity, power calculations, and required sample sizes. The Evaluation Design Report outline will be developed in consultation with the MCC M&E lead. Please reference the Evaluation Design Report Outline (See Section J) as a guide.

• As per MCC's Gender Policy and Gender Integration Guidelines, the Evaluator will ensure that evaluation design, data collection requirements, and analysis address gender and other social differences as appropriate. This involves including research questions that address social and gender dimensions of the project logic, assumptions, and project design; disaggregating impacts by gender, age, income, and other social difference (e.g. ethnicity) as appropriate; including special modules or studies as needed.

• The Evaluator will estimate the project's post-compact economic rate of return (ERR):

o In the Evaluation Design Report, the Evaluator will specify which key input, output and outcome parameters modeled in the MCC CBA will be measured as part of the proposed evaluation design. If the Evaluator believes it is infeasible or prohibitively expensive to measure a benefit stream in the MCC CBA, this should be made explicit and a justification provided for MCC to consider.

o The Evaluator should propose a CBA methodology to estimate the project's post-compact ERR. The methodology may be based on an existing MCC model or developed by the Evaluator. It should be documented in the EDR, citing sources where relevant. The Evaluator may recommend to MCC that it is not feasible or worthwhile to build a CBA model of the project. A justification for this recommendation should be provided in the EDR for MCC's consideration.

• The Evaluator will document feedback from local stakeholders and the response to that feedback in an annex to the final report. In order to obtain feedback, the Evaluator may lead local workshops and/or virtual reviews of the Design Report by MCA and Implementing Entities.

o In accordance with Section H, the Evaluator is responsible for allowing any key local stakeholders the opportunity to submit a public Statement of Difference or Support, as agreed with MCC and the MCA/designated representative.

• The Evaluator will submit the Evaluation Design Report to MCC and may be asked to present to the Evaluation Management Committee (See Section J). All feedback and response to feedback will be documented by the Evaluator in an annex to the final report.

o In accordance with Section H, MCC may: (i) select to obtain external peer review; and (ii) make the

Report and peer reviews public on the MCC website.

• The Evaluator will produce a Nesstar Metadata file (See Section J) summarizing the final Evaluation Design

Report for dissemination on the MCC Evaluation Catalog. Prior to publication, the Evaluator should review the current version of MCC's Evaluation Microdata Guidelines (See Section J) for guidance on preparation of public documentation that links to microdata to reduce re-identification risk.

• The Evaluator will continue to maintain the Evaluation Design Report as the evaluation develops. The Evaluation Design Report will remain a living document that is updated by the Evaluator as necessary and resubmitted to MCC as necessary.

Task 3: Develop Evaluation Materials

• The Evaluator will develop or provide technical support on development of data collection firm TORs, depending on the funding arrangements for data collection. Most likely MCA-G will contract and fund the data collection firm for the baseline, and the Evaluator will contract and fund any necessary follow up data collection. Note that the data collection methods, including the use of large scale surveys that will produce micro data should directly relate to the proposed evaluation methodology. In the proposal, the Evaluator should explain the rationale for using the selected types of data collection, and the associated cost. Tasks 3-5 can be interpreted to fit both qualitative and quantitative data.

• The Evaluator will lead development of any necessary survey instruments and interview protocols for primary data collection (including questionnaires and accompanying enumerator manuals) or a data extraction approach for existing sources.

• The Evaluator will develop survey materials in English, unless otherwise agreed with MCC. When necessary, the Evaluator will utilize existing instruments to improve coordination across MCC independent evaluations for meta-analysis. The Evaluator is responsible for pretesting Note that MCC differentiates between the "pre-test", defined as the testing of the instrument for content and flow, versus the "enumerator pilot", an exercise completed during training for enumerators to practice with the instrument, and for the survey team to practice data flow, but NOT to result in major changes to the instrument. survey instruments during their development and reviewing translations and back-translations conducted by data collection firms prior to data collection start (as necessary).

• The Evaluator is responsible for ensuring any surveys last no longer than 120 minutes without written consent from the Evaluation Management Committee.

• The Evaluator will document feedback from local stakeholders and response to the feedback. In order to obtain feedback, the Evaluator may lead local workshops and/or virtual reviews of the Evaluation Materials by MCA and Implementing Entities.

• The Evaluator will submit the Evaluation Materials to MCC and may be asked to present to the Evaluation Management Committee. All feedback and response to feedback will be documented by the Evaluator.

o The Evaluator is responsible for the quality of the English and local language survey instruments.

Task 4: Prepare and Supervise baseline data collection

• It will not be feasible to collect clean baseline data for the Georgia II Higher Education Evaluation however, it is critical that the Evaluator have the contact and basic demographic information from all students who will be part of the tracer study sample population. In addition, qualitative data collection should be conducted at regular intervals so as to capture any changes in important outcomes.

• The Evaluator is responsible for developing and overseeing all data collection and entry protocols and implementation. This requires reviewing the current version of MCC's Evaluation Microdata Guidelines (See Section J) and ensuring appropriate Institutional Review Board (IRB) review and approval of research protocols, survey instruments, and informed consent procedures (see MCC Informed Consent Statement Template (See Section J)).

o In accordance with Section H, the Evaluator should assess the local IRB requirements and determine whether or not additional, international-level IRB approval is required in order to ensure adherence to international standards.

§ If there are local requirements, the Evaluator must abide by them.

§ If the local IRB is HHS registered, no additional IRB is required.

§ If the local IRB is not HHS registered, the Evaluator also needs to sub-contract an HHS registered IRB. HHS registered IRBs are found here -https://ohrp.cit.nih.gov/search/search.aspx?styp=bsc

• The Evaluator is responsible for overseeing the training of data collection staff, field work, and data entry. The Evaluator is responsible for pilot testing the survey instruments and field procedures as a part of enumerator training. This must be managed in coordination with MCC and MCA. The Evaluator is responsible for reporting on the average survey time resulting from the pilot test and from field work.

• The Evaluator is responsible for conducting and documenting quality control checks of data as it is collected and entered in order to reduce data collection and entry error.

• The Evaluator is responsible for establishing a system for frequent communication with the data collection firm throughout data collection and entry, through email, phone, and site visits.

Task 5: Develop Baseline Report and Data Documentation Package

• The Evaluator will review raw files from the survey firm and provide any critical feedback on production of the survey firm's Data Collection Report.

• Before conducting baseline analysis, the Evaluator will review the current version of MCC's Evaluation Microdata

Guidelines (See Section J) for guidelines on management, documentation, and dissemination of the microdata.

• The Evaluator is responsible for developing the Baseline Report. The Baseline Report outline will be developed in consultation with the MCC M&E lead. Please reference the Baseline Report Outline as a guide (See Section J).

• The Evaluator will ensure baseline analysis including gender and social analysis is part of the evaluation design as described in the evaluation design report (task 2), including examining differences in baseline characteristics by gender, age, income and other social variables as appropriate.

• The Evaluator will document feedback from local stakeholders and response to feedback in an annex to the final report. In order to obtain feedback, the Evaluator may lead local workshops and/or virtual reviews of Baseline Report by MCA and Implementing Entities.

o In accordance with Section H, the Evaluator is responsible for allowing any key local stakeholders the opportunity to submit a public Statement of Difference or Support, as agreed with MCC and the MCA/designated representative.

• The Evaluator will submit the Baseline Report to MCC and present to the Evaluation Management Committee. All feedback and response to feedback will be documented by the Evaluator in an annex to the final report. Any revisions to the report deemed necessary by the Evaluator will be incorporated into the final draft.

o In accordance with Section H, MCC may: (i) select to obtain external peer review; and (ii) make the

Report and peer reviews public on the MCC website

Task 6: Disseminate Baseline Package

• Once the baseline report is reviewed by the Evaluation Management Committee, the Evaluator is responsible for:

o Presentation at MCC headquarters o Presentation at MCA headquarters

• The Evaluator should review the current version of MCC's Evaluation Microdata Guidelines (See Section J) for guidance on preparation of public documentation that links to microdata. All baseline materials (questionnaire(s), informed consents(s), research protocol(s), updated Metadata (as needed), internal and/or public-use Baseline Report, and Data Package Worksheet (See Section J)) should be submitted as a deliverable package.

• The Evaluator will update the Nesstar Metadata as appropriate for the MCC Evaluation Catalog.

• The Evaluator may also be responsible for reviewing any materials developed by MCC public relations for dissemination on the MCC website for quality assurance.

• The Evaluator may be asked to participate in other MCC-financed dissemination and training events, such as

MCC M&E College and MCC Evaluation Summits.

Task 7: Monitor program implementation and conduct risk assessment

• The Evaluator is responsible for continuous monitoring of program implementation to ensure implementation occurs according to design and any deviations from design are properly documented. This requires collaboration with MCC, MCA sector leads and implementing entities to ensure continuous support for and commitment to the evaluation design. This may require collaboration with MCC/MCA to set up a monitoring or information-sharing system. Risks and implementation updates should be reported regularly by the Independent Evaluator as part of the Monthly Progress Report.

• For impact evaluations, this requires status updates of implementation according to the treatment (to ensure the intervention is implemented in the agreed treatment areas) and control (to ensure the intervention is not implemented in the agreed control areas and to monitor any other threats to internal validity such as contamination by other programs). The Evaluator is responsible for flagging any serious concerns or risks regarding the internal validity of the evaluation throughout this period.

Task 8: Revise Endline Evaluation Materials

• The Evaluator will develop the data collection firm TORs and sub-contract for any necessary data collection.

• Evaluator is responsible for any necessary updates and revisions to the Endline Evaluation Materials, including questionnaires and enumerator training materials, based on lessons learned. Note that the data collection methods, including the use of large scale surveys that will produce micro data should directly relate to the proposed evaluation methodology. In the proposal, the Evaluator should explain the rationale for using the selected types of data collection, and the associated cost. Tasks 8-10 can be interpreted to fit both qualitative and quantitative data.

• The Evaluator is responsible for ensuring surveys last no longer than…