MCC-17-RFQ-0005_Final.docx

PAGE 1 OF

1. REQUISITION NO.

2. CONTRACT NO.

3. AWARD/EFFECTIVE DATE

4. ORDER NO.

5. SOLICITATION NUMBER

6. SOLICITATION ISSUE DATE

a. NAME

b. TELEPHONE NO. (No Collect Calls)

8. OFFER DUE DATE/LOCAL

TIME

9. ISSUED BY

CODE

10. THIS ACQUISITION IS

UNRESTRICTED OR

SET ASIDE:

% FOR:

SMALL BUSINESS

HUBZONE SMALL

BUSINESS

SERVICE-DISABLED

VETERAN-OWNED

SMALL BUSINESS

WOMEN-OWNED SMALL BUSINESS

(WOSB) ELIGIBLE UNDER THE WOMEN-OWNED

SMALL BUSINESS PROGRAM

EDWOSB

8(A)

NAICS:

SIZE STANDARD: $15M

11. DELIVERY FOR FOB DESTINA-

TION UNLESS BLOCK IS

MARKED

SEE SCHEDULE

12. DISCOUNT TERMS

13a. THIS CONTRACT IS A

RATED ORDER UNDER

DPAS (15 CFR 700)

13b. RATING

14. METHOD OF SOLICITATION

RFQ

IFB

RFP

15. DELIVER TO

CODE

16. ADMINISTERED BY

CODE

17a. CONTRACTOR/OFFEROR

CODE

FACILITY CODE

18a. PAYMENT WILL BE MADE BY

CODE

TELEPHONE NO.

DUNS:

DUNS+4:

PHONE:

FAX:

17b. CHECK IF REMITTANCE IS DIFFERENT AND PUT SUCH ADDRESS IN OFFER 18b. SUBMIT INVOICES TO ADDRESS SHOWN IN BLOCK 18a UNLESS BLOCK BELOW IS CHECKED

SEE ADDENDUM

19.

20.

21.

22.

23.

24.

ITEM NO.

SCHEDULE OF SUPPLIES/SERVICES

QUANTITY

UNIT

UNIT PRICE

AMOUNT

(Use Reverse and/or Attach Additional Sheets as Necessary)

25. ACCOUNTING AND APPROPRIATION DATA

26. TOTAL AWARD AMOUNT (For Govt. Use Only) 27a. SOLICITATION INCORPORATES BY REFERENCE FAR 52.212-1, 52.212-4. FAR 52.212-3 AND 52.212-5 ARE ATTACHED. ADDENDA

ARE

ARE NOT ATTACHED.

27b. CONTRACT/PURCHASE ORDER INCORPORATES BY REFERENCE FAR 52.212-4. FAR 52.212-5 IS ATTACHED. ADDENDA

ARE

ARE NOT ATTACHED

28. CONTRACTOR IS REQUIRED TO SIGN THIS DOCUMENT AND RETURN _______________

29. AWARD OF CONTRACT: REF. ___________________________________ OFFER

COPIES TO ISSUING OFFICE. CONTRACTOR AGREES TO FURNISH AND

DATED ________________________________. YOUR OFFER ON SOLICITATION

DELIVER ALL ITEMS SET FORTH OR OTHERWISE IDENTIFIED ABOVE AND ON ANY

(BLOCK 5), INCLUDING ANY ADDITIONS OR CHANGES WHICH ARE

ADDITIONAL SHEETS SUBJECT TO THE TERMS AND CONDITIONS SPECIFIED

SET FORTH HEREIN IS ACCEPTED AS TO ITEMS:

30a. SIGNATURE OF OFFEROR/CONTRACTOR 31a. UNITED STATES OF AMERICA (SIGNATURE OF CONTRACTING OFFICER) 30b. NAME AND TITLE OF SIGNER (TYPE OR PRINT) 30c. DATE SIGNED 31b. NAME OF CONTRACTING OFFICER (TYPE OR PRINT) 31c. DATE SIGNED

AUTHORIZED FOR LOCAL REPRODUCTION

(REV. 2/2012)

PREVIOUS EDITION IS NOT USABLE

Prescribed by GSA - FAR (48 CFR) 53.212

7. FOR SOLICITATION

INFORMATION CALL:

STANDARD FORM 1449

OFFEROR TO COMPLETE BLOCKS 12, 17, 23, 24, & 30

SOLICITATION/CONTRACT/ORDER FOR COMMERCIAL ITEMS

MCC-17-RFQ-0005

Carlos I. Torres Roman Email: torresromanci@mcc.gov

202-521-2688 11/14/2016

3:00 P.M. ET

Millennium Challenge Corporation Contracts and Grants Management Division 1099 Fourteenth Street NW Suite 700 Washington DC 20005

X

541614

N/A

Millennium Challenge Corporation

CGM

1099 Fourteenth Street NW, Suite 700 Washington DC 20005

Millennium Challenge Corporation Contracts and Grants Management Division 1099 Fourteenth Street NW Suite 700 Washington DC 20005

Interior Business Center Interior Business Center M/S D-2773 7301 West Mansfield Avenue Lakewood CO 80235-2230

This is a combined synopsis/solicitation for commercial items prepared in accordance with the format in FAR Subpart 12.6 and Conducted per the procedures of AFR Subpart 13.5, as supplemented with additional information included in this notice.

MCC seeks a contractor to provide Independent Evaluation Services for the Georgia II Industry-Led Skills and Workforce Development Project.

See enclosed solicitation and all attachments for pertinent details.

TABLE OF CONTENTS

(End of Table of Contents)

SUPPLIES OR SERVICES AND PRICES/COSTS

Price/Cost Schedule

Price Amount

Time and Materials (T&M) Period of Performance (PoP) TBD (Estimated 12 MO)

OTHER DIRECT COSTS (ODCs) Period of Performance (PoP) TBD (Estimated 12 MO)

Time & Materials (T&M) Period of Performance (PoP) TBD (Estimated 12 MO)

OTHER DIRECT COSTS (ODCs) Period of Performance (PoP) TBD (Estimated 12 MO)

Time & Materials (T&M) Period of Performance (PoP) TBD (Estimated 12 MO)

OTHER DIRECT COSTS (ODCs) Period of Performance (PoP) TBD (Estimated 12 MO)

Time & Materials (T&M) Period of Performance (PoP) TBD (Estimated 12 MO)

OTHER DIRECT COSTS (ODCs) Period of Performance (PoP) TBD (Estimated 12 MO)

Time & Materials (T&M) Period of Performance (PoP) TBD (Estimated 12 MO)

OTHER DIRECT COSTS (ODCs) Period of Performance (PoP) TBD (Estimated 12 MO)

*MCC reserves the right to set the contract ceiling above the total estimated price to account for potential deviations due to unforeseen circumstances. However, the Contractor will be expected to perform in good faith in accordance with their proposed cost estimates.

FAR 52.216-1 -- Type of Contract (APR 1984) The Government contemplates award of a time-and-materials (T&M) contract resulting from this solicitation.

(End of Provision) Accounting and Appropriation Data

[RESERVED]

Funding Type This contract will be funded, at time of award, with No-Year Due Diligence funding.

Limitation of Funds – Incrementally Funded Contracts In accordance with clause MCC 52.232-72, “Limitation of Funds – Incrementally Funded Contracts,” funds in the amount of [TBD] have been allotted to this contract. It is contemplated that funds now obligated under this contract will cover the work to be performed until [TBD].

The Contractor shall not perform work on the contract which exceeds the total amount actually allotted by the Government to the contract. The Government is not obligated to reimburse the Contractor for costs incurred in excess of the total amount allotted by the Government to this contract. The contractor assumes the risk for any increased costs beyond what the Government obligates.

Labor Rates & Other Direct Costs (ODCs) Labor Rates at the inception of a contract period will carry through as “Fixed-Price” without adjustment for the life of that contract period.

Direct Labor Rates & ODCs – Using the template below, Offerors must include all proposed labor categories, fully-burdened hourly rates and Other Direct Costs (ODCs) for each contract period, to include any and all “Data Collection” costs.

Base Period – Period of Performance TBD (est. 12 months)

LABOR

hr.

hr.

hr.

hr.

hr.

Labor Sub-Total

ODCs(1)

ODC Sub-Total

Grand Total (Base Period)

Option Period I – Period of Performance TBD (est. 12 months)

LABOR

hr.

hr.

hr.

hr.

hr.

Labor Sub-Total

ODCs(1)

ODC Sub-Total

Grand Total (Option Period I)

Option Period II– Period of Performance TBD (est. 12 months)

LABOR

hr.

hr.

hr.

hr.

hr.

Labor Sub-Total

ODCs(1)

ODC Sub-Total

Grand Total (Option Period II)

Option Period III – Period of Performance TBD (est. 12 months)

LABOR

hr.

hr.

hr.

hr.

hr.

Labor Sub-Total

ODCs(1)

ODC Sub-Total

Grand Total (Option Period III)

Option Period IV – Period of Performance TBD (est. 12 months)

LABOR

hr.

hr.

hr.

hr.

hr.

Labor Sub-Total

ODCs(1)

ODC Sub-Total

Grand Total (Option Period IV)

NOTE (1): MCC anticipates approximately $200,000.00 worth of data collection subcontracting costs for which the requirements are unknown at this time. To be fair to all vendors providing a quotation, a plug number is utilized for data collection subcontracting costs in the respective year(s) in which data collection is expected to occur. However, if you believe this plug number is inadequate, please advise during the Question and Answer period of the solicitation. “Data Collection Subcontract(s),” as used in the tables above, refers only to the cost of subcontracting a data collection firm. It does not include any prime contractor labor associated with overseeing/managing the data collection effort; this time should be incorporated into the level of effort proposed under the respective “Direct Labor” CLIN. Subcontracting costs for data collection are subject to the requirements of FAR 52.244-2, Subcontracts.

NOTE (2): A job description shall be included for each labor category proposed. The job description must identify essential and non-essential tasks that are assigned to a specific position. It must also identify reporting relationships and shall also describe required qualifications, minimum requirements, working conditions and desirable qualifications.

(End of SECTION B)

DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK

Scope

Provide Independent Evaluation Services in support of the Industry-led Skills and Workforce Development Project.

Background

The United States of America, acting through the Millennium Challenge Corporation (“MCC”) on 26 July, 2013 signed a five-year, $140 million compact with the Government of Georgia. The Compact was ratified by the Parliament of Georgia on October 4, 2013.

The Compact investments are composed of three projects:

· Project 1: Improving General Education Quality Project (IGEQ)

· Project 2: Industry-led Skills and Workforce Development Project (ISWD)

· Project 3: STEM Higher Education Project (SHE) This Contract will cover evaluation services for the entire Industry-led Skills and Workforce Development Project (ISWD). The ISWD Project is a $16 million project under the Compact, which aims to improve the linkage between market-demanded skills and the supply of Georgians with technical skills relevant to the local economy through investments in Technical and Vocational Education and Training (TVET). Investments in TVET are necessary to address industry demand for skilled technicians and to reach potential beneficiaries who may not have the opportunity to obtain further education and training. In September 2014 MCA-G contracted PEM GmbH (PEM) to implement the four main activities under the ISWD Project.

The four activities under this project are:

1) Activity 1: Competitive Program Improvement Grants: To run a competitive process to solicit and fund innovative, industry-driven proposals from Georgian TVET providers for establishing new or expanding/improving existing training programs, to meet industry needs and thus provide a mechanism for incentivising the development and implementation of sustainable and innovative industry-led programmes that set leading examples for TVET reform in Georgia. And, to enhance the employability of TVET graduates leading to a higher proportion of employed graduates with higher salaries.

2) Activity 2: Strengthening TVET Provider Practice: Strengthen TVET provider practice by identifying, strengthening, documenting, disseminating, and promoting uptake of best practices across the sector; run a competitive process for small grants for the creation of innovative tools for TVET Providers.

3) Activity 3: Strengthening TVET Sector Policy; Strengthen national policy with respect to industry engagement in the Georgian TVET sector.

4) Activity 4: Annual TVET Conference: Host an annual TVET conference.[footnoteRef:1] [1: The total funding for the ISWD project is USD 16,000. $12,000 is devoted to Activity 1 and $4,000 is devoted to Activities 2-4. ]

The project logic (See Annex I) holds that the technical assistance provided by the implementing entity will support TVET programs across Georgia in better aligning themselves with the needs of the private sector and providing higher level TVET certificates, and thus offering students the skills to compete in the labor market. In addition, the Competitive Program Improvement Grants will support development and/or implementation of appropriate courses as well as assisting TVET providers (public and private) to be more competitive through financial and technical assistance.

The anticipated timeframe for this Contract is January 2017 to December 2021. Because the Compact is currently under implementation, data collection for these evaluations will be funded by the Compact, at the discretion of the Millennium Challenge Account-Georgia (MCA-G) and MCC. However, there is a possibility that the evaluation will require a follow-up data collection after the Compact closes in July 2019. In that case the Independent Evaluator will be responsible for procuring and managing the data collection firm.

Objective MCC requires an Independent Evaluator to design and implement the evaluation of the Industry-led Skills and Workforce Development Project (ISWD). The objective of the evaluation is to understand the impact of the MCC-funded investments in the TVET sector and to test if the project logic holds true. The Evaluator should strive to compare the outcomes of the four activities listed above in comparison to the outcomes for Georgian TVET providers and graduates without the investment. Further, the evaluation shall answer the question, were these investments the best way to incentivize improvement and innovation among Georgian TVET providers?

Roles and Responsibilities

Each organization, MCC, the MCA-G and its designated representatives, and the independent evaluator, play critical roles in designing, implementing, and disseminating economic analysis and independent evaluations. Their roles are as follows:

Millennium Challenge Corporation (MCC) MCC is responsible for oversight of the Contractor and quality control of its activities, including the following specific responsibilities:

Set the evaluation priorities and evaluation questions. Using the program logic as the foundation, MCC will develop the evaluation priorities and evaluation questions based on existing evidence, the evidence gap, and the learning priorities for the sector and agency.

Build buy-in and ownership of the evaluation. MCC will work with the independent evaluator and MCA to ensure the evaluation design follows the rules of program design and implementation. When possible, MCC will work with the independent evaluator and MCA to develop a program implementation roll-out plan that enables a rigorous evaluation design. When it is not possible to develop a coordinated roll-out plan, MCC and the evaluator will ensure the evaluation design is based on feasible methodologies given program design and implementation. MCC will work with MCA to ensure commitment to the roll-out plan by local stakeholders, and if is not possible to adhere to the roll-out plan, will inform the independent evaluator in order to adapt the evaluation design.

Determine the evaluation budgets. MCC will develop an evaluation budget, including independent evaluators, expected data collection requirements, and any other activities related to the evaluation.

Manage the independent evaluators. MCC will develop Terms of Reference for the independent evaluators and manage the contracting process, including Request for Proposals, Proposal Review, selection of the independent evaluator, development of a work plan, and supervision of the independent evaluator in line with the work plan and agreed deliverables.

Manage the data collection firms. As appropriate, MCC will develop Terms of Reference for the data collection firms, with technical support from MCA and the independent evaluator. In this scenario, the MCC is responsible for managing the contracting process, including Request for Proposals, Proposal Review, selection of the data collection firm, development of a work plan, and supervision of the data collection firm in line with the work plan and agreed deliverables.

Quality review of evaluation materials. Using the program logic as a foundation, MCC is responsible for reviewing evaluation materials, including the design report and survey materials, for feasibility, technical and factual accuracy, and adherence with international standards.

Quality review of evaluation reports. MCC is responsible for reviewing independent evaluators’ reports to ensure technical and factual accuracy. While MCC will provide feedback, the independent evaluator is ultimately the author of the evaluation reports and may or may not take into consideration MCC’s comments. MCC therefore develops an Official Statement for each evaluation report to confirm the technical and factual accuracy of the report, or clarify any differences on technical or factual issues between MCC and the independent evaluator. Through this review process, MCC’s role is to enable the independent evaluator’s independence in order to produce a high quality, unbiased evaluation of the program. MCC should ensure that various stakeholders understand that while MCC, MCA, and other stakeholders may provide guidance and leadership on evaluation questions and feasibility of methodology, once an approved evaluation design is in place, feedback on analysis and interpretation of results is limited to statements on the continued technical and factual accuracy of the analysis.

Lead final public dissemination efforts. MCC is responsible for leading public dissemination of results, in particular ensuring a feedback loop internally in order to inform future decision making on similar program investment, design, and implementation. This requires ensuring the evaluation results are disseminated on the MCC website, and results are disseminated to appropriate stakeholders in a timely fashion.

Millennium Challenge Account – Georgia (MCA-G) The MCA-G is responsible for building local ownership and commitment to the evaluation, oversight of the data collection firm, and quality control of evaluation activities, including the following specific responsibilities:

Build buy-in and ownership of the evaluation. MCA will work with the independent evaluator and MCC to ensure the evaluation design follows the rules of program design and implementation. When possible, MCA will work with the independent evaluator and MCC to develop a program implementation roll-out plan that enables a rigorous evaluation design. MCA will work with MCC to ensure commitment to the roll-out plan by local stakeholders, and if it is not possible to adhere to the roll-out plan, will inform MCC in order to request the independent evaluator adapt the evaluation design.

Manage the data collection firms. During the Compact lifetime, the MCA will develop Terms of Reference for the data collection firms, with technical support from MCC and the independent evaluator. In this scenario, MCA is responsible for managing the contracting process, including Request for Proposals, Proposal Review, selection of the data collection firm, development of a work plan, and supervision of the data collection firm in line with the work plan and agreed deliverables.

Quality review of evaluation materials. Using the program logic as a foundation, MCA is responsible for reviewing evaluation materials, including the design report and survey materials, for feasibility, technical and factual accuracy, and adherence with international standards.

Quality review of evaluation reports. MCA, or a designated representative, is responsible for reviewing independent evaluators’ reports to ensure technical and factual accuracy. While MCA will provide feedback, the independent evaluator is ultimately the author of the evaluation reports and may or may not take into consideration MCA’s comments. MCA therefore develops an Official Statement for each evaluation report to confirm the technical and factual accuracy of the report, or clarify any differences on technical or factual issues between MCA and the independent evaluator.

Facilitate public dissemination efforts. MCA, or a designated representative, is responsible for facilitating public dissemination of results, in particular ensuring a feedback loop in-country in order to inform future decision making on similar program investment, design, and implementation.

Independent Evaluator The independent evaluator is responsible for the overall design, implementation, and dissemination of the evaluation, including the following specific responsibilities:

Develop a rigorous evaluation design given rules of implementation and feasibility of options. Using the program logic as the foundation, independent evaluators must assess the program design and implementation to develop the most rigorous evaluation design feasible, whether it is a performance or impact evaluation, and identify the most appropriate evaluation methodology feasible given the context. The independent evaluator is responsible for developing an evaluation design report, in consultation with MCC and MCA, which summarizes the evaluation design, sampling, analysis plan, and other critical design elements of the evaluation. (Tasks 1-2)

Support MCC and MCA to build buy-in and ownership of evaluation. The independent evaluator will meet with MCC and local stakeholders to develop the evaluation design based on program design and implementation. When possible, the independent evaluator will work with MCC and MCA to advice on a program implementation roll-out plan that enables a rigorous evaluation design. In addition, the independent evaluator will continuously present the evaluation objectives, materials, and interim and final results for MCC and local stakeholders to maintain commitment to the evaluation. (Task 2 and Task 7)

Develop evaluation materials that are held to international standards. Using the program logic as a foundation, the independent evaluators define how key outcomes will be measured and develop the survey instruments or other data collection tools in order to answer critical evaluation questions. Note that “survey” here refers primarily to quantitative surveys, but may also refer to qualitative instruments where appropriate (Task 3 and Task 8)

Ensure appropriate review of evaluation materials and research protocols. Independent evaluators are responsible for ensuring all evaluation materials, including the evaluation design, survey instruments, sampling strategy, data collection and entry protocols are reviewed by an institutional review board (IRB). IRB review and approval is required before any data collection commences. The independent evaluators will document all approvals and informed consent procedures throughout the evaluation time period. (Task 4 and Task 9)

Manage the data collection firms. After Compact closure, in July 2019, the evaluator will develop any necessary Terms of Reference for the data collection firms, with technical support from MCC and the MCA. In this scenario, the independent evaluator is responsible for managing the contracting process, including Request for Proposals, Proposal Review, selection of the data collection firm, development of a work plan, and supervision of the data collection firm in line with the work plan and agreed deliverables. (Task 4 and Task 9)

Supervise data collection. Whether the MCA, MCC or independent evaluator contracts the data collection firm, the independent evaluator is responsible for quality control over data collection activities, including interviewer training, pre-testing, and implementing data quality review checks during data collection. (Task 4 and Task 9)

Lead data cleaning, analysis, interpretation of results. The independent evaluator is responsible for leading any necessary data cleaning and consultation with the data collection firm to produce final analysis files. The independent evaluator is responsible for leading all analysis in line with the agreed analysis plan developed with MCC, and consulting with MCC should the analysis plan need to be revised or adapted. (Task 5 and Task 10)

Produce evaluation reports. The independent evaluator is responsible for sharing initial evaluation reports with local stakeholders and MCC for review and feedback. However, given the independent evaluator is the author of the evaluation reports, local stakeholder and MCC feedback may or may not be taken into consideration for the final report. The independent evaluator is responsible for documenting all feedback and evaluator responses. (Task 5 and Task 10)

Lead public dissemination efforts. The independent evaluator is responsible for leading public dissemination efforts facilitated by MCA and MCC (such as local workshops and conferences), but also additional conferences or other opportunities to publicly disseminate the results of the evaluation. The evaluator will advise MCC on any public dissemination opportunities and collaborate as appropriate. As per Section H, the independent evaluator is responsible for ensuring that all public release documentation is reviewed to minimize any risk to respondent confidentiality. This may require submission of internal use only and public release reports. (Task 11).

Project Outputs and Outcomes

More TVET courses are provided at the higher TVET levels.

Industry engagement processes are developed and included in the selection and running of new courses.

New courses are ‘accredited’ as part of the GoG TVET offerings.

Industry has access to Georgian labor with the skills needed to fill current and/or future vacancies. Industry has confidence that the TVET sector’s output meets their needs ‘Students’[footnoteRef:3] will have the skills to get better jobs. [3: The term “student” as used in this document refers to participates in the TVET programs. These individuals could be industry based employees or traditional college students. ]

As a result of better job placement and salaries we should see an increase in household income.

Better alignment of industry needs with TVET activity as TVET providers work with industry on all aspects of TVET provision (identify skills in demand, design, delivery, assessment, governance and so on) Employer satisfaction survey.

Tracer studies.

Employment data.[footnoteRef:4] [4: Employment data may be hard to find in this context but the Evaluator may propose data collection methods that fit the task.]

2) Activities 2 & 4: Strengthening TVET Provider Practice (STPP) and Annual TVET conference: Strengthen TVET provider practice by identifying, strengthening, documenting, disseminating, and promoting uptake of best practices across the sector;

Good practice examples are documented and disseminated.

Small scale Good Practice Grants are disbursed TVET providers better align their activities to good practices[footnoteRef:5] resulting in better alignment of TVET activity to the needs of industry and business. [5: See Annex II PEM’s Good Practice Hand Book]

Recipients of the Good Practice Grants develop tools and materials that other TVET providers can use The tools developed through the Good Practice Grants are useful in strengthening the connection between industry firms with TVET providers Employer satisfaction survey that measures industry has confidence in the TVET sector’s ability to meet their needs.

Offering of Industry Placements, data on outreach programs managed by the TVET providers Rate of industry sponsorship of the conference.

PEM materials on TVET sector policy are put into use by TVET providers and the MoE.

Georgian TVET policy is structured to align the design, delivery, assessment, and outcomes to the needs to business and industry Formal processes are in place to engage industry in TVET policy development and implementation.

Students who complete training get better job placements and earn better salaries.

Industry has confidence in the TVET sector’s ability to meet their needs.

Employer satisfaction survey Offerings of Industry Placements, data on outreach programs managed by the TVET providers

Project Beneficiaries

The direct beneficiaries of the ISWD project are the students that enroll in the improved TVET programs. Indirect beneficiaries are the immediate families of those students. Teachers, administrators, and other TVET sector stakeholders will participate in the project, though they aren’t considered beneficiaries according to MCC’s definitions.

Evaluation Methodology

Based on the current project logic, MCC has defined the underlying assumptions for how MCC thinks the selected investment will lead to changes in employment outcomes, and ultimately poverty reduction. The current program logic and key assumptions have informed the list of key evaluation questions below. However, the Independent Evaluator is expected to explicitly assess the program logic, key assumptions, and evaluation questions as part of the Evaluation Design Report, recommending revisions to MCC, as necessary. MCC does not expect the Evaluator to answer every question included. The list below represents a broad set of potential questions of interest to MCC. Part of the Evaluator’s role (see Task 1, below) is to assess if the questions included here can be answered, allowing MCC to guide the Evaluator to leverage evaluation resources towards the goals of learning and accountability.

The evaluation of the ISWD project will be a performance evaluation that includes analysis of longitudinal data on employment rates and placements, as well as rigorous qualitative research. The evaluation should assess the impact of each the four components listed above. Although the timelines for each component may not match up perfectly, the evaluator should propose an innovative strategy to that incorporates all of the components in one evaluation, and ultimately on final report.

The evaluation of Activity 1 should include two areas: 1) monitoring the progress of the students enrolled in the programs that receive the Large Competitive Grants, including post-graduation achievements; and 2) assessing the quality and success of the MCC funded programs with regards to private sector engagement, and linking curriculum to the skills demanded by the labor market.

The evaluator may propose a methodology that groups Activities 2-4 together as they are closely related. The evaluation should, however, address the central evaluation questions for each activity (listed below). The evaluation of these three activities should assess how PEM’s industry outreach and promotion of best practices and sector policies have strengthened ties to the private sector, through examining private sector support for Georgian TVET programs and the perception of TVET programs in Georgia among employers. The precise best practices have not been defined yet, however PEM’s recommendation on best practice areas can be found in Annex II: PEM’s Best Practice Guidance. The methodology for evaluating Activities 2-4 will depend on the how this component is further developed. Qualitative Methods could be used to understand what affects adoption rates of the identified best practices, how they affect TVET program administration, and how the interaction between TVET providers and the private sector changes as a result.

Outcomes should be monitored through both quantitative and qualitative methods, including the use of longitudinal employment data when available, tracer studies to follow a sample of those students who enter the Compact funded TVET programs, and those that enter other TVET programs. Quantitative methods would be best applied to measure the job placement outcomes, and salaries.

The evaluation should also include qualitative analysis that assesses the needs of employers and their experience with TVET graduates, as well as the success of TVET providers and the MoE in adopting the best practices identified in Activity 2 and the policies identified in Activity 3. The Evaluator should propose a rigorous qualitative methodology, including sampling methodology and rational. A crucial objective of the qualitative analysis will be to provide a qualitative map of different stakeholder needs. The evaluation should assess what industry firms seek in staffing vocational positions and if TVET providers are producing graduates who fit that profile. In addition, the evaluation should seek to understand the process in which TVET providers engage with the private sector. Ideally, the Evaluator would use incremental data analysis to compare outcomes before and after project implementation. Due to the varied nature of the TVET programs in Georgia case studies of specific providers will not be sufficient to measure all of the intended outcomes.

The level of effort devoted to each activity should be proportional to the size of the activity, indicated in Section C.2 by funding provided for the implementation of each.

Evaluation Questions

Activity 1: Competitive Program Improvement Grants

Are high-demand competencies being built?

Are there different outcomes for men and women? Do the programs succeed in attracting women to non-traditional fields?

Do competitive grants incentivize design in TVET programs that attract women into non-traditional sectors?

What are future, sustainable funding options: Were schools able to get significant levels of co-financing from the private sector? Are schools able to charge above the government scholarship level for TVET 4-5 courses?

Do competitive grants for TVET programs increase employment of Georgians in targeted sectors at targeted levels?

Has the grants program incentivized those TVET programs to improve and innovate?

Did graduates from those programs that were rated higher during the TEP have higher rates of job placement and starting salaries?

Does number of workers imported to fill technician-level positions by private sector decrease?

Activity 2 and 4: Strengthening TVET Provider Practice and Annual TVET Conference

Do TVET institutions see the value of adopting best practices? Do they think investing in them will pay off in the long run? Will they continue the best practices and new tools once the grant funding is gone?

Have other TVET providers begun to use the tools and material developed by the good practice grant recipients?

Do non-grant awardees find the conferences useful?

Do more TVET providers or other institutions apply for grants after the annual conferences?

How have the annual TVET Conferences affected the interaction between TVET providers and employers?

Does the promotion of best practices increase the quality of the TVET programs in Georgia?

Has the industry outreach as part of the TVET project been successful in creating stronger ties between these programs and the private sector?

Do TVET Providers understand the labor needs of the private sector?

Will this sector support lead to greater private sector investment in STEM sectors?

Will clearer articulation by industry of precisely which skills they need and use lead to more targeted programs offered by TVET programs?

Will industry demand driven programs increase employability and income earning potential for graduates of these programs?

Activity 3: Strengthening TVET Sector Policy

How long should it take TVET providers to adapt?

Has Ministry piloted the structure for SCs proposed by PEM?

Where SCs able to secure active participation of industry?

Has Ministry adopted or plans to adopt the proposed SCs’ structure?

Are the changes made under the program sustainable in the future?

Has Ministry adopted policies to support Career Education Guidance (CEG) services at TVET programs?

How many schools has functional services?

Has private sector supported CEG at schools?

Data Sources

· Administrative data on enrollment from TVET Competitive Grant program recipients

· Employment data[footnoteRef:6] [6: Employment data may be hard to find or gain access to. The evaluator should propose a methodology for data collection based on what they believe is feasible and reliable. ]

· Financial data from the grant recipients to examine financial viability

· Interviews with TVET providers who receive the grants

· Surveys of beneficiaries

· Survey of industry firms on recruitment and employment of TVET graduates

· Tracer Studies of students from the TVET grantees

· Sector employment rates, and resources devoted nationally to those sectors.

Independent Evaluator Tasks In order to execute the design and implementation of the evaluation, the Contractor is responsible for the following tasks:

Task 1: Assess Evaluation Plan The Evaluator is responsible for conducting an Evaluability Assessment on all existing documentation related to the program and proposed evaluation plan provided by MCC and MCA using the MCC Evaluability Assessment Tool.

MCC and the Evaluator should identify all MCC and MCA self-evaluation activities and determine how the independent evaluation will (i) validate findings of self-evaluation(s), (ii) use self-evaluation(s) to inform the independent evaluation, particularly around understanding what was designed, what was implemented, and understanding any necessary course corrections, (iii) complement the self-evaluation(s) by expanding on self-evaluation activities.

The Evaluator will provide an Evaluability Assessment Report on the Problem Diagnostic, Program Logic, Risks & Assumptions, Proposed Research Questions, Methodology, and other Metrics

Develop a Work Plan with expected deadlines for deliverables. Any revisions to the Work Plan will be submitted in the Evaluator’s Progress Reports.

Task 2: Develop Evaluation Design Report The Evaluator will conduct missions to the field as needed to discuss program design and implementation issues to inform the evaluation design, as well as engage local stakeholders (MCA and Implementing Entities, as well as local government) on the proposed evaluation. This includes hosting local stakeholder workshops to develop/refine research questions, assess appropriate and feasible quantitative and qualitative evaluation methodologies, develop evaluation timeline, and build local stakeholder commitment to the evaluation design.

The Evaluator will develop the Evaluation Design Report, including all technical analysis required for assessing and confirming internal validity, power calculations, and required sample sizes. The Evaluation Design Report outline will be developed in consultation with the MCC M&E lead. Please reference Annex IV Evaluation Design Report Outline as a reference.

As per MCC’s Gender Policy and Gender Integration Guidelines, the Evaluator will ensure that evaluation design, data collection requirements, and analysis address gender and other social differences as appropriate. This involves including research questions that address social and gender dimensions of the project logic, assumptions, and project design; disaggregating impacts by gender, age, income, and other social difference (e.g. ethnicity) as appropriate; including special modules or studies as needed.

The Evaluator will document feedback from local stakeholders and the response to that feedback. In order to obtain feedback, the Evaluator may lead local workshops and/or virtual reviews of the Design Report by MCA and Implementing Entities.

The Evaluator will submit the Evaluation Design Report to MCC and may be asked to present to the Evaluation Management Committee. All feedback and response to feedback will be documented by the Evaluator.

As per Section H, MCC may: (i) select to obtain external peer review; and (ii) make the Report and peer reviews public on the MCC website.

The Evaluator will continue to maintain the Evaluation Design Report as the evaluation develops. The Evaluation Design Report will remain a living document that is updated by the Evaluator as necessary and resubmitted to MCC as necessary.

Task 3: Develop Evaluation Materials The Evaluator will provide technical support on development of data collection firm TORs.

The Evaluator will lead development of any necessary survey instruments for data extraction from existing sources or data collection for new sources.

The Evaluator will develop survey materials in English, unless otherwise agreed with MCC. When necessary, the Evaluator will utilize existing instruments to improve coordination across MCC independent evaluations for meta-analysis. The Evaluator is responsible for any necessary pilot testing of survey instruments and review of back-translations conducted by data collection firms prior to data collection start.

The Evaluator is responsible for ensuring household surveys last no longer than 120 minutes without written consent from the Evaluation Management Committee. The Evaluator is responsible for reporting on the average survey time resulting from the pilot test and from field work.

The Evaluator will document feedback from local stakeholders and response to the feedback. In order to obtain feedback, the Evaluator may lead local workshops and/or virtual reviews of the Evaluation Materials by MCA and Implementing Entities.

The Evaluator will submit the Evaluation Materials to MCC and may be asked to present to the Evaluation Management Committee. All feedback and response to feedback will be documented by the Evaluator. The Evaluator is responsible for the quality of the English and local language survey instruments.

Task 4: Prepare and Supervise baseline data collection The Evaluator is responsible for developing and overseeing all data collection and entry protocols and implementation. This requires ensuring appropriate Institutional Review Board (IRB) review and approval of research protocols, survey instruments, and informed consent procedures (please reference the Annex V MCC Informed Consent Statement Template).

The Evaluator is responsible for overseeing the training of data collection staff, field work, and data entry. This must be managed in coordination with MCC and MCA.[footnoteRef:7] [7: Please note: during the Compact lifetime the MCA is responsible for procuring the data collection firm. However, the Evaluator is responsible for overseeing data collection and should budget for this work.]

The Evaluator is responsible for conducting and documenting quality control checks of data as it is collected and entered in order to reduce data collection and entry error.

The Evaluator is responsible for establishing a system for frequent communication with the data collection firm throughout data collection and entry, through email, phone, and site visits.

Task 5: Develop Baseline Report and Data Files The Evaluator will review raw files from the survey firm and provide any critical feedback on production of the survey firm’s Data Collection Report.

Using the raw dataset, the Evaluator will produce a cleaned analysis dataset that follows MCC’s guidelines for public use data, see Annex VI - MCC Data Documentation and Anonymization Requirements.

In addition, the Evaluator will produce a full set of metadata for each survey that follows MCC’s guidelines for public use data, see Annex VI- MCC Data Documentation and Anonymization Requirements and Nesstar Template Annex VII.

The Evaluator is responsible for developing the Baseline Report.