MCC-16-RFQ-0131_Guatemala_Education_Project.pdf

PAGE 1 OF

1. REQUISITION NO.

2. CONTRACT NO. 3. AWARD/EFFECTIVE DATE 4. ORDER NO. 5. SOLICITATION NUMBER 6. SOLICITATION ISSUE DATE

a. NAME b. TELEPHONE NO. (No Collect Calls) 8. OFFER DUE DATE/LOCAL

TIME

9. ISSUED BY CODE 10. THIS ACQUISITION IS UNRESTRICTED OR SET ASIDE: % FOR:

SMALL BUSINESS

HUBZONE SMALL

BUSINESS

SERVICE-DISABLED

VETERAN-OWNED

SMALL BUSINESS

WOMEN-OWNED SMALL BUSINESS

(WOSB) ELIGIBLE UNDER THE WOMEN-OWNED

SMALL BUSINESS PROGRAM

EDWOSB

8(A)

NAICS:

SIZE STANDARD:

11. DELIVERY FOR FOB DESTINA-

TION UNLESS BLOCK IS

MARKED

SEE SCHEDULE

12. DISCOUNT TERMS

13a. THIS CONTRACT IS A

RATED ORDER UNDER

DPAS (15 CFR 700)

13b. RATING

14. METHOD OF SOLICITATION

RFQ IFB RFP

15. DELIVER TO CODE 16. ADMINISTERED BY CODE

17a. CONTRACTOR/OFFEROR CODE FACILITY CODE 18a. PAYMENT WILL BE MADE BY CODE

TELEPHONE NO. DUNS: DUNS+4:

PHONE: FAX:

17b. CHECK IF REMITTANCE IS DIFFERENT AND PUT SUCH ADDRESS IN OFFER

18b. SUBMIT INVOICES TO ADDRESS SHOWN IN BLOCK 18a UNLESS BLOCK BELOW IS CHECKED

SEE ADDENDUM

19. 20. 21. 22. 23. 24.

ITEM NO. SCHEDULE OF SUPPLIES/SERVICES QUANTITY UNIT UNIT PRICE AMOUNT

(Use Reverse and/or Attach Additional Sheets as Necessary)

25. ACCOUNTING AND APPROPRIATION DATA 26. TOTAL AWARD AMOUNT (For Govt. Use Only)

27a. SOLICITATION INCORPORATES BY REFERENCE FAR 52.212-1, 52.212-4. FAR 52.212-3 AND 52.212-5 ARE ATTACHED. ADDENDA ARE ARE NOT ATTACHED.

27b. CONTRACT/PURCHASE ORDER INCORPORATES BY REFERENCE FAR 52.212-4. FAR 52.212-5 IS ATTACHED. ADDENDA ARE ARE NOT ATTACHED

28. CONTRACTOR IS REQUIRED TO SIGN THIS DOCUMENT AND RETURN _______________ 29. AWARD OF CONTRACT: REF. ___________________________________ OFFER

COPIES TO ISSUING OFFICE. CONTRACTOR AGREES TO FURNISH AND DATED ________________________________. YOUR OFFER ON SOLICITATION

DELIVER ALL ITEMS SET FORTH OR OTHERWISE IDENTIFIED ABOVE AND ON ANY (BLOCK 5), INCLUDING ANY ADDITIONS OR CHANGES WHICH ARE

ADDITIONAL SHEETS SUBJECT TO THE TERMS AND CONDITIONS SPECIFIED SET FORTH HEREIN IS ACCEPTED AS TO ITEMS:

30a. SIGNATURE OF OFFEROR/CONTRACTOR 31a. UNITED STATES OF AMERICA (SIGNATURE OF CONTRACTING OFFICER)

30b. NAME AND TITLE OF SIGNER (TYPE OR PRINT) 30c. DATE SIGNED 31b. NAME OF CONTRACTING OFFICER (TYPE OR PRINT) 31c. DATE SIGNED

AUTHORIZED FOR LOCAL REPRODUCTION (REV. 2/2012)

PREVIOUS EDITION IS NOT USABLE Prescribed by GSA - FAR (48 CFR) 53.212

7. FOR SOLICITATION

INFORMATION CALL:

STANDARD FORM 1449

OFFEROR TO COMPLETE BLOCKS 12, 17, 23, 24, & 30

SOLICITATION/CONTRACT/ORDER FOR COMMERCIAL ITEMS 78

MCC-16-RFQ-0131 05-24-2016

(202) 772-6319

06-24-2016 @ 1 PM ET

Millennium Challenge Corporation

Contracts and Grants Management Division

1099 Fourteenth Street NW

Suite 700, Merle Phillips, PSC Rep. MCC

Washington DC 20005

X

N/A

CGM

1099 Fourteenth Street NW, Suite 700

Contracts and Grants Management Division

1099 Fourteenth Street NW

Suite 700

Interior Business Center

M/S D-2773

7301 West Mansfield Avenue

Lakewood CO 80235-2230

MCC seeks a Contractor to provide Independent Evaluation

Services for the Guatemala Threshold Program Education

Project.

See CONTINUATION Page

X X

Jacqueline H. Naranjo, CO

MCC-16-RFQ-0131 (Guatemala Education Project) Page 2

Table of Contents

SECTION A

A.1 SF 1449 SOLICITATION/CONTRACT/ORDER FOR COMMERCIAL ITEMS

SECTION B – SUPPLIES OR SERVICES AND PRICE/COSTS

B.1 PRICE/COST SCHEDULE

B.2. FAR 52.216-1 -- Type of Contract (APR 1984)

B.3. Accounting and Appropriation Data

B.4. Funding Type

B.5 Limitation of Funds – Incrementally Funded Contracts

B.6 Labor Rates & Other Direct Costs (ODCs)

SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK

C.1. Introduction and Background

C.2. Scope

C.3 Evaluation Questions and Proposed Methodology

C.4 Roles and Responsibilities

C.5 Tasks

SECTION D – PACKAGING AND MARKING

SECTION E – INSPECTION AND ACCEPTANCE

E.1. Inspection of Reports

E.2. Quality Control Plan

SECTION F - DELIVERIES OR PERFORMANCE

F.1. Period of Performance

F.2. Place of Performance

F.3. Deliverables

F.4. Reporting Deliverables

F.5. Deliverable Format

F.6. Acceptance Criteria

SECTION G – CONTRACT ADMINISTRATION DATA

G.1. MCC 52.201-70 CONTRACTING OFFICER’S REPRESENTATIVE (COR) AND/OR PROJECT

MONITOR (PM) (JULY 2012)

G.2. MCC 52.232-70 INVOICE INSTRUCTIONS (MCC CUSTOM)

SECTION H – SPECIAL CONTRACT REQUIREMENTS

H.1. MCC 52.203-70 CONTRACTOR NON-DISCLOSURE AGREEMENT (JULY 2012)

H.2. MCC 52.204-70 SECURITY REQUIREMENTS FOR CONTRACTOR PERSONNEL (MAY 2013)

H.3. MCC 52.209-70 ORGANIZATIONAL CONFLICTS OF INTEREST: PRECLUSION FROM

IMPLEMENTATION CONTRACT (JULY 2012)

H.4. MCC 52.232-72 LIMITATION OF FUNDS – INCREMENTALLY FUNDED CONTRACTS (NOV 2006)

MCC-16-RFQ-0131 (Guatemala Education Project) Page 3

H.5. MCC 52.232-73 Travel Reimbursement (MAY 2013)

H.6. MCC 52.232-74 ODC REIMBURSEMENT (JAN 2007)

H.7. MCC 52.242-70 CONTRACTOR PERFORMANCE ASSESSMENT RATING SYSTEM (CPARS)

REGISTRATION (AUG 2011)

H.8. MCC 52.243-70 INCREASE IN SERVICES (OCT 2006)

H.9. MCC 52.245-70 CONTRACTOR ACQUIRED MCC GOVERNMENT PROPERTY OVERSEAS (JULY

2012)

H.10. KEY PERSONNEL

H.11. TEAM & KEY PERSONNEL QUALIFICATIONS

H.12. Provision on Peer Review and Posting of Reports

H.13. Provision on Insurance Maintenance

H.14. Disclaimer for Deliverables

H.15. Provision on Stakeholder Input for Evaluation Products

H.16. Provision on Human Subjects Protection

H.17. Personally Identifiable Information and Confidentiality

H.18. Provision on Delivered Work Product Free from 3rd Party Rights

PART II – CONTRACT CLAUSES

SECTION I – CONTRACT CLAUSES

I.1. 52.212-4 CONTRACT TERMS AND CONDITIONS—COMMERCIAL ITEMS/ALTERNATE I (MAY

2015/MAY 2014)

I.2. 52.212-5 CONTRACT TERMS AND CONDITIONS REQUIRED TO IMPLEMENT STATUTES OR

EXECUTIVE ORDERS—COMMERCIAL ITEMS (MAR 2016)

I.3. 52.214-34 SUBMISSION OF OFFERS IN THE ENGLISH LANGUAGE (APR 1991)

I.4. 52.214-35 SUBMISSION OF OFFERS IN U.S. CURRENCY (APR 1991)

I.5. 52.217-9 OPTION TO EXTEND THE TERM OF THE CONTRACT (MAR 2000)

I.6. FAR 52.252-2 - CLAUSES INCORPORATED BY REFERENCE (FEB 1998)

PART III – LIST OF DOCUMENTS, EXHIBITS AND OTHER ATTACHMENTS

SECTION J – LIST OF ATTACHMENTS

SECTION K – REPRESENTATIONS, CERTIFICATIONS AND OTHER STATEMENTS OF OFFERORS OR

RESPONDENTS

SECTION L – INSTRUCTIONS, CONDITIONS, AND NOTICES TO OFFERORS

L.1. INQUIRIES/QUESTIONS CONCERNING THE SOLICITATION

L.2. EXCLUSION OF QUOTES AND PRE-AWARD COMMUNICATIONS WITH OFFERORS

L.3. QUOTATION DUE DATE

L.4. QUOTATION RESPONSE

L.5. QUOTATION VOLUMES

L.6. QUOTATION FORMAT

L.7. PAGE LIMITS

L.8. PREPARATION OF VOLUME I – TECHNICAL QUOTATION

L.9. PREPARATION OF VOLUME II – COST QUOTATION

SECTION M – EVALUATION FACTORS FOR AWARD

MCC-16-RFQ-0131 (Guatemala Education Project) Page 4

M.1. FAR 52.212-1 – INSTRUCTIONS TO OFFERORS – COMMERCIAL ITEMS (OCT 2015)

M.2. FAR 52.212-2 – EVALUATION – COMMERCIAL ITEMS (OCT 2014)

M.3. EVALUATION FACTOR 1 – TECHNICAL CAPABILITY

M.4. EVALUATION FACTOR 2 – PAST PERFORMANCE

M.5. EVALUATION FACTOR 3 – PRICE

M.6 EVALUATION METHODOLOGY

M.7. BASIS FOR AWARD

MCC-16-RFQ-0131 (Guatemala Education Project) Page 5

SECTION B – SUPPLIES OR SERVICES AND PRICE/COSTS

B.1 PRICE/COST SCHEDULE

CLIN Description of Supplies/Services Qty U/I Unit

Price Amount

0001 BASE PERIOD

DIRECT LABOR

Period of Performance (PoP) TBD

(Estimated 12 MO)

PoP Start: TBD

PoP End: TBD

1 LO ______________ ______________

0002 BASE PERIOD

OTHER DIRECT COSTS (ODCs)

Period of Performance (PoP) TBD

(Estimated 12 MO)

PoP Start: TBD

PoP End: TBD

1 LO ______________ ______________

1001 OPTION PERIOD I

DIRECT LABOR

Period of Performance (PoP) TBD

(Estimated 12 MO)

PoP Start: TBD

PoP End: TBD

1 LO ______________ ______________

1002 OPTION PERIOD I

OTHER DIRECT COSTS

Period of Performance (PoP) TBD

(Estimated 12 MO)

PoP Start: TBD

PoP End: TBD

1 LO ______________ ______________

2001 OPTION PERIOD II

DIRECT LABOR

Period of Performance (PoP) TBD

(Estimated 12 MO)

PoP Start: TBD

PoP End: TBD

1 LO ______________ ______________

2002 OPTION PERIOD II

1 LO ______________ ______________

MCC-16-RFQ-0131 (Guatemala Education Project) Page 6

OTHER DIRECT COSTS

Period of Performance (PoP) TBD

(Estimated 12 MO)

PoP Start: TBD

PoP End: TBD

3001 OPTION PERIOD III

DIRECT LABOR

Period of Performance (PoP) TBD

(Estimated 12 MO)

PoP Start: TBD

PoP End: TBD

1 LO ______________ ______________

3002 OPTION PERIOD III

OTHER DIRECT COSTS

Period of Performance (PoP) TBD

(Estimated 12 MO)

PoP Start: TBD

PoP End: TBD

1 LO ______________ ______________

4001 OPTION PERIOD IV

DIRECT LABOR

Period of Performance (PoP) TBD

(Estimated 12 MO)

PoP Start: TBD

PoP End: TBD

1 LO ______________ ______________

4002 OPTION PERIOD IV

OTHER DIRECT COSTS

Period of Performance (PoP) TBD

(Estimated 12 MO)

PoP Start: TBD

PoP End: TBD

1 LO ______________ ______________

TOTAL ESTIMATED PRICE:* _______________

*MCC reserves the right to set the contract ceiling above the total estimated price to account for potential deviations due to unforeseen circumstances. However, the Contractor will be expected to perform in good faith in accordance with their proposed cost estimates.

B.2. FAR 52.216-1 -- Type of Contract (APR 1984)

The Government contemplates award of a Time & Materials (T&M) contract resulting from this solicitation.

(End of Provision)

MCC-16-RFQ-0131 (Guatemala Education Project) Page 7

B.3. Accounting and Appropriation Data

[RESERVED]

B.4. Funding Type

This contract will be funded, at time of award, with No-Year Due Diligence funding.

B.5 Limitation of Funds – Incrementally Funded Contracts

In accordance with clause MCC 52.232-72, “Limitation of Funds – Incrementally Funded Contracts,” funds in the amount of [TBD] have been allotted to this contract. It is contemplated that funds now obligated under this contract will cover the work to be performed until [TBD].

The Contractor shall not perform work on the contract which exceeds the total amount actually allotted by the

Government to the contract. The Government is not obligated to reimburse the Contractor for costs incurred in excess of the total amount allotted by the Government to this contract. The contractor assumes the risk for any increased costs beyond what the Government obligates.

B.6 Labor Rates & Other Direct Costs (ODCs)

Labor rates at the inception of a contract period will carry through as “Fixed-Price” without adjustment for the life of that contract period.

Direct Labor Rates & ODCs – Using the template below, Offerors must include all proposed labor categories, fully-burdened hourly rates and Other Direct Costs (ODCs) for each contract year identified in B.1.

Base Period -– Period of Performance TBD (est. 12 months)

Item # Description(2) U/I Qty Rate Total

LABOR

hr.

hr.

hr.

hr.

Labor Sub-Total

ODCs(1)

0002 Subcontract(s) – Data Collection Lot 1 $400,000.00 $400,000.00

ODC Sub-Total

Grand Total

Option Period I - Period of Performance TBD (est. 12 months)

LABOR

hr.

hr.

hr.

hr.

Labor Sub-Total

MCC-16-RFQ-0131 (Guatemala Education Project) Page 8

Option Period II - Period of Performance TBD (est. 12 months)

LABOR

hr.

hr.

hr.

hr.

Labor Sub-Total

2002 Subcontract(s) – Data Collection Lot 1 $100,000.00 $100,000.00

Option Period III - Period of Performance TBD (est. 12 months)

LABOR

hr.

hr.

hr.

hr.

Labor Sub-Total

Option Period IV - Period of Performance TBD (est. 12 months)

LABOR

hr.

hr.

hr.

hr.

Labor Sub-Total

4002 Subcontract(s) – Data Collection (Endline) Lot 1 $500,000.00 $500,000.00

MCC-16-RFQ-0131 (Guatemala Education Project) Page 9

NOTE (1): MCC anticipates approximately $1 million worth of data collection subcontracting costs for which the requirements are unknown at this time. To be fair to all vendors providing a quotation, a plug number is utilized for data collection subcontracting costs in the respective years in which data collection is expected to occur. However, if you believe this plug number is inadequate, please advise during the Question and Answer period of the solicitation. “Data

Collection Subcontract(s),” as used in the tables above, refers only to the cost of subcontracting a data collection firm. It does not include any prime contractor labor associated with overseeing/managing the data collection effort; this time should be incorporated into the level of effort proposed under the respective “Direct Labor” CLIN. Subcontracting costs for data collection are subject to the requirements of FAR 52.244-2, Subcontracts.

NOTE (2): A job description shall be included for each labor category proposed. The job description must identify essential and non-essential tasks that are assigned to a specific position.

It must also identify reporting relationships and shall also describe required qualifications, minimum requirements, working conditions and desirable qualifications.

(End of SECTION B)

MCC-16-RFQ-0131 (Guatemala Education Project) Page 10

SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK

C.1. Introduction and Background

The United States of America, acting through the Millennium Challenge Corporation (“MCC”) on April 8, 2015 signed a

$28 million Threshold Grant Agreement with the Government of Guatemala. The program will support the Government of Guatemala in implementing policy and institutional reforms aimed at providing quality educational opportunities for

Guatemala’s youth that have relevance to the labor market, and mobilizing additional government resources that are needed to address binding constraints to economic growth. The Guatemala Threshold1 Program is composed of two projects; an Education Project and a Resource Mobilization Project, with the focus of this consultancy on the Education

Project.

C.2. Scope

The Contractor shall provide support to MCC for Independent Evaluation Services in support of the Education Project in

Guatemala, under the Guatemala Threshold Program.

C.2.1. Project 1: Education Project

This Contract will cover evaluation services for the entire Education Project, which is comprised of three activities, described below. The objective of the Education Project is to support the efforts undertaken by the Government in the implementation of institutional reforms, defined in the Proposal for the Transformation of Secondary Education and Ruta

Crítica (see Annex A), that are designed to provide quality education that prepares a diverse Guatemalan youth to be successful in the labor market.

Project 1, Activity 1: Quality of Education in Support of Student Success

The Quality of Education Activity seeks to strengthen competencies of teachers, principals, and administrators across all school modalities2 in order to promote equitable, high quality secondary education. The activity seeks to promote long-term sustainable improvement through the implementation of a continuous teacher education program and establishment of school networks.

With the support of the Education Consultant hired by MCC under the Threshold Program, the Ministry of Education

(MINEDUC) will develop a Teacher Continuing Education Program (Diplomado), initiate its implementation, and refine its design throughout the lifetime on the project. While the Education Consultant will design and pilot the training

1 Threshold programs assist countries to become compact eligible by supporting their efforts to implement key policy and institutional reforms and thereby demonstrate their commitment to MCC’s eligibility criteria. If successfully implemented, these reforms will reduce constraints to faster economic growth and provide MCC critical information about a candidate country’s political will and capacity to undertake the types of reforms that would have the greatest impact in compacts. MCC develops each threshold program in partnership with the candidate country through a structured and disciplined diagnostic and design process. The process begins with a rigorous analysis of the binding constraints to economic growth. MCC and the candidate country then assess the policies and institutions that reinforce those constraints and design a program that will support reforms aimed at alleviating the constraints.

2 The lower secondary level includes regular schools and a variety of alternative modalities. These include regular INEB (Instituto

Nacional de Educación Básica – National Basic Education Institutes), PEMEM (Proyecto Extensión y Mejoramiento de la Enseñanza

Media – Secondary Education Extensions and Improvement Project), experimental institutes, Telesecundaria (Tele-Secondary) schools, and NUFED (Núcleos Familiares Educativos para el Desarrollo - Family Education Centers for Development) under the out of school subsystem.

MCC-16-RFQ-0131 (Guatemala Education Project) Page 11 program and supervise its implementation, the delivery of the post-pilot trainings will be provided to cohorts by one or more national (public and/or private) universities under a separate agreement.

The Quality of Education Activity also includes the establishment of School Networks (Redes Escolares). The Education

Consultant will support MINEDUC to establish functional school networks throughout the program area of influence with the goal of increasing parent involvement and improving teaching, learning and accountability. Below is a table of preliminary target numbers for the Quality of Education Activity.

As part of the Quality of Education Activity MINEDUC will train an estimated 2,400 lower secondary school teachers and 240 lower secondary school principals. In addition, MINEDUC will facilitate the establishment of approximately 450 parent councils within the project area.

Project 1, Activity 2: Improving Technical and Vocational Education and Training (TVET)

The TVET activity aims to support MINEDUC and other GOG entities in developing new tools and strategies for TVET that respond to the current and projected labor market demand as envisioned in the Plan for the Transformation of

Secondary Education (PTNM). The TVET activity will build capacity for TVET Providers to develop curriculum, provide trainings in MINEDUC and other stakeholders, assist in the implementation of TVET curriculum and programs, and strengthen links between TVET programs and the private sector.

Project 1, Activity 3: Strengthening of Institutional and Planning Capacity

This activity seeks to strengthen the institutional capacity of MINEDUC to better plan and budget for the equitable provision of a quality secondary education. This Activity will also support the broader institutional strengthening envisioned in the Plan for the Transformation of Secondary Education.

The Education Consultant will assist MINEDUC in (1) improving quality of education, closing equity gaps in quality of education provided/delivered and ensure that resources are well managed across lower secondary modalities, (2) enhancing its capacity to plan and manage teacher recruitment and professional development in lower secondary, and (3) strengthening its capacity to assess and plan secondary education supply and demand (including TVET).

MCC-16-RFQ-0131 (Guatemala Education Project) Page 12

Program Logic3

Activities Outputs Short term outcomes

Medium term outcomes

Identified problems

Low level of quality of the educational system.

Activity 1:

Quality of

Education in Support of student success

Pedagogical Support & Learning Communities

Teacher Training and Certificate Program

Implemented

Formation of School Networks

Technical support for PTAs in their in monitoring school success

Increase in Teacher Competency

Teacher Diagnostics Conducted

Increased Time on Task in Classroom

Implementation of Action Plans by Parent

Councils

Improved dissemination of information about secondary offerings to primary school parents Increased Retention and promotion of secondary students

Improved Student Learning as measured on standardized tests

Lack of alignment with the labor market

Activity 2:

Improving

Technical and Vocational Education and Training

(TVET)

Hands-on Work Training

New degrees are better aligned with labor market needs

New Secondary degrees are competency based

Gender sensitive job counseling

Increased number of secondary graduates with competencies demanded by labor.

market

Improved mechanism for feedback of the labor market to inform TVET offerings.

Increased the offering and quality of technical training in secondary education

Inadequate funding for Education

Activity 3:

Strengthening of Institutional and Planning

Capacity

Integrated perspective of all secondary education modalities

Stronger process for Competitive, merit based recruitment of teachers

Improved management information systems

Better understanding of territorial demand for education

Adequate spending per student to provide minimum inputs for each type of secondary school.

Increased number of Lower Secondary

Graduates nationally

Program Objective

G u at em a la n y ou th a re b et te r p re p ar ed f or s u cc es s in t h e la b or m ar ke t

C.3 Evaluation Questions and Proposed Methodology

C.3.1. The Evaluator may submit an individual work plan with specific deliverable milestones and potentially (though not necessarily) individual design/interim/final reports. The decision to divide/join the evaluation deliverables for the above three activities will be made in collaboration with MCC, the Education Consultant, and MINEDUC.

C.3.2. Where necessary to capture accurate measurement of results, the Independent Evaluator may propose data collection in addition to the data collection already recommended or required elsewhere in this Statement of Work.

C.3.3. The Evaluator will be responsible for implementation of data collection. However, arrangements regarding design, oversight, and management of data collection will be finalized in agreement between MCC, the Education Consultant, MINEDUC, the Implementing Entities, and the Independent Evaluator after contract award. However, MCC currently expects the Evaluator to have at a minimum a strong oversight role in MCC-funded data collection.

3 The program logic included in the RFP is the most up-to-date given the project documents. However, it is the Evaluator’s role to assess the program logic, ensure that it reflects the underlying assumptions as understood by both MCC and MINEDUC.

MCC-16-RFQ-0131 (Guatemala Education Project) Page 13

C.3.4. Within the current program logic, MCC has defined the current assumptions that underpin how MCC thinks the selected investment outputs will lead to changes in key education outcomes. The current program logic has also informed the list of key evaluation questions below. The Independent Evaluator is expected to explicitly assess the program logic, key assumptions, and evaluation questions as part of the Evaluation Design Report, recommending revisions to MCC, as necessary. MCC does not expect the Evaluator to answer every question included, but the list below provides a broad set of potential questions of interest to MCC. Part of the Evaluator’s role (see Task 1 below) is to assess the answerability of the included evaluation questions, with guidance from MCC to best target evaluation resources towards the goals of learning and accountability. The Evaluator should assess whether the proposed project design is the most effective way to improve student learning outcomes in Guatemala and if the scope of the intervention was adequate to achieve the intended results The Evaluator should propose a methodology and relevant data collection instruments to assess the impact of the intervention for the outcomes laid out in the program logic, as well as assess the project design and its effectiveness.

C.3.5. All data collection efforts, in conformity with the Evaluator’s approved Evaluation Design Report, will be the full responsibility of the Evaluator, including preparation of TORs, procurement of a data collection firm (if necessary), implementation, quality control, and analysis.

C.3.6. Education Project

C.3.6.1. Project 1, Activity 1: Quality of Education in Support of Student Success

Evaluation Questions

Process Questions

Key Assumptions Key Evaluation Questions

Certification and additional knowledge are sufficient incentives for teachers to complete the Diplomado

What effect do the teacher diagnostics have on teacher motivation?

Are the teacher diagnostics useful in training teachers and helping them reach a higher competency level?

Are the training instructors and the Ministry able to respond appropriately and with appropriate training when teacher need more support?

Did a majority of teachers complete the training?

What obstacles did teachers face in completing the training?

Teachers in public schools lack support in developing skills and mastering the national curriculum

What kinds of pedagogical support are the most important for teachers?

Do teachers and their assigned pedagogical advisors meet regularly?

Parents will have incentives to actively participate in the parent councils

Did the Parents Councils implement the Action

Plans initiated by the Activity as planned?

What were the results of the plans? How effective were they?

MCC-16-RFQ-0131 (Guatemala Education Project) Page 14

Key Assumptions Key Evaluation Questions

Lack of funds for transportation, school materials, and tutoring/mentoring programs are contributing factors in the high dropout rates

Are parent councils able to identify and successfully mitigate factors that lead students to drop out of school?

Do dropout rates decrease with additional funds from municipalities and capacity building for parent councils?

Is the additional support from local government targeted at the right families and students?

Are indigenous families represented in the parent councils?

What kind of support to the parent councils provide female students and their families to encourage those students to stay in school?

Impact Questions

A lack of training and knowledge of the national curriculum has led to low teacher performance

Has teacher competency improved as a result of the teacher training and certificate program?

After participating in the Diplomado teachers will be motivated to improve their pedagogy

Did teachers adopt new pedagogical approaches as a result of their training? This could include:

More active learning

More attention paid to different learning styles of students

Adaption of lessons for language minorities

Equal attention paid to both genders and students of all socio-economic backgrounds

The knowledge gained through the

Diplomado will help teachers better manage classroom time

Did time devoted to learning in the classroom increase as a result of the intervention?

How does this vary across different teaching subjects?

Low teacher quality is a factor in high desertion rates, low attendance, and/or low demand for education

How did teacher trainings’ improvement in teacher performance affect student enrollment in secondary schools (promotion rates, retention rates, dropout rates)?

MCC-16-RFQ-0131 (Guatemala Education Project) Page 15

Teacher training will improve the quality of schools and create a better learning environment for students

How were student learning outcomes affected by teacher trainings’ improvement in teacher performance?

How do changes in student learning outcomes vary across gender, socio-economic and language groups?

What were the reasons for these changes? Did they come as a result of the teacher training program, the parent councils, schools networks, or an interaction between all of the components?

The parent councils will contribute to better communication between teachers, students, and the broader community

How did parents’ perceptions of secondary education change as a result of broader dissemination of information to the parents of primary school students?

Do the establishment of parent councils have an impact on decreasing the dropout rate?

Proposed Evaluation Methodology

The Quality of Education Activity may be evaluable using rigorous methodology depending on the rollout strategy proposed in the design of the intervention (first cohort expected in 2017) and on the buy-in from MINEDUC and other stakeholders. Specifically, a randomized rollout or stratified random selection of Education Project beneficiaries would allow for rigorous evaluation of the changes in teaching behaviors and learning outcomes which result from the in-service professional development certificate program, pedagogical support, and creation of school networks.

The Evaluator will be expected to interface with the MCC-hired consultant for the Quality of Education Activity, and parties within MINEDUC charged with designing the Education Project, with both parties expected to work together to ensure the effective implementation of both the activity and the evaluation. The goal of this interaction will be the

Evaluator’s assessment of the evaluability of various potential implementation designs and the achievement of a final design which allows for both learning about the ideal implementation of teacher professional development activities in

Guatemala and will help the Evaluator test the underlying assumptions in the intervention design.

If an impact evaluation that utilizes a randomized control trial or quasi experimental method is not feasible, particularly if a strong buy-in from the Ministry and other local stakeholders is not achieved, the evaluation may take the form of a performance evaluation, which would follow only the beneficiaries of the project, and not seek to establish a control group. Both a performance evaluation and an impact evaluation should include rigorous use of qualitative methods, which would assess how the program was implemented and how each aspect of the activity interacted to work towards improving secondary education in Guatemala.

MCC-16-RFQ-0131 (Guatemala Education Project) Page 16

C.3.6.2. Project 1, Activity 2: Improving Technical and Vocational Education and Training (TVET)

Evaluation Questions

Process Questions

Key Assumptions Key Evaluation Questions

Improvements to TVET programs will attract more participants.

Have new and improved TVET programs been able to recruit sufficient numbers of participants?

Do a majority of the participants complete the courses and earn their certification?

Are students enrolling in TVET sectors that are thought to be meeting labor market demands?

With technical assistance TVET providers will be able to successfully engage with the private sector.

What kinds of interaction are taking place between

TVET programs and employers?

What is the frequency and the quality of these interactions?

Are they meaningful or just part of checking a box?

Once TVET providers target skills and competencies needed by the private sector, graduates of those programs will be more employable.

What are the employment rates for the graduates of the various TVET programs?

Are TVET graduates getting better paying jobs as a result of the activity compare to TVET students before the activity was implemented?

Is there a qualitative difference between graduates from the more recent MCA-funded TVET programs and the previous TVET programs? Do they have better technical skills?

Impact Questions

TVET providers in Guatemala do not currently have a good understanding of private sector labor needs. This lack of information is an obstacle in training vocational students in competitive fields.

How do private firms recruit new employees for vocational positions?

What is the industry’s involvement in the offered

TVET programs?

Do TVET programs make use of labor market data to determine course offerings?

How is labor market information obtained and used to develop TVET programs/courses or determine which TVET programs/courses to provide?

MCC-16-RFQ-0131 (Guatemala Education Project) Page 17

How do the TVET program offerings align with perceived labor market demands?

Do TVET graduates find jobs in the sector they trained in?

As a result of the activity, have our perceptions of the labor market changed? Was the data/information used to make these conclusions is accurate – based on hiring of recent graduates, and their wages/salaries?

Once TVET providers target skills and competencies needed by the private sector, graduates of those programs will be more employable.

How have employment rates been affected for the graduates of the various TVET programs?

Are TVET graduates getting better paying jobs as a result of the activity?

TVET Programs do not currently take into account gender differences, and women are underrepresented in many vocational positions.

What types of differences are observed between females and males in terms of TVET program enrollment?

Has gender-sensitive job counseling had an effect on the opportunities or perceptions of opportunities offered to females?

Are female graduates more likely to find employment in non-traditional areas?

Proposed Evaluation Methodology

The TVET activity will be evaluated through a performance evaluation, which will measure the outcomes of the beneficiaries, and monitor the reform and improvement of the TVET system in Guatemala. The Evaluator will be expected to assess the potential effects of improved TVET program offerings.

C.3.6.3. Project 1, Activity 3: Strengthening of Institutional and Planning Capacity

Evaluation Questions

There are incentives to make institutional changes, when possible improvements to the education system are identified.

Has MINEDUC adopted an improved process for competitive, merit-based recruiting of teachers?

What are the political and institutional barriers to adopting these changes?

What are the challenges involved in reforming the professional development system for teachers in

Guatemala?

MCC-16-RFQ-0131 (Guatemala Education Project) Page 18

Are there incentives that could be used to recruit and maintain higher quality teachers?

The technical assistance provided is sufficient to establish a sustainable information system.

Has MINEDUC adopted an improved management information system? If so, how has improved information influenced the institutional and planning capacity of MINEDUC? If not, then why not?

Have schools begun to use the MIS? Why or why not?

Budgeting decisions can make changes in spending per students in a relatively short timeframe.

How has secondary school funding and budgeting been influenced? Has the equity gap of spending per student throughout the secondary education system tightened – especially by modality?

Proposed Evaluation Methodology

The Evaluator will perform a performance evaluation of this activity, which assesses the degree to which the strengthened institutional capacity of MINEDUC has led to better planning and budgeting for equitable provisions of a quality secondary education, and to what extent the activity supports the broader institutional strengthening envisioned in the Plan for the Transformation of Secondary Education. The evaluation of Activity 3 should assess the success of the political and institutional reforms under the Threshold Program.4 The Evaluator will provide political economy analysis of the formal and informal decision-making that determine resource allocation for education in Guatemala as a result of Activity 3. The goal of this analysis will be to understand the political and economic factors driving decisions in the management of

Guatemala’s education system. The Evaluator will analyze how institutional planning does or does not occur in practice including budgeting, implementation, and oversight, relative to what is written or documented. The Evaluator will assess how education funding is allocated and in what priority vis a vis capital expenditures.

C.4 Roles and Responsibilities

C.4.1. Roles and Responsibilities: The MCC, government entities, and the independent evaluator, play critical roles in designing, implementing, and disseminating the independent evaluations. The MCC and government entity roles are as follows:

C.4.1.1. Millennium Challenge Corporation (MCC): MCC is responsible for oversight of the independent evaluator and quality control of evaluation activities, including the following specific responsibilities:

C.4.1.1.1. Set the evaluation priorities and evaluation questions. Using the program logic as the foundation, MCC will develop the evaluation priorities and evaluation questions based on existing evidence, the evidence gap, and the learning priorities for the sector and agency.

C.4.1.1.2. Build buy-in and ownership of the evaluation. MCC will work with the independent evaluator and MINEDUC to ensure the evaluation design follows the rules of program design and implementation. When possible, MCC will work with the independent evaluator and MINEDUC to develop a program implementation roll-out plan that enables a rigorous evaluation design. When it is not possible to develop a coordinated roll-out plan, MCC and the evaluator will ensure the evaluation design is based on feasible methodologies given program

4 A political economy analysis should explore some aspects of the way incentives shape behavior as well as the power, interests, agency, ideas, the subtleties of building and sustaining coalitions, and the role of contingency in the road sector.

MCC-16-RFQ-0131 (Guatemala Education Project) Page 19 design and implementation. MCC will work with MINEDUC to ensure commitment to the roll-out plan by local stakeholders, and if is not possible to adhere to the roll-out plan, will inform the independent evaluator in order to adapt the evaluation design.

C.4.1.1.3. Determine the evaluation budgets. MCC will develop an evaluation budget, including independent evaluators, expected data collection requirements, and any other activities related to the evaluation.

C.4.1.1.4. Manage the independent evaluators. MCC will develop Terms of Reference for the independent evaluators and manage the contracting process, including Request for Proposals, Proposal Review, selection of the independent evaluator, development of a work plan, and supervision of the independent evaluator in line with the work plan and agreed deliverables.

C.4.1.1.5. Quality review of evaluation materials. Using the program logic as a foundation, MCC is responsible for reviewing evaluation materials, including the design report and survey materials, for feasibility, technical and factual accuracy, and adherence with international standards.

C.4.1.1.6. Quality review of evaluation reports. MCC is responsible for reviewing independent evaluators’ reports to ensure technical and factual accuracy. While MCC will provide feedback, the independent evaluator is ultimately the author of the evaluation reports and may or may not take into consideration MCC’s comments. MCC therefore develops an Official Statement for each evaluation report to confirm the technical and factual accuracy of the report, or clarify any differences on technical or factual issues between MCC and the independent evaluator.

C.4.1.1.6.1. Through this review process, MCC’s role is to enable the independent evaluator’s independence in order to produce a high quality, unbiased evaluation of the program. MCC should ensure that various stakeholders understand that while MCC, MINEDUC, and other stakeholders may provide guidance and leadership on evaluation questions and feasibility of methodology, once an approved evaluation design is in place, feedback on analysis and interpretation of results is limited to statements on the continued technical and factual accuracy of the analysis.

C.4.1.1.6.2. Lead final public dissemination efforts. MCC is responsible for leading public dissemination of results, in particular ensuring a feedback loop internally in order to inform future decision makers on similar program investment, design, and implementation. This requires ensuring the evaluation results are disseminated on the MCC website, and results are disseminated to appropriate stakeholders in a timely fashion.

C.4.1.2. National Competitiveness Program (Programa Nacional de Competitividad, known as

“PRONACOM”) under the Ministry of Economy: PRONOCAM was named the “Accountable Entity” under the

Threshold Grant Agreement. The Ministry of Education (MINEDUC) will oversee and implement the Education

Project, in coordination with PRONACOM. PRONACOM, and MINEDUC are responsible for building local ownership and commitment to the evaluation, oversight of the data collection firm, and quality control of evaluation activities. Within MINEDUC the “Dirección general de evaluación e investigación educativa” (DIGEDUCA) is responsible for any data collection and analysis and will thus play a large role in shaping and carrying out the evaluation. MINEDUC, and DIGEDUCA will have the following specific responsibilities:

C.4.1.2.1. Build buy-in and ownership of the evaluation. For the Education Project PRONACOM and

MINEDUC will work with the independent evaluator and MCC to ensure the evaluation design follows the rules of program design and implementation. When possible, both entities will work with the independent evaluator and

MCC to develop a program implementation roll-out plan that enables a rigorous evaluation design. PRONACOM

MCC-16-RFQ-0131 (Guatemala Education Project) Page 20 and MINEDUC will work with MCC to ensure commitment to the roll-out plan by local stakeholders, and if it is not possible to adhere to the roll-out plan, will inform MCC in order to request the independent evaluator to adapt the evaluation design.

C.4.1.2.2. Quality review of evaluation materials. Using the program logic as a foundation, MINEDUC is responsible for reviewing evaluation materials, including the design report and survey materials, for feasibility, technical and factual accuracy, and adherence with international standards.

C.4.1.2.3. Quality review of evaluation reports. MINEDUC, or a designated representative, is responsible for reviewing independent evaluators’ reports to ensure technical and factual accuracy. While MINEDUC will provide feedback, the independent evaluator is ultimately the author of the evaluation reports and may or may not take into consideration MINEDUC’s comments. MINEDUC therefore develops an Official Statement for each evaluation report to confirm the technical and factual accuracy of the report, or clarify any differences on technical or factual issues between MINEDUC and the independent evaluator.

C.4.1.2.4. Facilitate public dissemination efforts. MINEDUC, or a designated representative, is responsible for facilitating public dissemination of results, in particular ensuring a feedback loop in-country in order to inform future decision making on similar program investment, design, and implementation.

C.4.1.3. Independent Evaluator Roles and Responsibilities: The Independent Evaluator is responsible for the overall design, implementation, and dissemination of the evaluation, including the following specific responsibilities:

C.4.1.3.1. Develop a rigorous evaluation design given rules of implementation and feasibility of options.

Using the program logic as the foundation, independent evaluators must assess the program design and implementation to develop the most rigorous evaluation design feasible, whether it is a performance or impact evaluation, and identify the most appropriate evaluation methodology feasible given the context. The independent evaluator is responsible for developing an evaluation design report, in consultation with MCC and MINEDUC, which summarizes the evaluation design, sampling, analysis plan, data collection plan, and other critical design elements of the evaluation. (Tasks 1-2)

C.4.1.3.2. Conduct Economic Analysis appropriate to the project. This can include but is not limited to

Cost-Benefit Analysis or Cost-Effectiveness Analysis. The analysis should be conducted ex-ante as well as updated after project completion and should be included in the baseline and final reports.

C.4.1.3.3. Support MCC and MINEDUC to build buy-in and ownership of evaluation. The independent evaluator shall meet with MCC and local stakeholders to develop the evaluation design based on program design and implementation. When possible, the independent evaluator will work with MCC and MINEDUC to advise on a program implementation roll-out plan that enables a rigorous evaluation design. In addition, the independent evaluator shall continuously present the evaluation objectives, materials, and interim and final results for MCC and local stakeholders to maintain commitment to the evaluation. (Task 2 and Task 7)

C.4.1.3.4. Develop evaluation materials that are held to international standards. Using the program logic as a foundation, the independent evaluators define how key outcomes will be measured and develop the survey instruments or other data collection tools in order to answer critical evaluation questions. Note that “survey” here refers primarily to quantitative surveys, but may also refer to qualitative instruments where appropriate (Task 3 and Task 8)

C.4.1.3.5. Ensure appropriate review of evaluation materials and research protocols. Independent evaluators are responsible for ensuring all evaluation materials, including the evaluation design, survey

MCC-16-RFQ-0131 (Guatemala Education Project) Page 21 instruments, sampling strategy, data collection and entry protocols are reviewed by an institutional review board

(IRB). IRB review and approval is required before any data collection commences. The independent evaluators shall document all approvals and informed consent procedures throughout the evaluation time period. (Task 4 and

Task 9)

C.4.1.3.6. Implement data collection, including the management of data collection firms. The evaluator shall develop Terms of Reference for the data collection firms, with technical support from MCC and the

MINEDUC. In this scenario, the independent evaluator is responsible for managing the contracting process, including Request for Proposals, Proposal Review, selection of the data collection firm, development of a work plan, and supervision of the data collection firm in line with the work plan and agreed deliverables. (Task 4 and

Task 9)

C.4.1.3.7. Supervise data collection and ensure quality control. The independent evaluator is responsible for quality control over data collection activities, including interviewer training, pre-testing, and implementing data quality review checks during data collection. (Task 4 and Task 9)

C.4.1.3.8. Lead data cleaning, analysis, interpretation of results. The independent evaluator is responsible for leading any necessary data cleaning and consultation with the data collection firm to produce final analysis files. The independent evaluator is responsible for leading all analysis in line with the agreed analysis plan developed with MCC, and consulting with MCC should the analysis plan need to be revised or adapted. (Task 5 and Task 10)

C.4.1.3.9. Produce evaluation reports. The independent evaluator is responsible for sharing initial evaluation reports with local stakeholders and MCC for review and feedback. However, given the independent evaluator is the author of the evaluation reports, local stakeholder and MCC feedback may or may not be taken into consideration for the final report. The independent evaluator is responsible for documenting all feedback and evaluator responses. (Task 5 and Task 10)

C.4.1.3.9.1. Through this review process, the independent evaluator’s role is to assert independence in order to produce a high quality, unbiased evaluation of the program. The independent evaluator should ensure that various stakeholders understand that while MCC, MINEDUC, and other stakeholders may provide guidance and leadership on evaluation questions and feasibility of methodology, once an approved evaluation design is in place, feedback on analysis and interpretation of results is limited to statements on the continued technical and factual accuracy of the analysis.

C.4.1.3.10. Lead public dissemination efforts. The independent evaluator is responsible for leading public dissemination efforts facilitated by MINEDUC and MCC (such as local workshops and conferences), but also additional conferences or other opportunities to publicly disseminate the results of the evaluation. The evaluator shall advise MCC on any public dissemination opportunities and collaborate as appropriate. As per Section H, the independent evaluator is responsible for ensuring that all public release documentation is reviewed to minimize any risk to respondent confidentiality. This may require submission of internal use only and public release reports.

(Task 11).

MCC-16-RFQ-0131 (Guatemala Education Project) Page 22

C.5 Tasks

C.5.1. Task 1: Assess Evaluation Plan

C.5.1.1. The Evaluator is responsible for conducting an Evaluability Assessment on all existing documentation related to the program and proposed evaluation plan provided by MCC and MINEDUC using the MCC Evaluability

Assessment Tool.

C.5.1.2. MCC and the Evaluator shall identify all MCC and MINEDUC self-evaluation activities and determine how the independent evaluation will (i) validate findings of self-evaluation(s), (ii) use self-evaluation(s) to inform the independent evaluation, particularly around understanding what was designed, what was implemented, and understanding any necessary course corrections, (iii) complement the self-evaluation(s) by expanding on self-evaluation activities.

C.5.1.3. The Evaluator shall provide an Evaluability Assessment Report on the Problem Diagnostic, Program

Logic, Risks & Assumptions, Proposed Research Questions, Methodology, and other Metrics

C.5.1.4. Develop a Work Plan with expected deadlines for deliverables. Any revisions to the Work Plan will be submitted in the Evaluator’s Progress Reports.

C.5.2. Task 2: Develop Evaluation Design Report

C.5.2.1. The Evaluator shall conduct missions to the field as needed to discuss program design and implementation issues to inform the evaluation design, as well as engage local stakeholders (MINEDUC and

Implementing Entities, as well as local government) on the proposed evaluation. This includes hosting local stakeholder workshops to develop/refine research questions, assess appropriate and feasible quantitative and qualitative evaluation methodologies, develop evaluation timeline, and build local stakeholder commitment to the evaluation design.

C.5.2.2. The Evaluator shall develop the Evaluation Design Report, including all technical analysis required for assessing and confirming internal validity, power calculations, and required sample sizes. The Evaluation Design Report outline will be developed in consultation with the MCC M&E lead. Please reference Annex C Evaluation Design Report

Outline as a reference.

C.5.2.3. As per MCC’s Gender Policy and Gender Integration…