DRAFT_iELS_PWS_(10.4.16).pdf

Version Date: October 4, 2016

PERFORMANCE WORK STATEMENT

U.S. IMPLEMENTATION OF OECD INTERNATIONAL EARLY

LEARNING STUDY PILOT

Contents

List of Exhibits

Introduction

Overview

Background

Study Elements

Timeline

Purpose of Procurement

Legislative Authority

Period of Performance

Requirements

Core Tasks

Task 1: Contract Administration (Firm-fixed Price)

1.1 Project initiation and planning

1.1.1 Post-award conference

1.1.2 Produce master schedule

1.1.3 Project Management Plan

1.1.4 Staffing Plan

1.1.5 Data Security Plan

1.1.6 Staff Security Requirements

1.1.7 Transmittals of study products and quality of deliverables

1.1.8 Study brochure

1.2 Contract management

1.2.1 Project update meetings with contractor

1.2.2 Other meetings with NCES/ED

1.2.3 Monthly study progress and financial reports

1.2.4 Project management portal

1.2.5 Maintenance of data files

1.2.6 Relationship of the contractor to NCES and other agencies

1.2.7 Readiness for work under conditions of unusual speed and flexibility

1.2.8 Quality control measures

1.3. Obtain OMB clearance for field test data collection

1.4. Obtain OMB clearance for the main study data collection

1.5. Briefings and descriptive materials

Task 2: Study Design and Instruments (Cost Reimbursable)

2.1. Field test sampling plan and sample

2.2 Main study sampling plan and sample

2.3. Adapt, prepare, and produce field test instruments

2.4. Adapt, prepare, and produce main study instruments

Task 3: Data Collection (Cost-reimbursable)

3.1. Recruit Schools and Students for Field Test

3.2. Recruit Schools and Students for Main Study

3.3 Recruit and train data collectors for field test

3.4. Recruit and train data collectors for main study

3.5. Collect data for field test

3.6. Collect data for main assessment

3.7. Produce field test dataset

3.8. Produce main assessment dataset

3.9. Score field test assessment and code questionnaire data

3.10. Score main assessment and code questionnaire data

3.11. Field test data review memo

Task 4. Prepare and Disseminate Results (Firm-fixed Price)

4.1 Survey operations/technical report

4.2 Non-response bias analysis report

4.3. Develop and Produce iELS U.S. national report

4.3.1. National report analysis plan and table/figure shells

4.3.2 Produce national report

4.4. Produce main study electronic public-use and restricted-use datafiles and user’s guides

Optional Tasks

Task 5 (Optional): Use Expert Help (Cost-reimbursable)

Task 6 (Optional): Additional Sample (Cost Reimbursable)

Task 7 (Optional): Additional Dissemination of Results (Firm-fixed Price)

Task 8 (Optional): Parent Instrument in Spanish (Cost-reimbursable)

8.1 Field test administration of Spanish-language parent instrument

8.2 Main study administration of Spanish-language parent instrument

Task 9 (Optional): iELS school-level reports (Firm Fixed Price)

Task 10 (Optional): Lessons learned and improvement recommendations for iELS-2 development (Cost-reimbursable)

Appendix A. Tentative Timeline for iELS

Appendix B. Data Confidentiality, Security, Quality Control, and Contractor Staff Security Clearances 58

B-1. Overview of IES/NCES Confidentiality Statutes Confidentiality of Individuals and Institutions Protection and Security of Data

B-2. Data Security Plan Definition of Personally Identifiable Information and Direct Identifiers Development of Data Security Plan Implementation of Data Security Plan

B-3. Adherence to NCES Statistical Standards NCES Statistical Standards

B-4. Quality Control Measures Requirement Quality Control Measures

B-5. Contractor Employee Security Screening Contractor Employee Security Screening Requirements

B-6. Requirement of Affidavit of Nondisclosure Affidavit of Nondisclosure

B-7. Security and Confidentiality Training for Contractor Staff Security and Confidentiality Training

B-8 Information Collection Request (ICR) Requirement

Appendix C. Disclosure Analysis Plan

List of Exhibits Exhibit 1

International Early Learning Study Pilot

(IESE16014AP)

Introduction

The National Center for Education Statistics (NCES) in the Institute of Education Sciences

(IES) of the U.S. Department of Education (ED) supports a variety of international activities to collect and report data on education in the United States and other countries. It is the

NCES’ goal to provide valuable comparative information about the U.S. educational performance in an international context. NCES participates in international student assessments in order to:

provide comparable indicators on student performance and schooling practices across countries in order to benchmark U.S. student performance; and suggest hypotheses about the relationship between student performance and factors that may influence performance and about areas where students have particular strengths or weaknesses.

The study covered by this procurement is part of the larger international program that NCES has participated in through collaboration with a variety of international organizations and partners, such as the Organization for Economic Cooperation and Development (OECD), the

International Association for the Evaluation of Educational Achievement (IEA), and ministries of education and other government agencies throughout the world.

Through its participation, NCES seeks to strengthen the quality, consistency and timeliness of international data. To continue this effort, the United States must follow through with well-organized and well-executed data gathering activities within our national boundaries. These efforts will allow NCES to provide policy-relevant data for comparisons of the U.S. education system with education systems in other countries, and to help inform decision making on the part of national, state, and local policy makers.

This Performance Work Statement (PWS) is written in three sections. The first section provides an overview of the International Early Learning Study Pilot (hereafter referred to as iELS), including its purpose, its enabling legislation, and general parameters of the work to be done under this contract. The second section specifies the core tasks, subtasks, and deliverables associated with this work. The third section describes a set of optional tasks and subtasks.

Overview

Background

The iELS is a new initiative of the Organization for Economic Cooperation and Development

(OECD), an intergovernmental organization of industrialized countries. The OECD also coordinates the Program for International Student Assessment (PISA), an assessment of 15-year-olds; the Program for the International Assessment of Adult Skills (PIAAC); and the

Teaching and Learning International Survey (TALIS). The iELS, sometimes referred to as the

Child Well-being and Early Learning Study or Pilot, or sometimes as the Early Learning

Outcomes Study, has its origins in the work of the OECD’s Early Childhood Education

Network, which has undertaken work on assessing quality in early childhood education. The idea of measuring child outcomes across countries was raised during the Network’s 2012 priority rating exercise. During 2015, the OECD Secretariat developed a conceptual framework on measuring early learning outcomes, in collaboration with interested countries.

This framework has informed countries’ efforts to develop a study that would address child outcomes at the end of early childhood education, as children are transitioning to primary school. More specifically, countries’ objectives for the study are to better understand:

Children’s early learning and development in a broad range of domains, including social and emotional skills as well as cognitive skills

The relationship between children’s early learning and children’s participation in early childhood education and care (ECEC)

The role of contextual factors, including children’s individual characteristics and their home backgrounds and experiences.

The study has a system-level focus and will not allow the identification of individual institutions, staff, children or parents.

Large-scale international, comparative assessments of school-aged children and adults have been successfully conducted for decades. The OECD’s PISA and PIAAC assessments are two examples. Others include the Trends in International Mathematics and Science (TIMSS) and the Progress in International Reading Literacy Study (PIRLS), which assess children in grades

4 (TIMSS and PIRLS) and 8 (TIMSS). Although there are examples of national assessments of young children (e.g., NCES’ Early Childhood Longitudinal Study), and there are assessment instruments that have been validated for use across multiple countries and languages, there is little experience with cross-national assessments of younger children for the purpose of providing comparative data such as that yielded by PISA, TIMSS and PIRLS.

While the study will build on the experiences of other large-scale international assessments, a large body research on young children, and experience measuring skills in young children, this is a new study without an established set of instruments, protocols, and standards. It is a pilot effort among 3-6 countries that will provide a model that can be expanded upon in future years.

The iELS is managed by the OECD with guidance from participating countries (Early

Learning Group of Countries) that meet regularly to provide input on the direction of the study. The iELS will be implemented internationally by organizations contracted by the

OECD. The OECD has overall managerial responsible for iELS internationally, including overseeing contractors that implement the international activities. The OECD is in the process of contracting with one or more organizations (hereafter referred to as the international contractor” or “international contractors”) to carry out iELS development, implementation, analysis, and overall coordination across countries. In this PWS, all mention of relevant international contractors, organizations, or sponsors refer to these institutions. As in other

OECD large-scale assessments like PISA and PIAAC, each country will appoint a National

Project Manager (in the United States this is a representative from NCES) to manage the study in the country. The OECD and international contractors will provide countries with study instruments and a series of manuals and technical standards that document the procedures and requirements of the study, which will be available to the contractor.

Study Elements

Although design specifications, instruments, protocols, and standards are still to be developed, a study framework developed in 2015 provides an overall picture of the direction of the study, based on the interests of the countries involved. The following is adapted from that conceptual framework and updated with additional information.

Age range

Countries that have been involved in scoping this study have expressed a clear preference for an age-based rather than stage-based target population, to support valid international comparisons of children’s early learning outcomes. Countries have agreed that the assessment should occur at the point where “nearly all” children are in some form of education or care provision. Countries have agreed on an age band between 4.5 to 5.5 years, although not all countries will be able to assess children across this entire age band. For the United States, the expectation is that children enrolled in Kindergarten will be assessed near the beginning of the Kindergarten school year (on average, 5.5 years old). This is the point at which it is feasible to reach at least 95 percent of children through a school-based sampling approach and this is an important transition point for children.

It is expected that as part of this pilot the international contractor will develop a methodology based on age adjustments and weightings to ensure comparability and validity of the data across countries.

Domains

It is expected that the domains to be assessed will represent a balance of both cognitive and social-emotional skills that, as a package, will provide coherent and reliable insights into children’s early learning. Six possible domains were identified as having high potential, based on an analysis of early skills that are predictive of positive life outcomes and through consultation with interested countries. The specific list of domains to be included in the study may be a subset of these six, or may include some of these and alternative domains.

Language/emergent literacy

Mathematics/numeracy

Executive function

Social skills

Physical skills (e.g., fine motor)

Approaches to learning

It is expected that the iELS will include direct and indirect assessments of children’s skills.

Contextual information

Contextual information provides insights into the relationships between children’s learning and development and important demographic, social, economic and educational variables.

Although the specific contextual data to be collected and methods to be used are still to be determined, it is expected at this time that data on the following categories will be collected from teachers and parents: ECEC participation, home learning environment, and individual characteristics.

ECEC Participation

Information on the child’s ECEC history could include:

The age of entry to ECEC

The intensity of the child’s ECEC participation, eg part-time, full-time

The duration of ECEC participation, eg number of months

Continuity, ie the number and duration of breaks between periods of participation, and the number of different providers/settings the child has attended, and

Type of provision, such as the type of setting and pedagogical approach.

Home learning environment

Aspects of the home learning environment that are of particular interest are:

The learning and development-related activities parents undertake with their children and the frequency of these activities. Such activities are likely to include some or all of the following: reading books, singing, telling stories, playing games, doing puzzles and doing art and crafts

The extent to which parents are actively engaged in their child’s ECEC experiences, where the child is participating in ECEC. This includes the extent to which parents and

ECEC staff share information and strategies to support the child’s learning and development.

Individual characteristics

The aspects of children’s individual characteristics to be collected relate to each child’s:

Gender

Ethnicity

Linguistic background

Migration history and status

Socio-economic and family background

Disabilities and special learning needs.

Timeline

Although the specific timeline of the study is still to be determined, it is expected that participating countries in the northern hemisphere will field-test instruments and procedures in 2017 (October-November 2017 in the United States) and in the southern hemisphere in

April 2018. The operational data collection will be conducted in 2018 in the northern hemisphere (October-November 2018 in the United States) and in 2019 in the southern hemisphere. Results will be reported in early 2020. Appendix A includes a tentative timeline of key milestones for the study.

Purpose of Procurement

NCES requires a contract to implement the iELS in the United States. The overall goal for the procurement is data collection in the United States that meets all international and NCES guidelines and that makes available on a timely basis high-quality internationally comparative assessment data. Furthermore, the study shall result in a report presenting U.S. data and the international comparisons from the perspective of an American constituency. The core contract is for the U.S. data collection for a nationally-representative sample of children meeting the international requirements for the target population.

The iELS will follow the established design and standards set forth by the OECD, the Early

Learning Group of Countries, and the OECD’s contractor(s). The design and standards will cover, but are not limited to, administration procedures, sampling guidelines and requirements, survey instruments, data collection, and data coding and processing. The OECD and international contractors also establish the overall study schedule and key deadlines to which the U.S. data collection activities must adhere. The OECD and international contractors provide manuals, guidelines, and technical standards that stipulate the study procedures and standards. The contractor shall follow the international guidelines for the iELS and shall meet established NCES standards and procedures (NCES standards can be found at http://nces.ed.gov/statprog/).

The contractor shall be clear that NCES is the sole authority to manage the iELS in the United

States and to negotiate with the OECD or international consortium on any issues related to the study. The contractor shall not use the study instruments or any part of them for any other purposes without NCES’ written consent. Under no circumstances shall the contractor conduct any activities related to iELS, including conducting the assessment in sub-national jurisdictions (e.g., states or school districts), without permission from NCES.

Legislative Authority

The authority for this study can be found in NCES’ longstanding mission to “collect, and analyze and disseminate statistics and other information related to education in the United

States and in other nations” (stated in section 406 of the General Education Provisions Act, as amended (20 U.S.C. 1221e-1) and in the Education Sciences Reform Act of 2002 (P.L. 279-

107, Part C, Sec. 153):

(a) GENERAL DUTIES.-The Statistics Center shall collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including…(6) acquiring and disseminating data on educational activities and student achievement (such as the Third International Math and Science Study) in the

United States compared with foreign nations.

(See http://www.ed.gov/policy/rschstat/leg/PL107-279.pdf for the full Education Sciences

Reform Act.)

Period of Performance

The anticipated period of performance for this contract is sixty (60) months.

Requirements

Independently and not as an agent of the United States Government, the contractor shall provide all personnel, materials, services, and facilities necessary for the project and perform the tasks as described below.

The contract includes (1) preparing and conducting a field test of the iELS instruments, (2) preparing and conducting the main study or main assessment for iELS, and (3) producing http://nces.ed.gov/statprog/ http://www.ed.gov/policy/rschstat/leg/PL107-279.pdf public- and restricted-use release files, data documentation, and analytic and methodology reports. It may also include various optional tasks, such as assessing an expanded sample, administration of the parent instrument in Spanish, and dissemination.

A preliminary timeline for iELS, with major events and activities, is in Appendix A. The contractor shall note that the timelines are subject to change. Reference below to “the study” or “the project” means the iELS (and may include the optional tasks). Due to the timing of the iELS, some activities at the international level may have begun by the time the contractor begins work.

Core Tasks

Task 1: Contract Administration (Firm-fixed Price)

The contractor shall work closely with the Contracting Officer’s Representative (COR) to implement the design of the study included under this procurement and the approach presented in the proposal.

1.1 Project initiation and planning

Within the first week of the start of the contract, the contractor’s project director(s) and key staff will have reviewed the known technical methods and procedures of iELS (to be provided by NCES when available), as well as the international operations and production of past administrations of PISA, which may serve as a broad model for some of the procedures and requirements of the iELS. Appendix A of this PWS also provides tentative scheduling information about the iELS.

1.1.1 Post-award conference

Within 2 weeks of the contract award, as approved by the COR, the contractor’s project director and other key staff members who would be able to answer questions about sampling issues, data collection processes, costs, etc., from the contractor project team shall hold a meeting in Washington D.C. at facilities made available by ED. The meeting will include the

COR and key NCES project staff and the ED Contracting Officer (CO) and Contracting

Specialist (CS). The contractor shall focus the meeting to review the overall contract tasks, identify potential problems and possible solutions in the project and contract, and discuss areas of concern related to the proposed project staffing plan and other management requirements. The primary purpose of this meeting is to refine the management, staffing, and scheduling plans. These refinements shall provide management information for use by both the contractor and the government in monitoring the work to be performed. This meeting shall also provide the opportunity for ED staff to share project experience and materials with the contractor. The contractor shall prepare and bring all necessary materials for a productive and successful initial meeting (1.1.1).

Deliverables: 1.1.1 Post-award meeting materials.

1.1.2 Produce master schedule

The contractor shall produce a project schedule that details key milestones; critical tentative dates specified in appendix A and the “Schedule of Deliverables (SOD)” and the contractor’s production deadlines for the Office of Management and Budget (OMB) clearance, instrument revision and adaptation, assessment administrator training, data collection, scoring/coding, datafile preparation and submission, analysis and reporting. The project schedule shall include the following information in an easy-to-understand format:

Start and end dates of all tasks and subtasks;

Key milestone dates;

Completion dates of subtasks, drafts, or other events;

Key deliverable dates.

The master schedule shall make clear the primary leads and contact person for each task, subtask, or deliverable (as applicable).

The contractor shall submit a revised master schedule for the field test within 2 weeks of contract award (1.1.2), and thereafter submit a revised master schedule to the COR at each update meeting (see subtask 1.2.1). The revised schedule and all schedules going forward must reflect the international contractor milestones and deliverables, which are subject to change throughout the course of the study. The contractor shall submit a revised master schedule for the main study no later than 7 months prior to contacting schools for recruiting.

The contractor shall be prepared to update the schedule within 24 hours of a request for it by the COR.

Deliverables: 1.1.2.1 Revised master schedule for field test;

1.1.2.2 Revised master schedule for main study

1.1.3 Project Management Plan

The project management plan shall detail the management techniques that will be employed to ensure timely, efficient, and cost-effective instrument preparation, data collection, processing, analysis, and reporting. The plan shall reflect the requirements specified in the contract or describe and justify any revisions to the original plan specified in the contract and approved by NCES. These might include changes in data collection approach, data processing, plans for analysis and reporting, as well as changes in the proposed project organization, staffing plan, and schedule. The plan will fully describe the rationale for any proposed changes.

The contractor shall prepare and submit the project management plan to the COR within 3 weeks of contract award (1.1.3.1). The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the project management plan is approved by the COR, the contractor shall manage the project in accordance with the approved plan (1.1.3.2) and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

Deliverables: 1.1.3.1 Draft project management plan;

1.1.3.2 Final project management plan

1.1.4 Staffing Plan

The staffing plan shall detail the project’s overall staffing, including contractors and subcontractor staff dedicated to the study, as well as the training or preparatory activities to ensure that all staff are familiar with and abide by the requirements of the study (including, but not limited to, the study’s need for high quality and timely completion of all activities and for strictly enforced confidentiality procedures).

The ability of the COR to obtain timely and accurate information and decisions from contractor staff is considered of the utmost importance, and will remain a high priority for the contractor. The contractor shall be responsible for the timely and satisfactory completion of work carried out by subcontractors, and shall convey to subcontractors the need to comply with the rules, regulations, guidelines, standards, timelines, and deliverable schedules of all tasks and sub-tasks included under this procurement.

The contractor shall prepare and submit a staffing plan to the COR within 3 weeks of contract award (1.1.4.1). The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the

COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the staffing plan is approved by the COR, the contractor shall implement the plan (1.1.4.2) and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

Deliverables: 1.1.4.1 Draft staffing plan;

1.1.4.2 Final staffing plan

1.1.5 Data Security Plan

NCES is required by law to develop and enforce standards designed to protect the confidentiality of individually identifiable respondents, including students and schools. These requirements cover the collection, reporting, and publication of data; they are described in detail in appendix B. The contractor shall ensure that all contractor-led operations, including those of subcontractors, adhere to federal laws and standards in this regard.

To describe how the contractor will do this, the contractor shall prepare and submit a draft data security plan to the COR within 3 weeks of contract award (1.1.5.1). The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the data security plan is approved by the

COR, the contractor shall implement the plan (1.1.5.2) and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

The security of this study’s data is of utmost importance to NCES and the security plan will be reviewed by NCES’ Chief Statistician. For a detailed discussion of the data security requirements of the contract, see appendix B.

Deliverables: 1.1.5.1 Draft data security plan;

1.1.5.2 Final data security plan

1.1.6 Staff Security Requirements

All contractor staff to work on the project for 30 days or more must complete a series of security forms and undergo a federal background check per OMB requirements, with higher levels of security clearance required for those who visit schools or who may come in contact with confidential data. The contractor shall ensure that all staff who may come in contact with confidential data shall (a) be trained in the proper handling of confidential data and (b) first sign and then notarize an affidavit of nondisclosure before having access to any data with confidential information. The contractor shall execute these affidavits of nondisclosure and retain them in a secure location. The contractor shall provide a list to the COR of all the individuals who have signed affidavits, the position in the organization of the person, and the person’s relationship to the study, within 4 weeks of the award of the contract (1.1.6).

As new persons are assigned to the project, the security forms shall be filled out and an affidavit shall be executed for them on the first working day of assignment to the study and an updated list provided to the COR, no less than on a quarterly basis. The contractor shall notify the COR if any contractor staff leave the project. The contractor shall be aware of and comply with all contractor personnel security regulations and all Department IT security policy requirements, as described in Appendix B.

Deliverables: 1.1.6 List of all personnel with signed Affidavits of Nondisclosure

1.1.7 Transmittals of study products and quality of deliverables

The contractor shall transfer and retrieve study-related documents and files through the

Internet, to or from NCES, or other designated contractors. The ability to transfer and retrieve documents through the Internet is important to maintaining the integrity of electronic versions of documents that may eventually be posted on the NCES web site or distributed internally.

Thus, it is imperative that the contractor works closely with NCES web site personnel to set up and ensure that all transfer protocols are followed and that transfer and retrieval of documents (especially from the contractor’s management website, see subtask 1.2.4, and the

NCES secure data server on the NCES Member’s website) is completed to the satisfaction of

NCES. The contractor shall conduct a test of its transfer/retrieval system(s), within 4 weeks of the award of the contract and prior to making use of the system for key contract deliverables, and shall eliminate any problems found to ensure a trouble-free system (1.1.7).

For the contract, the contractor shall transmit all correspondence directly to the NCES COR via e-mail that is compatible with Microsoft Outlook. Compatibility will extend to the ability to access attachments sent as part of e-mails. As of 2016, NCES uses Microsoft Office 2010:

Word, PowerPoint, and Excel as its primary software packages for communicating doc-uments, charts, and tables. NCES uses the Department of Education’s Microsoft Outlook software as its communications package. All deliverables are to be delivered in native file format, in the version of Word, PowerPoint, Excel, or other software packages used by NCES.

All final deliverables that are to be assigned a NCES publication number or that will be made public through the NCES website will be delivered in PDF format (or another appropriate format approved by the COR) in accordance with the NCES requirements published in

“Policies, Guidelines, Standards and Procedures for Web Publishing for IES and NCES”

(available at https://members.nces.ed.gov/ webstandards/), and must adhere to Section 508 requirements for accessibility. More information on accessibility is available through the

Federal IT Accessibility Initiative at: http://www.section508.gov.

Although the term “draft” is used throughout this PWS, it is important that the contractor understand what the Government means, and does not mean, by “draft.” A draft is a complete product of high quality that the contractor (and the government) can publish with confidence as to its accuracy and usefulness. A “draft” is not simply what is available on the due date— regardless of completeness or quality. NCES anticipates that only minor changes to drafts will be needed for them to become final deliverables. If the contractor has a subcontractor who is preparing deliverables, the prime contractor shall review and stand by the subcontractor’s work.

Every final deliverable will be sent with a cover memorandum signed or initialed by the project director or someone of higher corporate stature than the project director. The cover memorandum will certify that the contractor’s organization stands behind the quality of the product. This holds for all final versions of products—even those prepared primarily by consultants or subcontractors. At no time will the government take direct delivery of a product from a consultant or subcontractor.

All draft deliverables will be due at the office of the COR and the CS on or before their due date. Deliverables must be sent in both electronic copy and, if requested by the COR, hard copy format. Hard copies of deliverables must be sent to the COR such that they arrive on or before their due date. Unless otherwise specified, three copies of each hard copy deliverable will be required.

Deliverables: 1.1.7 Test of transfer/retrieval system(s)

1.1.8 Study brochure

The contractor shall assist NCES in developing an informational brochure for the study that will describe the study’s general goals. At a minimum, it will identify the purpose of the project, the sponsoring organizations, the types of data to be collected, and the data collection schedule. The brochures will be distributed to individuals and organizations requesting information about the study and will be included as part of the information packet used to secure cooperation of study participants. The brochure should be visually appealing through the use of bright colors, photos, and other graphics as appropriate. The contractor shall also make the content of the brochure available in a format that can be posted on the NCES website.

The contractor shall submit a draft of the brochure 6 weeks after contract award (1.1.8.1).

The COR will review the drafts and provide comments and suggestions within 2 weeks.

NCES will provide the contractor with an electronic version of the logo of the study, which the contractor shall use in the production of the brochure and other published materials.

Based on initial COR comments, the contractor shall revise the draft of the brochure and will resubmit it for full NCES review (see Exhibit 1, under Task 4, for a description of the NCES review process). Once approved, an electronic version of the brochure will be delivered within 3 weeks, along with the final print copies, for inclusion on the NCES web site https://members.nces.ed.gov/%20webstandards/ http://www.section508.gov/

(1.1.8.2). The electronic version of the brochure will meet NCES web site design and formatting guidelines, which currently requires PDF and HTML formatting, with graphics as

.jpg or .gif files. The contractor shall produce enough copies of the final, printed brochure for all information packets needed to secure cooperation of study participants and deliver an additional 50 copies to the COR for NCES.

Deliverables: 1.1.8.1 Draft informational brochure;

1.1.8.2 Final informational brochure

1.2 Contract management

Because this contract includes a series of time-sensitive and complex activities, it is critical that it be monitored and managed effectively. To that end, NCES requests the following general study management activities.

1.2.1 Project update meetings with contractor

For the life of the contract, the contractor will hold regular weekly or biweekly (or less frequently, if the COR so determines) meetings for key contractor staff to update the COR or other NCES staff on the project’s progress. The COR may propose to hold these meetings via telephone conference calls, over the Internet, or face-to-face if the meeting is critical. For budget purposes, the contractor shall plan up to 40 meetings a year throughout the contract, of which up to 12 meetings a year may be face-to-face. The contractor shall participate in the development of the goals and agenda for each meeting. The contractor shall be responsible for distributing the agenda and other meeting materials, including the current version of the project’s master schedule, to no more than 12 meeting attendees. Materials to be distributed will be approved, in advance, by the COR.

Deliverables: 1.2.1 Regular project meetings materials including master schedule

1.2.2 Other meetings with NCES/ED

The contractor shall send one representative to all National Project Manager (NPM) meetings for the study; there are typically two each year. The contractor shall send the Project Director to the meetings, or the staff member who has primary responsibility for conduct of the study.

Additional personnel may be sent, if needed, upon approval of the COR. For costing purposes, the contractor should assume 5 international NPM meetings in a foreign location over the course of the contract. For each NPM meeting attended, the contractor shall submit to the COR and CO a trip report that documents the purpose and outcomes of the meeting, along with any follow-up actions required, within 3 business days after the end of the meeting. The contractor will work with the COR to develop a common format to be used for such trip reports for the first NPM meeting attended by the contractor.

Deliverables: 1.2.2 NPM meeting trip reports

1.2.3 Monthly study progress and financial reports

The contractor shall submit to the COR and CS a regular monthly progress report, by the 15th day of the month following the month the report covers. This report shall be delivered before or concurrent with the contractor’s monthly billed invoice so that it is available to check and approve each month’s invoice. At a minimum, in each monthly progress report, the contractor shall include information on the following aspects of the study and its operation:

The past month’s work, including a description of continuing work, new work, and completed work, with key dates on which tasks or activities were initiated or completed during the month;

The upcoming month’s work, including a description of the work and key dates by which time tasks or activities will be initiated or completed during the month;

Major accomplishments for the month;

List of products and milestones submitted or achieved during month, with a short description of each; and

Problems/Issues and how each has been or will be addressed by the contractor.

The contractor shall also submit to the COR and CS a regular monthly financial data report by the 15th day of the month following the month the report covers. This report shall be delivered before or concurrent with the contractor’s monthly billed invoice so that it is available to check and approve each month’s invoice. In each monthly financial data report, the contractor shall, at a minimum, include

(1) information on the following aspects of the study and its operation:

Past month’s cost of work completed, by task, subtask, and cumulatively;

Percent of contracted amount spent to date, by task;

Estimated cost to complete, total estimated cost, and total contracted cost;

Difference between the total estimated cost and the contracted cost; and

Staff hours spent on each task during the prior month;

(2) appropriate records and information to permit the COR and CS to certify that the services that are listed on monthly bills are actually used and are for official purposes; and

(3) a manpower report prepared and signed by the project director that summarizes actual personnel assignments for the month just completed, showing for each named individual the hours charged, by task. For individuals not named in the contract, the manpower report will show hours spent by position (e.g., data collectors). All hours billed to the contract will be summarized in the report.

The contractor shall include detailed information in the monthly financial data reports on any anticipated or actual cost overruns or cost savings, whether incurred within a single task or cumulatively. From this information, the contractor shall ensure that the COR is clear about the progress being made on tasks, the work that remains to be accomplished, the financial resources used and available, and any problems and proposed solutions. At the request of the

CO or CS, if additional issues need to be addressed in either monthly report, the contractor shall submit an addendum within 2 days of the request.

The format for the monthly report will be consistent from month to month. The same format will be used to report on any subcontractor activity costs.

The monthly report shall serve as a justification for all work billed for the month. Therefore, the contractor shall take care to ensure not only that all work billed for the month is described in sufficient detail in the monthly report to support an audit, but that all detailed information

(on completed work) contained in the monthly report is transparently and easily associated with its respective cost or billing. The monthly progress and financial reports shall be submitted together each month and the contractor shall respond to questions regarding either document within 1 day.

Deliverables: 1.2.3 Monthly study progress and financial reports

1.2.4 Project management portal

No more than 1 month after the start of the project, the contractor shall establish, and thereafter maintain, space on an internal server to house all project reports, interim and final versions of deliverables, and other appropriate project documentation. The space shall be accessible to the COR and other appropriate contractor or outside staff. This will aid in ensuring that all project staff have the most up-to-date versions of materials and will complement (but not replicate) websites maintained by the OECD and International

Contractors. The space shall be password protected with access limited to NCES, contractor staff, and key subcontractor staff as appropriate. No data collected from the respondents shall be placed on the project management portal.

Deliverables: 1.2.4 Project portal for housing electronic copies of project materials

1.2.5 Maintenance of data files

The contractor shall maintain complete data files for all components of iELS, and all supplemented files created and obtained in support of the study, for the life of the contract.

Maintenance of data files means keeping data file documentation available for reproduction on request of the COR or other authorized individuals. Certain data files that will be maintained will not be made available to the public because of privacy considerations. In these cases, file maintenance means that the data files will be kept intact, in a form that can be updated or copied. Data files will not be allowed to expire, to be released, overwritten, or otherwise destroyed. These files will be maintained with a software system capable of interlinking them for analysis (or other) purposes while, at the same time, preserving the necessary confidentiality requirements.

1.2.6 Relationship of the contractor to NCES and other agencies

The contractor shall report all activities for this contract and activities related to iELS to the

COR. All data, publications, and products of the study are the sole property of the U.S.

government, and shall be provided to it as specified below and upon request. Under no circumstances will the contractor, or any subcontractors, distribute or release any data, publications, or products of the study in any form to any individual or party without prior permission of the COR. Any contractor or subcontractor conference presentations or papers, based on this study, shall be reviewed and cleared by NCES prior to any such presentation.

This restriction will remain in effect throughout the life of the contract, and until NCES officially releases the data, publications, or products to the public.

The contractor shall work through the COR in all relationships and discussions with the

OECD, international contractors, foreign governments, foreign education professionals, and all other agencies and individuals related to the study. The contractor shall not convey to any of these entities information that may be construed as NCES, IES, or Department of

Education policy, except where related to technical aspects of the study or where given explicit permission by the COR.

1.2.7 Readiness for work under conditions of unusual speed and flexibility

In the event that the study’s international plans and time schedules change rapidly due to various necessities, the contractor shall be prepared to work with changing requirements. For example, time schedules may become very tight because the OECD or its contractors are unable to meet their projected study timelines. Whenever necessary, the contractor shall be prepared to perform activities under conditions of emergency speed. Emergency speed will be defined according to four priority levels:

Priority 1: within 24 hours of request

Priority 2: within 72 hours of request

Priority 3: within 5 working days of request

Priority 4: within 10 working days of request

In the event that changes to plans or time schedules require the completion of a task or activity under emergency speed, the COR will notify the contractor of the request, after consultation with the CO, and identify the timeframe (i.e., priority level) under which the task or activity will be completed by the contractor.

1.2.8 Quality control measures

NCES is firmly committed to determining how well each project meets its goals, the quality of data collected (including response rates at the school and the student levels), and the implications of the experiences and outcomes of the project for future projects. Project quality control should be integrated into every phase of the project. For instance, potential problems should be identified early so that possible solutions are tested as part of the field test. Item reliability and validity tests for questionnaires and assessments shall be built into the data collection efforts, and results from these tests shall inform data collection and processing.

Nonresponse analyses shall be conducted as part of the weighting process, and information about the survey methodology and sample design effects shall be made available to data users.

Quality control also includes (1) managing staff in the field to ensure the high quality of their performance; (2) instituting procedures relevant to confidentiality and protection of data

(including the secure transmittal of confidential data); and (3) ensuring deliverables are submitted on time and are of sufficient quality (in terms of accuracy, coherence, clarity, and comprehensiveness) to meet NCES statistical standards, OECD/iELS standards, professional standards, and NCES and IES style guidelines. As noted earlier, NCES expects that only minor changes to drafts will be needed for products to become final deliverables.

Again, all work conducted under this contract shall, at a minimum, meet the standards and guidelines set forth in the publication NCES Statistical Standards, and amendments implemented since that time. The standards can be accessed at http://nces.ed.gov/statprog/.

The IES guidelines can be accessed at http://nces.ed.gov/statprog/style.asp.

http://nces.ed.gov/statprog/style.asp

1.3. Obtain OMB clearance for field test data collection

The contractor shall prepare a clearance package for obtaining OMB approval of the iELS field test and non-cognitive survey instruments that will be used in the study, and any other materials that will be distributed to districts, schools students, and parents (e.g., parent permission letters), in accordance with current OMB, ED, and NCES requirements for such packages.

Preparing an OMB clearance package entails assembling an extensive amount of information to (a) justify the data collection as a whole as well as individual background questions; (b) estimate the cost of the data collection to the federal government and the amount of burden (in both time and cost) on respondents; and (c) explain the data’s uses, efforts taken to minimize the burden on respondents, any provisions for payments or gifts to respondents, steps to assure confidentiality, and the statistical methods that are to be employed in collecting and preparing the data (e.g., dealing with nonresponse). OMB requires that all this information be written up in a Supporting Statement. An OMB clearance package for submission must include the Supporting Statement, copies of all materials that will go to respondents (such as recruitment materials, survey questionnaire, letters of consent, etc.), and information on any national adaptations of the instruments (see subtask 2.3). The contractor shall prepare a draft

OMB clearance package for the field test no later than six months prior to beginning school recruitment, and preferably earlier.

After COR review and internal NCES review, the contractor shall revise the draft OMB clearance package within 5 business days of receipt of comments. NCES will submit the revised OMB clearance packages to ED and OMB for review and approval. Failure to submit the OMB clearance package by the deadline may result in the contractor being unable to collect data in a timely manner. The contractor shall be aware that the time between submission of the OMB clearance package and receipt of a clearance number from

OMB for inclusion on all survey documents takes a minimum of four months, and upwards of six months, sometimes more. The contractor shall take these time constraints into consideration when preparing and submitting the OMB clearance package. The contractor shall work diligently and expeditiously with the COR to seek OMB clearance prior to recruiting schools.

In preparing the OMB clearance package, the contractor shall follow instructions for completing the Information Collection Submission Worksheet and writing the Supporting

Statement required for Paperwork Reduction Act submissions (for information on the requirements for a clearance package, officially called at present an Information Collection

Review, see http://www.whitehouse.gov/omb/inforeg_infocoll/). The contractor shall prepare all written materials needed in the formal clearance and approval process according to OMB requirements as outlined in the Information Collection Submission Worksheet. The package will include the U.S.-adapted background questionnaires but not the cognitive (assessment) instruments for the study. This approval process is expected to involve the following series of steps (called out as deliverables):

Contractor shall provide a draft package to the COR and shall revise it within 5 business days of receiving the COR’s comments (1.3.1).

http:…