P15 - PWS_041212.pdf

DRAFT

PERFORMANCE WORK STATEMENT

U.S. PARTICIPATION IN THE PROGRAM FOR INTERNATIONAL

STUDENT ASSESSMENT (PISA) 2015

Contents

List of Exhibits

Introduction

Background on PISA 2015

Purpose of Procurement

Enabling Legislation

Work Statement

Task 1: Contract Administration (Firm-fixed Price)

1.1 Project initiation and planning

1.1.1 Post-award conference

1.1.2 Produce master schedule

1.1.3 Project Management Plan

1.1.4 Staffing Plan

1.1.5 Data Security Plan

1.1.6 Staff Security Requirements

1.1.7 Transmittals of study products and quality of deliverables

1.1.8 Study brochure

1.2 Contract management

1.2.1 Project update meetings with contractor

1.2.2 Other meetings with NCES/ED

1.2.3 Monthly study progress and financial reports

1.2.4 Project management portal

1.2.5 Maintenance of data files

1.2.6 Relationship of the contractor to NCES and other agencies

1.2.7 Readiness for work under conditions of unusual speed and flexibility

1.2.8 Quality control measures

1.3. Obtain OMB clearance for field test data collection

1.4. Obtain OMB clearance for the main study data collection

1.5. Briefings and descriptive materials

Task 2: Study Design and Instruments (Firm-fixed Price)

2.1. Field test sampling plan and sample

2.2 Main study sampling plan and sample

2.3. Adapt, prepare, and produce field test instruments

2.4. Adapt, prepare, and produce main study instruments

Task 3: Data Collection (Cost-reimbursable)

3.1. Recruit Schools and Students for Field Test

3.2. Recruit Schools and Students for Main Study

3.3 Recruit and train data collectors for field test

3.4. Recruit and train data collectors for main study

3.5. Collect data for field test

3.6. Collect data for main assessment

3.7. Produce field test dataset

3.8. Produce main assessment dataset

3.9. Score field test assessment and code questionnaire data

3.10. Score main assessment and code questionnaire data

Task 4. Prepare and Disseminate Results (Firm-fixed Price)

4.1. Produce technical reports

4.1.1. Field test data review memo

4.1.2 Survey operations report

4.1.3 Non-response bias analysis report

4.2. Develop and Produce PISA U.S. national report

4.2.1. National report analysis plan and table/figure shells

4.2.2 Produce national report

4.3. PISA school-level reports

4.4. Produce main study electronic public-use and restricted-use datafiles and user’s guides

Task 5: Use Expert Help (Cost-reimbursable)

Task 6: PISA Validation Study (Cost-reimbursable)

Task 7 (Optional): Manage Subnational Participation in PISA (Cost-reimbursable)

Task 8 (Optional): Conference for PISA 2015 Schools (Cost-reimbursable)

Task 9 (Optional): Additional Subject-Matter Assessments (Cost Reimbursable)

Task 10 (Optional): Disseminate Results (Firm-fixed Price)

Task 11 (Optional): Item development (Firm-fixed Price)

Appendix A. Tentative Timeline for PISA 2015 Data Collection

Appendix B. Data Confidentiality, Security, Quality Control, and Contractor Staff Security Clearances 64

B-1. Overview of IES/NCES Confidentiality Statutes Confidentiality of Individuals and Institutions Protection and Security of Data

B-2. Data Security Plan Definition of Personally Identifiable Information and Direct Identifiers Development of Data Security Plan Implementation of Data Security Plan

B-3. Adherence to NCES Statistical Standards NCES Statistical Standards

B-4. Quality Control Measures Requirement Quality Control Measures

B-5. Contractor Employee Security Screening Contractor Employee Security Screening Requirements

B-6. Requirement of Affidavit of Nondisclosure Affidavit of Nondisclosure

B-7. Security and Confidentiality Training for Contractor Staff Security and Confidentiality Training

B-8 Information Collection Request (ICR) Requirement

Appendix C. Sampling Requirements for PISA 2012

Appendix D. Disclosure Analysis Plan

List of Exhibits Exhibit 1

U.S. Participation in the Program for International Student Assessment (PISA) 2015

(IESNE120015)

Introduction

The National Center for Education Statistics (NCES) in the Institute of Education Sciences

(IES) of the U.S. Department of Education (ED) supports a variety of international activities to collect and report data on education in the United States and other countries. It is the

NCES‘ goal to provide valuable comparative information about the U.S. educational performance in an international context. NCES participates in international student assessments in order to:

provide comparable indicators on student performance and schooling practices across countries in order to benchmark U.S. student performance; and suggest hypotheses about the relationship between student performance and factors that may influence performance and about areas where students have particular strengths or weaknesses.

The study covered by this procurement is part of the larger international program that NCES has participated in through collaboration with a variety of international organizations and partners, such as the Organization for Economic Cooperation and Development (OECD), the

International Association for the Evaluation of Educational Achievement (IEA), and ministries of education and other government agencies throughout the world.

Through its participation, NCES seeks to strengthen the quality, consistency and timeliness of international data. To continue this effort, the United States must follow through with well-organized and well-executed data gathering activities within our national boundaries. These efforts will allow NCES to provide policy-relevant data for comparisons of the U.S. education system with education systems in other countries, and to help inform decision making on the part of national, state, and local policy makers.

This Performance Work Statement (PWS) is written in three sections. The first section provides an overview of the Program for International Student Assessment (PISA) 2015 data collection, its purpose, its enabling legislation, and general parameters of the work to be done under this contract. The second section specifies the core tasks, subtasks, and deliverables associated with this work. The third section describes a set of optional tasks and subtasks for

PISA 2015.

Background on PISA 2015

PISA is an international assessment that focuses on 15-year-olds' performance in reading literacy, mathematics literacy, and science literacy every three years. PISA also assesses other subjects, although less frequently, such as general problem-solving and financial literacy.

PISA emphasizes functional skills that students have acquired as they near the end of mandatory schooling. PISA is organized by the Organization for Economic Cooperation and

Development (OECD), an intergovernmental organization of industrialized countries. PISA was first administered in 2000. PISA was implemented in 43 countries in 2000, in 41 countries in 2003, 57 countries in 2006, 65 countries in 2009, and 67 countries in 2012. More information on PISA can be found at http://www.pisa.oecd.org and http://nces.ed.gov/surveys/pisa.

PISA is steered by the PISA Governing Board (PGB), comprising representatives from the

OECD member countries, and implemented internationally by organizations contracted by the

OECD. The OECD Secretariat serves as the secretariat for the PGB and has overall managerial responsible for PISA internationally, including overseeing contractors that implement the international activities. The OECD has contracted with multiple organizations

(hereafter referred to as the ―international consortium‖ or ―international contractors‖) to carry out PISA 2015 development, implementation, analysis, and overall coordination across countries. In this PWS, all mention of relevant international contractors, organizations, or sponsors refer to these institutions. Each country appoints a National Project Manager (in the

United States this is a representative from NCES) to manage the study in the country. The

OECD and international contractors provide countries with a series of manuals and technical standards that document the procedures and requirements of the study.

PISA 2015 will include assessments of science, reading and mathematics literacy. The science assessment will be based on a new framework and will include some new item development, although it is expected that it will still be possible to report trends in performance. There may also be additional optional subject-matter assessments, such as environmental literacy, financial literacy and/or collaborative problem-solving. Students will complete a questionnaire and schools will complete a questionnaire.

PISA has been moving from using only paper-and-pencil assessments to including more computer-based assessments. PISA 2006 included an optional computer-based science assessment, although the United States did not participate in that component. In PISA 2009, PISA administered an optional reading assessment on computer; again, the United States did not participate. In 2012, reading and mathematics were primarily assessed via paper-and-pencil assessment but there were optional computer-based assessments as well and the United

States did participate. In 2012, problem-solving was also assessed, only on computer. For

PISA 2015, the goal is for computer-based assessments to have an even greater role. It is expected that science, mathematics, and reading will all be available in a computer-based format. Science, mathematics and reading may be administered in some countries in a paper-and-pencil format. However, the United States intends to administer only the computer-based mathematics, reading and science assessments and not the paper-based. Additional optional assessments (see Optional Task 9) may be computer-based or paper-based. To support the transition from paper-based to computer-based format, the PISA 2015 field test will include a study to establish the comparability between the two assessment modes. PISA 2015 may also incorporate some form of computer-adaptive testing.

The shift to greater use of computer-based assessments in PISA 2015 has accelerated the international development schedule as well as within-country operations. The first National

Project Managers meeting is being held earlier than normal (tentatively scheduled for

November 2012); at this meeting, countries will begin to make decisions about sampling plans and receive training in item development and other procedures. The assessment instruments (to which countries are invited to contribute) will be developed and finalized earlier in the cycle than usual in order to ensure that national test delivery systems can be created and finalized in time for the spring 2014 field test; countries are slated to begin http://www.pisa.oecd.org/ http://nces.ed.gov/surveys/pisa adapting field test instruments in June 2013 so national versions of the test delivery system can be finalized in September 2013.

Purpose of Procurement

NCES requires a contract to implement U.S. participation in PISA 2015. The overall goal for the procurement is data collection in the United States for PISA 2015 that meets all international and NCES guidelines and that makes available on a timely basis high-quality internationally comparative assessment data. Furthermore, the study shall result in a report presenting U.S. data and the international comparisons from the perspective of an American constituency. The core contract is for the U.S. data collection for a nationally-representative sample of 15-year-old students. States and/or districts may also participate in PISA and this is discussed under Optional Task 7. This contract will also include a validity study to examine the relationship between performance on PISA and educational and workforce outcomes by recruiting students that participate in PISA 2012 and administering a questionnaire in 2015, when those students are 18 years old.

PISA 2015 will follow the established design and standards set forth by the OECD and PGB.

These design and standards will cover, but are not limited to, administration procedures, sampling guidelines and requirements, survey instruments, data collection, data coding and processing. The OECD and international contractors also establish the overall study schedule and key deadlines to which the U.S. data collection activities must adhere. The OECD and international contractors provide manuals, guidelines, and technical standards that stipulate the PISA procedures and standards. Manuals from previous rounds of PISA, as well as technical documentation, example assessment items, and PISA reports, are available at http://www.pisa.oecd.org/pages/0,3417,en_32252351_32236130_1_1_1_1_1,00.html. The contractor shall follow the international guidelines for PISA 2015 and shall meet established

NCES standards and procedures (NCES standards can be found at http://nces.ed.gov/statprog/).

The contractor shall be clear that NCES is the sole authority to manage PISA in the United

States and to negotiate with the OECD or international consortium on any issues related to the study. The contractor shall not use the PISA instruments or any part of them for any other purposes without NCES‘ written consent. Under no circumstances shall the contractor conduct any activities related to PISA, including conducting the assessment in sub-national jurisdictions (e.g., states or school districts), without permission from NCES.

The contractor shall bid key staff who have experience working on an international project and who have extensive knowledge of and experience with the operations of an international assessment with a complex sample design. Given the large volume of highly technical work required by this contract and the fast-paced international schedule that must be met, there will not be time after the start of the contract for key contractor personnel to familiarize themselves with the technical content or operations of PISA.

Enabling Legislation

The authority for this study can be found in NCES‘ longstanding mission to ―collect, and analyze and disseminate statistics and other information related to education in the United http://nces.ed.gov/statprog/

States and in other nations‖ (stated in section 406 of the General Education Provisions Act, as amended (20 U.S.C. 1221e-1) and in the Education Sciences Reform Act of 2002 (P.L. 279-

107, Part C, Sec. 153):

(a) GENERAL DUTIES.-The Statistics Center shall collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including…(6) acquiring and disseminating data on educational activities and student achievement (such as the Third International Math and Science Study) in the

United States compared with foreign nations.

(See http://www.ed.gov/policy/rschstat/leg/PL107-279.pdf for the full Education Sciences

Reform Act.)

Work Statement

Independently and not as an agent of the United States Government, the contractor shall provide all personnel, materials, services, and facilities necessary for the project and perform the tasks as described below.

The anticipated period of performance for this contract is sixty (60) months. The contract shall be incrementally funded and includes (1) preparing and conducting a field test of the

PISA 2015 instruments, (2) preparing and conducting the main study or main assessment for

PISA 2015, and (3) producing public- and restricted-use release files, data documentation, and analytic and methodology reports. It may also include various optional tasks, such as administering additional subject-matter assessments, conducting a validation study with students that participated in PISA 2012, and implementing PISA at the sub-national (e.g., state) level.

A preliminary timeline for PISA 2015, with major events and activities, is in Appendix A.

The contractor shall note that the timelines are subject to change. Reference below to ―the study‖ or ―the project‖ means the PISA 2015 study (and may include the optional tasks). Due to the timing of PISA 2015, some activities at the international level may have begun by the time the contractor begins work.

Task 1: Contract Administration (Firm-fixed Price)

The contractor shall work closely with the Contracting Officer‘s Representative (COR) to implement the design of the study included under this procurement and the approach presented in the proposal.

1.1 Project initiation and planning

It is assumed that upon the start of the contract, the contractor‘s project director(s) and key staff will have reviewed and have a working knowledge of the technical methods and procedures as well as the international operations and production of past administrations of

PISA. For preparation of a proposal, bidders may wish to refer to the PISA 2009 technical report and technical standards and manuals, and other materials available at the OECD website at http://www.pisa.oecd.org/pages/0,3417,en_32252351_32236130_1_1_1_1_1,00.html.

http://www.ed.gov/policy/rschstat/leg/PL107-279.pdf http://www.pisa.oecd.org/pages/0,3417,en_32252351_32236130_1_1_1_1_1,00.html

Appendix A of this Performance Work Statement also provides scheduling information about

PISA 2015.

1.1.1 Post-award conference

Within 2 weeks of the contract award, as approved by the COR, the contractor‘s project director and four other key staff members who would be able to answer questions about sampling issues, data collection processes, costs, etc., from the contractor project team shall hold a meeting in Washington D.C. at facilities made available by ED. The meeting will include the COR and key NCES project staff and the ED Contracting Officer (CO) and

Contracting Specialist (CS). The contractor shall focus the meeting to review the overall contract tasks, identify information needed from past rounds of PISA, identify potential problems and possible solutions in the project and contract, and discuss areas of concern related to the proposed project staffing plan and other management requirements. The primary purpose of this meeting is to refine the management, staffing, and scheduling plans.

These refinements shall provide management information for use by both the contractor and the government in monitoring the work to be performed. This meeting shall also provide the opportunity for ED staff to share project experience and materials with the contractor. The contractor shall prepare and bring all necessary materials for a productive and successful initial meeting (1.1.1).

Deliverables: 1.1.1 Post-award meeting materials.

1.1.2 Produce master schedule

The contractor shall produce a project schedule that details key milestones; critical dates specified in appendix A and the ―Schedule of Deliverables (SOD)‖ and the contractor‘s production deadlines for the Office of Management and Budget (OMB) clearance, instrument revision and adaptation, assessment administrator training, data collection, scoring/coding, datafile preparation and submission, analysis and reporting. This schedule is required with the submission of the proposal and will be reviewed by NCES before the contract is awarded.

The project schedule shall include the following information in an easy-to-understand format:

Start and end dates of all tasks and subtasks;

Key milestone dates;

Completion dates of subtasks, drafts, or other events;

Key deliverable dates.

The master schedule shall make clear the primary leads and contact person for each task, subtask, or deliverable (as applicable).

The contractor shall submit a revised master schedule for the field test within 2 weeks of contract award (1.1.2), and thereafter submit a revised master schedule to the COR at each update meeting (see subtask 1.2.1). The revised schedule and all schedules going forward must reflect the international contractor milestones and deliverables, which are subject to change throughout the course of the study. The contractor shall submit a revised master schedule for the main study no later than 7 months prior to contacting schools for recruiting.

The contractor shall be prepared to update the schedule within 24 hours of a request for it by the COR.

Deliverables: 1.1.2.1 Revised master schedule for field test; 1.1.2.2 Revised master schedule for main study

1.1.3 Project Management Plan

The contractor shall prepare and submit a project management plan to the COR within 3 weeks of contract award (1.1.3.1). The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the project management plan is approved by the COR, the contractor shall manage the project in accordance with the approved plan (1.1.3.2) and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

The plan shall detail the management techniques that will be employed to ensure timely, efficient, and cost-effective instrument preparation, data collection, processing, analysis, and reporting. The plan shall reflect the requirements specified in the contract or describe and justify any revisions to the original plan specified in the contract and approved by NCES.

These might include changes in data collection approach, data processing, plans for analysis and reporting, as well as changes in the proposed project organization, staffing plan, and schedule. The plan will fully describe the rationale for any proposed changes.

Bidders must address in their proposal how they propose to manage the project to ensure timely, efficient, and cost-effective instrument preparation, data collection, processing, analysis, and reporting.

Deliverables: 1.1.3.1 Draft project management plan; 1.1.3.2 Final project management plan

1.1.4 Staffing Plan

The contractor shall prepare and submit a staffing plan to the COR within 3 weeks of contract award (1.1.4.1). The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the

COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the staffing plan is approved by the COR, the contractor shall implement the plan (1.1.4.2) and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

The staffing plan shall detail the project‘s overall staffing, including contractors and subcontractor staff dedicated to the study, as well as the training or preparatory activities to ensure that all staff are familiar with and abide by the requirements of the study (including, but not limited to, the study‘s need for high quality and timely completion of all activities and for strictly enforced confidentiality procedures).

The ability of the COR to obtain timely and accurate information and decisions from contractor staff is considered of the utmost importance, and will remain a high priority for the contractor. The contractor shall be responsible for the timely and satisfactory completion of work carried out by subcontractors, and shall convey to subcontractors the need to comply with the rules, regulations, guidelines, standards, timelines, and deliverable schedules of all tasks and sub-tasks included under this procurement.

Bidders must describe in their proposals how they propose to staff the project and outline proposed staff responsibilities and training or preparatory activities to ensure high quality and timely completion of all activities.

Deliverables: 1.1.4.1 Draft staffing plan; 1.1.4.2 Final staffing plan

1.1.5 Data Security Plan

NCES is required by law to develop and enforce standards designed to protect the confidentiality of individually identifiable respondents, including students and schools. These requirements cover the collection, reporting, and publication of data; they are described in detail in appendix B. The contractor shall ensure that all contractor-led operations, including those of subcontractors, adhere to federal laws and standards in this regard.

To describe how the contractor will do this, the contractor shall prepare and submit a draft data security plan to the COR within 3 weeks of contract award (1.1.5.1). The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the data security plan is approved by the

COR, the contractor shall implement the plan (1.1.5.2) and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

The security of this study‘s data is of utmost importance to NCES and the security plan will be reviewed by NCES‘ Chief Statistician. For a detailed discussion of the data security requirements of the contract, see appendix B.

Deliverables: 1.1.5.1 Draft data security plan; 1.1.5.2 Final data security plan

1.1.6 Staff Security Requirements

All contractor staff to work on the project for 30 days or more must complete a series of security forms and undergo a federal background check per OMB requirements, with higher levels of security clearance required for those who visit schools or who may come in contact with confidential data. The contractor shall ensure that all staff who may come in contact with confidential data shall (a) be trained in the proper handling of confidential data and (b) first sign and then notarize an affidavit of nondisclosure before having access to any data with confidential information. The contractor shall execute these affidavits of nondisclosure and retain them in a secure location. The contractor shall provide a list to the COR of all the individuals who have signed affidavits, the position in the organization of the person, and the person‘s relationship to the study, within 4 weeks of the award of the contract (1.1.6).

As new persons are assigned to the project, the security forms shall be filled out and an affidavit shall be executed for them on the first working day of assignment to the study and an updated list provided to the COR, no less than on a quarterly basis. The contractor shall notify the COR if any contractor staff leave the project. The contractor shall be aware of and comply with all contractor personnel security regulations and all Department IT security policy requirements, as described in Appendix B.

Deliverables: 1.1.6 List of all personnel with signed Affidavits of

Nondisclosure

1.1.7 Transmittals of study products and quality of deliverables

The contractor shall transfer and retrieve study-related documents and files through the

Internet, to or from NCES, or other designated contractors. The ability to transfer and retrieve documents through the Internet is important to maintaining the integrity of electronic versions of documents that may eventually be posted on the NCES web site or distributed internally.

Thus, it is imperative that the contractor works closely with NCES web site personnel to set up and ensure that all transfer protocols are followed and that transfer and retrieval of documents (especially from the contractor‘s management website, see subtask 1.2.4, and the

NCES secure data server on the NCES Member‘s website) is completed to the satisfaction of

NCES. The contractor shall conduct a test of its transfer/retrieval system(s), within 4 weeks of the award of the contract and prior to making use of the system for key contract deliverables, and shall eliminate any problems found to ensure a trouble-free system (1.1.7).

For the contract, the contractor shall transmit all correspondence directly to the NCES COR via e-mail that is compatible with Microsoft Outlook. Compatibility will extend to the ability to access attachments sent as part of e-mails. As of 2012, NCES uses Microsoft Office 2010:

Word, PowerPoint, and Excel as its primary software packages for communicating doc-uments, charts, and tables. NCES uses the Department of Education‘s Microsoft Outlook software as its communications package. All deliverables are to be delivered in native file format, in the version of Word, PowerPoint, Excel, or other software packages used by NCES.

All final deliverables that are to be assigned a NCES publication number or that will be made public through the NCES website will be delivered in PDF format (or another appropriate format approved by the COR) in accordance with the NCES requirements published in

―Policies, Guidelines, Standards and Procedures for Web Publishing for IES and NCES‖

(available at https://members.nces.ed.gov/ webstandards/), and must adhere to Section 508 requirements for accessibility. More information on accessibility is available through the

Federal IT Accessibility Initiative at: http://www.section508.gov.

Although the term ―draft‖ is used throughout this PWS, it is important that the contractor understand what the Government means, and does not mean, by ―draft.‖ A draft is a complete product of high quality that the contractor (and the government) can publish with confidence as to its accuracy and usefulness. A ―draft‖ is not simply what is available on the due date— regardless of completeness or quality. NCES anticipates that only minor changes to drafts will be needed for them to become final deliverables. If the contractor has a subcontractor who is preparing deliverables, the prime contractor shall review and stand by the subcontractor‘s work.

Every final deliverable will be sent with a cover memorandum signed or initialed by the project director or someone of higher corporate stature than the project director. The cover memorandum will certify that the contractor’s organization stands behind the quality of the product. This holds for all final versions of products—even those prepared primarily by consultants or subcontractors. At no time will the government take direct delivery of a product from a consultant or subcontractor.

https://members.nces.ed.gov/%20webstandards/ http://www.section508.gov/

All draft deliverables will be due at the office of the COR and the CS on or before their due date. Deliverables must be sent in both electronic copy and, if requested by the COR, hard copy format. Hard copies of deliverables must be sent to the COR such that they arrive on or before their due date. Unless otherwise specified, three copies of each hard copy deliverable will be required.

Deliverables: 1.1.7 Test of transfer/retrieval system(s)

1.1.8 Study brochure

The contractor shall assist NCES in developing an informational brochure for the study that will describe the study‘s general goals. At a minimum, it will identify the purpose of the project, the sponsoring organizations, the types of data to be collected, and the data collection schedule. The brochures will be distributed to individuals and organizations requesting information about the study and will be included as part of the information packet used to secure cooperation of study participants. The brochure should also be colorful and attractive and use photos and other graphics as appropriate.

The contractor shall submit a draft of the brochure 6 weeks after contract award (1.1.8.1).

The COR will review the drafts and provide comments and suggestions within 2 weeks.

NCES will provide the contractor with an electronic version of the logo of the study, which the contractor shall use in the production of the brochure and other published materials.

Based on initial COR comments, the contractor shall revise the draft of the brochure and will resubmit it for full NCES review (see Exhibit 1, under Task 4, for a description of the NCES review process). Once approved, an electronic version of the brochure will be delivered within 3 weeks, along with the final print copies, for inclusion on the NCES web site

(1.1.8.2). The electronic version of the brochure will meet NCES web site design and formatting guidelines, which currently requires PDF and HTML formatting, with graphics as

.jpg or .gif files. The contractor shall produce enough copies of the final, printed brochure for all information packets needed to secure cooperation of study participants and deliver an additional 50 copies to the COR for NCES.

Deliverables: 1.1.8.1 Draft informational brochure; 1.1.8.2 Final informational brochure

1.2 Contract management

Because this contract includes a series of time-sensitive and complex activities, it is critical that it be monitored and managed effectively. To that end, NCES requests the following general study management activities.

1.2.1 Project update meetings with contractor

For the life of the contract, the contractor will hold regular weekly or biweekly (or less frequently, if the COR so determines) meetings for key contractor staff to update the COR or other NCES staff on the project‘s progress. The COR may propose to hold these meetings via telephone conference calls, over the Internet, or face-to-face if the meeting is critical. For budget purposes, the contractor shall plan up to 40 meetings a year throughout the contract, of which up to 12 meetings a year may be face-to-face. The contractor shall participate in the development of the goals and agenda for each meeting. The contractor shall be responsible for distributing the agenda and other meeting materials, including the current version of the project‘s master schedule, to no more than 12 meeting attendees. Materials to be distributed will be approved, in advance, by the COR.

Deliverables: 1.2.1 Regular project meetings materials including master schedule

1.2.2 Other meetings with NCES/ED

The contractor shall send one representative to all National Project Manager (NPM) meetings for the study; there are typically two each year. The contractor is strongly urged to send the

Project Director to the meetings, or the staff member who has primary responsibility for conduct of the study. Additional personnel may be sent, if needed, upon approval of the

COR. For costing purposes, the contractor should assume 5 international NPM meetings in a foreign location over the course of the contract, two of which may focus on scoring/coding training. For each NPM meeting attended, the contractor shall submit to the COR a trip report that documents the purpose and outcomes of the meeting, along with any follow-up actions required, within 3 business days after the end of the meeting. The contractor will work with the COR to develop a common format to be used for such trip reports for the first NPM meeting attended by the contractor.

Deliverables: 1.2.2 NPM meeting trip reports

1.2.3 Monthly study progress and financial reports

The contractor shall submit to the COR and CS a regular monthly progress report, by the 15th day of the month following the month the report covers. This report shall be delivered before or concurrent with the contractor‘s monthly billed invoice so that it is available to check and approve each month‘s invoice. At a minimum, in each monthly progress report, the contractor shall include information on the following aspects of the study and its operation:

The past month‘s work, including a description of continuing work, new work, and completed work, with key dates on which tasks or activities were initiated or completed during the month;

The upcoming month‘s work, including a description of the work and key dates by which time tasks or activities will be initiated or completed during the month;

Major accomplishments for the month;

List of products and milestones submitted or achieved during month, with a short description of each; and

Problems/Issues and how each has been or will be addressed by the contractor.

The contractor shall also submit to the COR and CS a regular monthly financial data report by the 15th day of the month following the month the report covers. This report shall be delivered before or concurrent with the contractor‘s monthly billed invoice so that it is available to check and approve each month‘s invoice. In each monthly financial data report, the contractor shall, at a minimum, include

(1) information on the following aspects of the study and its operation:

Past month‘s cost of work completed, by task, subtask, and cumulatively;

Percent of contracted amount spent to date, by task;

Estimated cost to complete, total estimated cost, and total contracted cost;

Difference between the total estimated cost and the contracted cost; and

Staff hours spent on each task during the prior month;

(2) appropriate records and information to permit the COR and CS to certify that the services that are listed on monthly bills are actually used and are for official purposes; and

(3) a manpower report prepared and signed by the project director that summarizes actual personnel assignments for the month just completed, showing for each named individual the hours charged, by task. For individuals not named in the contract, the manpower report will show hours spent by position (e.g., data collectors). All hours billed to the contract will be summarized in the report.

The contractor shall include detailed information in the monthly financial data reports on any anticipated or actual cost overruns or cost savings, whether incurred within a single task or cumulatively. From this information, the contractor shall ensure that the COR is clear about the progress being made on tasks, the work that remains to be accomplished, the financial resources used and available, and any problems and proposed solutions. At the request of the

CO or CS, if additional issues need to be addressed in either monthly report, the contractor shall submit an addendum within 2 days of the request.

The format for the monthly report will be consistent from month to month. The same format will be used to report on any subcontractor activity costs.

The monthly report shall serve as a justification for all work billed for the month. Therefore, the contractor shall take care to ensure not only that all work billed for the month is described in sufficient detail in the monthly report to support an audit, but that all detailed information

(on completed work) contained in the monthly report is transparently and easily associated with its respective cost or billing. The monthly progress and financial reports shall be submitted together each month and the contractor shall respond to questions regarding either document within 1 day.

Deliverables: 1.2.3 Monthly study progress and financial reports

1.2.4 Project management portal

No more than 1 month after the start of the project, the contractor shall establish, and thereafter maintain, space on an internal server to house all project reports, interim and final versions of deliverables, and other appropriate project documentation. The space shall be accessible to the COR and other appropriate contractor or outside staff. This will aid in ensuring that all project staff have the most up-to-date versions of materials and will complement (but not replicate) websites maintained by the OECD and International

Contractors. The space shall be password protected with access limited to NCES, contractor staff, and key subcontractor staff as appropriate. No data collected from the respondents shall be placed on the project management portal.

Deliverables: 1.2.4 Project portal for housing electronic copies of project materials

1.2.5 Maintenance of data files

The contractor shall maintain complete data files for all components of PISA 2015, and all supplemented files created and obtained in support of the study, for the life of the contract.

Maintenance of data files means keeping data file documentation available for reproduction on request of the COR or other authorized individuals. Certain data files that will be maintained will not be made available to the public because of privacy considerations. In these cases, file maintenance means that the data files will be kept intact, in a form that can be updated or copied. Data files will not be allowed to expire, to be released, overwritten, or otherwise destroyed. These files will be maintained with a software system capable of interlinking them for analysis (or other) purposes while, at the same time, preserving the necessary confidentiality requirements. If prior PISA datasets are required by the contractor, the COR will arrange for delivery for those datasets.

1.2.6 Relationship of the contractor to NCES and other agencies

The contractor shall report all activities for this contract and activities related to PISA 2015 to the COR. All data, publications, and products of the study are the sole property of the U.S.

government, and shall be provided to it as specified below and upon request. Under no circumstances will the contractor, or any subcontractors, distribute or release any data, publications, or products of the study in any form to any individual or party without prior permission of the COR. Any contractor or subcontractor conference presentations or papers, based on this study, shall be reviewed and cleared by NCES prior to any such presentation.

This restriction will remain in effect throughout the life of the contract, and until NCES officially releases the data, publications, or products to the public.

The contractor shall work through the COR in all relationships and discussions with the

OECD, international contractors, foreign governments, foreign education professionals, and all other agencies and individuals related to the study. The contractor shall not convey to any of these entities information that may be construed as NCES, IES, or Department of

Education policy, except where related to technical aspects of the study or where given explicit permission by the COR.

1.2.7 Readiness for work under conditions of unusual speed and flexibility

In the event that the study‘s international plans and time schedules change rapidly due to various necessities, the contractor shall be prepared to work with changing requirements. For example, time schedules may become very tight because the OECD or its contractors are unable to meet their projected study timelines. Whenever necessary, the contractor shall be prepared to perform activities under conditions of emergency speed. Emergency speed will be defined according to four priority levels:

Priority 1: within 24 hours of request

Priority 2: within 72 hours of request

Priority 3: within 5 working days of request

Priority 4: within 10 working days of request

In the event that changes to plans or time schedules require the completion of a task or activity under emergency speed, the COR will notify the contractor of the request, after consultation with the Contracting Officer (CO), and identify the timeframe (i.e., priority level) under which the task or activity will be completed by the contractor.

1.2.8 Quality control measures

NCES is firmly committed to determining how well each project meets its goals, the quality of data collected (including response rates at the school and the student levels), and the implications of the experiences and outcomes of the project for future projects. Project quality control should be integrated into every phase of the project. For instance, potential problems should be identified early so that possible solutions are tested as part of the field test. Item reliability and validity tests for questionnaires and assessments shall be built into the data collection efforts, and results from these tests shall inform data collection and processing.

Nonresponse analyses shall be conducted as part of the weighting process, and information about the survey methodology and sample design effects shall be made available to data users.

Quality control also includes (1) managing staff in the field to ensure the high quality of their performance; (2) instituting procedures relevant to confidentiality and protection of data

(including the secure transmittal of confidential data); and (3) ensuring deliverables are submitted on time and are of sufficient quality (in terms of accuracy, coherence, clarity, and comprehensiveness) to meet NCES statistical standards, OECD/PISA standards, professional standards, and NCES and IES style guidelines. As noted earlier, NCES expects that only minor changes to drafts will be needed for products to become final deliverables.

Again, all work conducted under this contract shall, at a minimum, meet the standards and guidelines set forth in the publication NCES Statistical Standards, and amendments implemented since that time. The standards can be accessed at http://nces.ed.gov/statprog/.

The IES guidelines can be accessed at http://nces.ed.gov/statprog/style.asp.

1.3. Obtain OMB clearance for field test data collection

The contractor shall prepare a clearance package for obtaining OMB approval of the PISA

2015 field test and non-cognitive survey instruments that will be used in the study, and any other materials that will be distributed to districts, schools students, and parents (e.g., parent permission letters), in accordance with current OMB, ED, and NCES requirements for such packages.

Preparing an OMB clearance package entails assembling an extensive amount of information to (a) justify the data collection as a whole as well as individual background questions; (b) estimate the cost of the data collection to the federal government and the amount of burden (in both time and cost) on respondents; and (c) explain the data‘s uses, efforts taken to minimize the burden on respondents, any provisions for payments or gifts to respondents, steps to assure confidentiality, and the statistical methods that are to be employed in collecting and preparing the data (e.g., dealing with nonresponse). OMB requires that all this information be written up in a Supporting Statement. An OMB clearance package for submission must include the Supporting Statement, copies of all materials that will go to respondents (such as recruitment materials, survey questionnaire, letters of consent, etc.), and information on any national adaptations of the instruments (see subtask 2.3). The contractor shall prepare a draft

OMB clearance package for the field test no later than August 1, 2013, and preferably earlier.

http://nces.ed.gov/statprog/style.asp

After COR review and internal NCES review, the contractor shall revise the draft OMB clearance package within 5 business days of receipt of comments. NCES will submit the revised OMB clearance packages to ED and OMB for review and approval. Failure to submit the OMB clearance package by the deadline may result in the contractor being unable to collect data in a timely manner. The contractor shall be aware that the time between submission of the OMB clearance package and receipt of a clearance number from

OMB for inclusion on all survey documents takes a minimum of four months, and upwards of six months, sometimes more. The contractor shall take these time constraints into consideration when preparing and submitting the OMB clearance package. The contractor shall work diligently and expeditiously with the COR to seek OMB clearance prior to recruiting schools.

In preparing the OMB clearance package, the contractor shall follow instructions for completing the Information Collection Submission Worksheet and writing the Supporting

Statement required for Paperwork Reduction Act submissions (for information on the requirements for a clearance package, officially called at present an Information Collection

Review, see http://www.whitehouse.gov/omb/inforeg_infocoll/). The contractor shall prepare all written materials needed in the formal clearance and approval process according to OMB requirements as outlined in the Information Collection Submission Worksheet. The package will include the U.S.-adapted background questionnaires but not the cognitive (assessment) instruments for the study. This approval process is expected to involve the following series of steps (called out as deliverables):

Contractor shall provide a draft package to the COR and shall revise it within 5 business days of receiving the COR‘s comments (1.3.1).

The COR provides the package to the NCES OMB liaison and chief statistician, and the contractor shall revise package according to comments received within 5 business days (1.3.2).

The NCES OMB liaison provides the package to the ED OMB liaison office (RIMS) and the contractor shall revise the package according to comments received within 5 business days (1.3.3).

ED provides package to OMB; the contractor shall answer OMB questions (1.3.4) within 5 business days (there may be more than one round of OMB questions) and revise the package according to final comments received within 3 business days

(1.3.5).

The contractor staff shall be prepared to join the COR in person or via conference call for a review of the materials with ED and OMB staff. The contractor shall ensure that authorization by OMB has been obtained before data collection begins. All data collection forms will bear the approval number assigned by OMB.

The contractor shall be aware that states, districts, and schools may not be contacted until

OMB has approved the study plan. The contractor shall take this into consideration when planning the field tests and main data collections.

Following approval, the contractor shall provide NCES/ED with one hard copy and one electronic copy of OMB‘s official approval or clearance for the study (1.3.6).

http://www.whitehouse.gov/omb/inforeg_infocoll/

Deliverables: 1.3.1 Draft OMB field test clearance package; 1.3.2 Revised

OMB field test clearance package; 1.3.3 Revised package based on RIMS comments; 1.3.4 Responses to OMB questions on package; 1.3.5 Revised final field test clearance package for OMB approval; 1.3.6 OMB clearance for field test

1.4. Obtain OMB clearance for the main study data collection

OMB may grant clearance for both the field test and the main study activities or may require that after the field test a new clearance be obtained for the study. In the case that new clearance is required, the COR will provide guidance on the type of materials needed to obtain this clearance, which may include either a memo describing any changes made in instruments or procedures from the field test to the main study, or a full new clearance package similar to that prepared for the field test.