Attachment A - DRAFT PWS TIMMS.PDF

DRAFT - PERFORMANCE WORK STATEMENT

U.S. Participation in the Trends in International Mathematics and Science Study

(TIMSS) 2015

Contents

List of Exhibits

Introduction

Purpose of Procurement

Enabling Legislation

Work Statement

Task 1. Contract Administration (Firm Fixed Price)

1.1 Project initiation and planning

1.1.1 Post-award conference

1.1.2 Produce master schedule

1.1.3 Project Management Plan

1.1.4 Staffing Plan

1.1.5 Data Security Plan

1.1.6 Staff Security Requirements:

1.1.7 Transmittals of study products and quality of deliverables

1.1.8 Study brochures

1.2 Contract management

1.2.1 Project update meetings with contractor

1.2.2 Other meetings with NCES/ED

1.2.3 Monthly study progress and financial reports

1.2.4 Internal Project management portal

1.2.5 Maintenance of data files

1.2.6 Relationship of the contractor to NCES and other agencies

1.2.7 Readiness for work under conditions of unusual speed and flexibility

1.2.8 Quality Control Measures

1.3 Obtain OMB clearance for field test data collection

1.4 Obtain OMB clearance for the main study’s data collection

1.5 Prepare for briefings on project

1.5.1 Prepare project briefing materials

1.5.2 Provide briefings or presentations

Task 2. Preparations for Assessments (Firm Fixed Price)

2.1 Field test sampling plan and sample

2.2 Main study sampling plan and sample

2.3 Prepare field test instruments

2.4 Prepare main study instruments

2.5 Conduct cognitive lab work for response items

Task 3. Recruiting, schools, teachers, and students (Cost Reimbursable)

3.1 Recruit Schools, Teachers, and Students for Field Test

3.2 Recruit Schools, Teachers, and Students for Main Assessment

Task 4. Data Collection and Delivery (Firm Fixed Price)

4.1 Recruit and train data collectors for field test

4.2 Recruit and train data collectors for main assessment

4.3 Collect data for field test

4.4 Collect data for main assessment

4.5 Produce field test dataset for IEA

4.6 Produce main assessment dataset for IEA

4.7 Prepare responses to IEA information requests about data collection

4.8 Prepare National curriculum questionnaires

4.9 Produce U.S. chapter for the TIMSS Encyclopedia

Task 5. Scoring and Coding (Cost Reimbursable)

5.1 Score field test assessment and code questionnaire data

5.2 Score main assessment and code questionnaire data (IDIQ, dependent on number of respondents)

Task 6. Prepare and Disseminate Results (Firm Fixed Price)

6.1 National report analysis plans

6.2 Produce reports

Non-response bias analysis report

Survey operations reports

Field test data review memo

National reports

TIMSS school reports

TIMSS released item sets

Task 7 (Optional): Utilize expert help (Cost Reimbursable)

Expert panel

Task 8 (Optional): Comparison of NAEP/TIMSS (Firm Fixed Price)

Task 9 (Optional): State benchmarking (Cost Reimbursable)

Task 10 (Optional): TIMSS-NAEP mathematics linking study (Cost Reimbursable)

Task 11 (Optional) TIMSS Advanced assessment (Cost reimbursable)

Task 12 (Optional): Produce main study electronic public-use datafiles and user’s guides

(Firm Fixed Price)

Task 13 (Optional): Disseminate Results (Firm Fixed Price)

Period of Performance:

Staff Skills and Commitment

Appendix A. Tentative Timelines for TIMSS 2015 Data Collections

Appendix B. Data Confidentiality, Security, Quality Control, and Contractor Staff

Security Clearances

B-1. Overview of IES/NCES Confidentiality Statutes

Confidentiality of Individuals and Institutions

Protection and Security of Data

B-2. Data Security Plan

Definition of Personally Identifiable Information and Direct Identifiers

Development of Data Security Plan

Implementation of Data Security Plan

B-3. Adherence to NCES Statistical Standards

NCES Statistical Standards

B-4. Quality Control Measures Requirement

Quality Control Measures

B-5. Contractor Employee Security Screening

Contractor Employee Security Screening Requirements

B-6. Requirement of Affidavit of Nondisclosure

Affidavit of Nondisclosure

B-7. Security and Confidentiality Training for Contractor Staff

Security and Confidentiality Training

B-8 Information Collection Request (ICR) Requirement

Appendix C. Sampling for TIMSS 2015

C-1. TIMSS Target Populations

C-2. TIMSS Sample Design

C-3. Exclusions

C-4. Response Rates

School-Level

Student-Level

C-5. School Sampling

The Frame (School List)

School Sample Selection

Replacement Schools

Treatment of Small Schools

C-6. Assessment Booklet Design

Appendix D. Disclosure Analysis Plan

Appendix E. Department IT Security

List of Exhibits

Exhibit 1

U.S. Participation in the Trends in International Mathematics and Science Study

(TIMSS) 2015

(IESNE120004)

Introduction

The National Center for Education Statistics (NCES) in the Institute of Education Sciences

(IES) of the U.S. Department of Education (hereafter, the Department) supports a variety of international activities to collect and report data on education in the United States and other countries. It is the NCES’ goal to provide valuable comparative information about the U.S.

students’ educational performance in an international context. NCES participates in international student assessments in order to:

provide comparable indicators on student performance and schooling practices across countries in order to benchmark U.S. student performance; and suggest hypotheses about the relationship between student performance and factors that may influence performance and about areas where students have particular strengths or weaknesses.

The study covered by this procurement is part of the larger international program that NCES has participated in through collaboration with a variety of international organizations and partners, such as the Organization for Economic Cooperation and Development (OECD), the

International Association for the Evaluation of Educational Achievement (IEA), and ministries of education and other government agencies throughout the world.

Through its participation, NCES seeks to strengthen the quality, validity, consistency, and timeliness of international data. To continue this effort, the United States must follow through with well-organized and well-executed data gathering activities within our national boundaries. These efforts will allow NCES to provide policy-relevant data for international comparisons of the U.S. adult population’s competencies and skills, and to help inform decision making on the part of national, state, and local policy makers, especially those concerned with economic development and workforce training.

This Performance Work Statement (PWS) is written in three sections. The first section provides an overview of the TIMSS 2015 data collection, its purpose, its enabling legislation, and general parameters of the work to be done under this contract. The second section specifies the core tasks, subtasks, and deliverables associated with this work. The third section describes a set of optional tasks and subtasks for TIMSS 2015.

Purpose of Procurement

NCES requires a contract to implement U.S. participation in the Trends in International

Mathematics and Science Study (TIMSS) 2015. TIMSS is an international assessment that focuses on the mathematics and science knowledge and skills of fourth- and eighth-graders.

TIMSS is organized by the International Association for the Evaluation of Educational

Achievement (IEA), an international organization of universities and governments based in

Amsterdam, the Netherlands. TIMSS has been designed to track changes in student achievement over time, and has been administered on a four-year cycle since starting in 1995.

There have been five cycles of TIMSS thus far (1995, 1999, 2003, 2007, and 2011). In 2011, 62 countries implemented TIMSS at the national level, at either fourth- or eighth-grade, or both. More information on TIMSS can be found at http://nces.ed.gov/timss.

The overall goal for the procurement is data collection in the United States for TIMSS 2015 that meets all international and NCES guidelines and that makes available on a timely basis high-quality internationally comparative assessment data. Furthermore, the study shall result in one or more publications that report on U.S. data and the international comparisons from the perspective of an American constituency.

TIMSS 2015 will follow the established design and standards as set forth by the IEA. These design and standards will cover, but are not limited to, administration procedures, sampling guidelines and requirements, survey instruments, data collection, data coding and processing.

The IEA has contracted with the International Study Center at Boston College to oversee the international development and coordination of the study. In the Performance Work Statement

(PWS) to follow, mention of the Study Center or the international organization refers to these institutions. The contractor shall follow the international guidelines for TIMSS 2015 and shall meet established NCES standards and procedures (NCES standards can be found at http://nces.ed.gov/statprog/).

The contractor shall be clear that NCES is the sole authority to manage TIMSS in the United

States and to negotiate with the international consortium on any issues related to the study.

The contractor shall not use the items for any other purposes without NCES’ written consent.

Under no circumstances shall the contractor conduct any activities related to TIMSS, including conducting the assessment in sub-national jurisdictions (e.g., states or school districts), without permission from NCES.

The contractor shall bid key staff who have experience working in an international project and who have extensive knowledge of and experience with the operations of an international assessment based on a complex sample design. Given the large volume of highly technical work required by this contract and the fast-paced international schedule that must be met, there will not be time after the start of the contract for key contractor personnel to familiarize themselves with the technical content or operations of TIMSS.

Enabling Legislation

The authority for this study can be found in NCES’ longstanding mission to “collect, and analyze and disseminate statistics and other information related to education in the United

States and in other nations” (stated in section 406 of the General Education Provisions Act, as amended (20 U.S.C. 1221e-1) and in the Education Sciences Reform Act of 2002 (P.L. 279-

107, Part C, Sec. 153):

http://nces.ed.gov/timss http://nces.ed.gov/statprog/

(a) GENERAL DUTIES.-The Statistics Center shall collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including…(6) acquiring and disseminating data on educational activities and student achievement (such as the Third International Math and

Science Study) in the United States compared with foreign nations.

(See http://www.ed.gov/policy/rschstat/leg/PL107-279.pdf for the full Education

Sciences Reform Act.)

Work Statement

Independently and not as an agent of the United States Government, the Contractor shall provide all personnel, materials, services, and facilities necessary for the project and perform the tasks as described below.

The anticipated period of performance for this contract is sixty (60) months. The contract shall be incrementally funded and includes (1) preparing and conducting a field test of the

TIMSS 2015 instruments, (2) preparing and conducting the main study or main assessment for TIMSS 2015, and (3) producing public- and restricted-use release files, data documentation, and analytic and methodology reports. It may also include various optional tasks, such as preparing and conducting a TIMSS-NAEP linking study as well as the TIMSS

Advanced 2015.

A preliminary timeline for TIMSS 2015, with major events and activities, is in appendix A.

The contractor shall note that the timelines are subject to change. Reference below to “the study” or “the project” means the TIMSS 2015 study (and may include the TIMSS Advanced

2015 assessment, if the United States chooses to participate in that additional component of

TIMSS and exercises the TIMSS Advanced Option, below). Due to the timing of TIMSS

2015, some activities at the international level may have begun by the time the contractor begins work.

Task 1. Contract Administration (Firm Fixed Price)

The contractor shall work closely with the Contracting Officer’s Representative (COR) to implement the design of the study included under this procurement and the approach presented in the proposal.

1.1 Project initiation and planning

It is assumed that upon the start of the contract, the contractor’s project director(s) and key staff will have reviewed and have a working knowledge of the technical methods and procedures as well as the international operations and production of past administrations of

TIMSS (as explained in detail in the TIMSS international technical reports available at http://timss.bc.edu/#).

http://www.ed.gov/policy/rschstat/leg/PL107-279.pdf http://timss.bc.edu/

1.1.1 Post-award conference

Within 2 weeks of the contract award, as approved by the COR, the contractor’s project director and four other key staff members who would be able to answer questions about sampling issues, data collection processes, costs, etc., from the contractor project team shall hold a meeting in Washington D.C. at facilities made available by the Department. The meeting will include the COR, the Contract Officer (CO), key NCES project staff, and the

Department’s Contract Specialist (CS). The contractor shall focus the meeting to review the overall contract tasks, identify information needed from past rounds of TIMSS, identify potential problems and possible solutions in the project and contract, and discuss areas of concern related to the proposed project staffing plan and other management requirements.

The primary purpose of this meeting is to refine the management, staffing, and scheduling plans. These refinements shall provide management information for use by both the contractor and the government in monitoring the work to be performed. This meeting shall also provide the opportunity for Department staff to share project experience and materials with the contractor. The contractor shall prepare and bring all necessary materials for a productive and successful initial meeting (deliverable 1.1.1)

Deliverables: 1.1.1 Post-award meeting materials.

1.1.2 Produce master schedule

The contractor shall produce a project schedule that details key milestones; critical dates specified in appendix A and the “Schedule of Deliverables;” and the contractor’s production deadlines for the Office of Management and Budget (OMB) clearance, instrument revision and adaptation, assessment administrator training, data collection and analysis, as well as reporting. This schedule is required with the submission of the proposal and will be reviewed by NCES before the contract is awarded.

The project schedule shall include the following information in an easy-to-understand format:

Start and end dates of all tasks and subtasks;

Key milestone dates;

Completion dates of subtasks, drafts, or other events;

Key deliverable dates.

The master schedule shall make clear the primary leads and contact person for each task, subtask, or deliverable (as applicable).

The contractor shall submit a revised master schedule for the field tests within 2 weeks of contract award (1.1.2), and thereafter submit a revised master schedule to the COR at each update meeting (see subtask 1.2.1). The contractor shall submit a revised master schedule for the main study no later than January 2013. The contractor shall be prepared to update the schedule within 24 hours of a request for it by the COR. The International Study Center will provide updated timelines at each meeting of the National Research Coordinators (NRCs).

Deliverables: 1.1.2 Revised master schedule for field test

1.1.3 Project Management Plan

The contractor shall prepare and submit a project management plan to the COR within 3 weeks of contract award (1.1.3.1). The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the project management plan is approved by the COR, the contractor shall manage the project in accord with the approved plan (1.1.3.2) and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

The plan shall detail the management techniques that will be employed to ensure timely, efficient, and cost-effective data collection, processing, analysis, and reporting. The plan shall reflect the requirements specified in the contract or describe and justify any revisions to the original plan specified in the contract and approved by NCES. These might include changes in data collection approach, data processing, plans for analysis and reporting, as well as changes in the proposed project organization, staffing plan, and schedule. The plan will fully describe the rationale for any proposed changes.

Deliverables: 1.1.3.1 Draft project management plan; 1.1.3.2 Final project management plan

1.1.4 Staffing Plan

The contractor shall prepare and submit a staffing plan to the COR within 3 weeks of contract award (1.1.4.1). The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the

COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the staffing plan is approved by the COR, the contractor shall implement the plan (1.1.4.2) and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

The staffing plan shall detail the project’s overall staffing, including contractors and subcontractor staff dedicated to the study, as well as the training or preparatory activities to ensure that all staff are familiar with and abide by the requirements of the study (including, but not limited to, the study’s need for high quality and timely completion of all activities and for strictly enforced confidentiality procedures).

The ability of the COR to obtain timely and accurate information and decisions from contractor staff is considered of the utmost importance, and will remain a high priority of the contractor. The contractor shall be responsible for the timely and satisfactory completion of work carried out by subcontractors, and shall convey to subcontractors the need to comply with the rules, regulations, guidelines, standards, timelines, and deliverable schedules of all tasks and sub-tasks included under this procurement.

Deliverables: 1.1.4.1 Draft staffing plan; 1.1.4.2 Final staffing plan

1.1.5 Data Security Plan

NCES is required by law to develop and enforce standards designed to protect the confidentiality of individually identifiable respondents, including students and schools. These requirements cover the collection, reporting, and publication of data; they are described in detail in appendix B. The contractor shall ensure that all contractor-led operations, including those of subcontractors, adhere to federal laws and standards in this regard.

To describe how the contractor will do this, the contractor shall prepare and submit a data security plan to the COR within 3 weeks of contract award (1.1.5.1). The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the data security plan is approved by the COR, the contractor shall implement the plan (1.1.5.2) and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

The security of this study’s data is of utmost importance to NCES and the security plan will be reviewed by NCES’ Chief Statistician. For a detailed discussion of the data security requirements of the contract, see appendix B.

Deliverables: 1.1.5.1 Draft data security plan; 1.1.5.2 Final data security plan

1.1.6 Staff Security Requirements:

Affidavits of Nondisclosure, Security and Confidentiality Training, and

Background Checks

All contractor staff to work on the project for 30 days or more must complete a series of security forms and undergo a federal background check per OMB requirements, with higher levels of security clearance required for those who visit schools or who may come in contact with confidential data. The contractor shall ensure that all staff who may come in contact with confidential data shall (a) be trained in the proper handling of confidential data and (b) first sign and then notarize an affidavit of nondisclosure before having access to any data with confidential information. The contractor shall execute these affidavits of nondisclosure and retain them in a secure location. The contractor shall provide a list to the COR of all the individuals who have signed affidavits, the position in the organization of the person, and the person’s relationship to the study, within 4 weeks of the award of the contract (1.1.6).

As new persons are assigned to the project, the security forms shall be filled out and an affidavit shall be executed for them on the first working day of assignment to the study and an updated list provided to the COR, no less than on a quarterly basis. The contractor shall notify the COR if any contractor staff leave the project. The contractor shall be aware of and comply with all contractor personnel security regulations, as described in appendix B, and all

Department IT security policy requirements for personnel, as described in appendix E.

Deliverables: 1.1.6 List of all personnel with signed Affidavits of Nondisclosure

1.1.7 Transmittals of study products and quality of deliverables

The contractor shall transfer and retrieve study-related documents and files through the

Internet, to or from NCES, or other designated contractors. The ability to transfer and retrieve documents through the Internet is important to maintaining the integrity of electronic versions of documents that may eventually be posted on the NCES web site or distributed internally.

Thus, it is imperative that the contractor works closely with NCES web site personnel to set up and ensure that all transfer protocols are followed and that transfer and retrieval of documents (especially from the contractor’s management website, see task 1.2.5, and the

NCES secure data server on the NCES Member’s website) is completed to the satisfaction of

NCES. The contractor shall conduct a test of its transfer/retrieval system(s), within 4 weeks of the award of the contract and prior to making use of the system for key contract deliverables, and shall eliminate any problems found to ensure a trouble-free system (1.1.7).

For the contract, the contractor shall transmit all correspondence directly to the NCES COR via e-mail that is compatible with Microsoft Outlook. Compatibility will extend to the ability to access attachments sent as part of e-mails. NCES uses Microsoft Office 2007: Word, PowerPoint, and Excel as its primary software packages for communicating documents, charts, and tables. NCES uses the Department of Education’s Microsoft Outlook software as its communications package. All deliverables are to be delivered in native file format, in the version of Word, PowerPoint, Excel, or other software packages used by NCES.

All final deliverables that are to be assigned a NCES publication number or that will be made public through the NCES website will be delivered in PDF format (or another appropriate format approved by the COR) in accordance with the NCES requirements published in

“Policies, Guidelines, Standards and Procedures for Web Publishing for IES and NCES”

(available at https://members.nces.ed.gov/ webstandards/), and must adhere to Section 508 requirements for accessibility. More information on accessibility is available through the

Federal IT Accessibility Initiative at: http://www.section508.gov.

Although the term “draft” is used throughout this PWS, it is important that the contractor understand what the Government means, and does not mean, by “draft.” A draft is a complete product of high quality that the contractor (and the government) can publish with confidence as to its accuracy and usefulness. A “draft” is not simply what is available on the due date— regardless of completeness or quality. NCES anticipates that only minor changes to drafts will be needed for them to become final deliverables.

Every final deliverable will be sent with a cover memorandum signed or initialed by the project director or someone of higher corporate stature than the project director. The cover memorandum will certify that the contractor’s organization stands behind the quality of the product. This holds for all final versions of products—even those prepared primarily by consultants or subcontractors. At no time will the government take direct delivery of a product from a consultant or subcontractor.

https://members.nces.ed.gov/%20webstandards/ http://www.section508.gov/

All draft deliverables will be due at the office of the COR and the CS on or before their due date. Deliverables must be sent in both electronic copy and, if requested by the COR, hard copy format. Hard copies of deliverables must be sent to the COR such that they arrive on or before their due date. Unless otherwise specified, three copies of each hard copy deliverable will be required

Deliverables: 1.1.7 Test of transfer/retrieval system(s)

1.1.8 Study brochures

The contractor shall assist NCES in developing an informational brochure for the study that will describe the study’s general goals. At a minimum, it will identify the purpose of the project, the sponsoring organizations, the types of data to be collected, and the data collection schedule. The brochures will be distributed to individuals and organizations requesting information about the study and will be included as part of the information packet used to secure cooperation of study participants.

The contractor shall submit a draft of the brochure 6 weeks after contract award (1.1.8.1).

The COR will review the drafts and provide comments and suggestions within 2 weeks.

NCES, or the IEA, will provide the contractor with an electronic version of the logo of the study, which the contractor shall use in the production of the brochure and other published materials. Based on initial COR comments, the contractor shall revise the draft of the brochure and will resubmit it for full NCES review (see Exhibit 1, under Task 6, for a description of the NCES review process). Once approved, an electronic version of the brochure will be delivered with the finished copies for inclusion on the NCES web site

(1.1.8.2). The electronic version of the brochure will meet NCES web site design and formatting guidelines, which currently requires PDF and HTML formatting, with graphics as

.jpg or .gif files. The contractor shall produce enough copies of the final, printed brochure for all information packets needed to secure cooperation of study participants and deliver an additional 50 copies to the COR for NCES. For the purposes of responding to this proposal, the contractor should assume, for administering TIMSS without options, of needing to contact about 700 schools, 350 school districts, and 40 states

Deliverables: 1.1.8.1 Draft informational brochure; 1.1.8.2 Final informational brochure

1.2 Contract management

Because this contract includes a series of time-sensitive and complex activities, it is critical that it be monitored and managed effectively. To that end, NCES requests the following general study management activities.

1.2.1 Project update meetings with contractor

For the life of the contract, the contractor will hold regular weekly or biweekly (or less frequently, if the COR so determines) meetings for key contractor staff to update the COR or other NCES staff on the project’s progress. The COR may propose to hold these meetings via telephone conference calls, over the Internet, or face-to-face if the meeting is critical. For budget purposes, the contractor shall plan up to 40 meetings a year throughout the contract, of which up to 12 meetings a year may be face-to-face. The contractor shall participate in the development of the goals and agenda for each meeting. The contractor shall be responsible for distributing the agenda and other meeting materials, including the current version of the project’s master schedule, to no more than 12 meeting attendees. Materials to be distributed will be approved, in advance, by the COR.

Deliverables: 1.2.1 Regular project meetings materials including master schedule

1.2.2 Other meetings with NCES/ED

The contractor shall send one representative to attend all National Research Coordinator

(NRC) meetings for the study, as well as other technical meetings held by the

IEA/International Study Center. The contractor is strongly urged to send the Project

Director/Principal Investigator to the meetings, or the staff member who has primary responsibility for conduct of the study. Additional personnel may be sent, if needed, upon approval of the COR. For the study, the contractor should assume 9 international NRC meetings in a foreign location over the course of the contract.

Deliverables: 1.2.2 Attendance at NRC meetings and technical meetings

1.2.3 Monthly study progress and financial reports

The contractor shall submit to the COR and CS a regular monthly progress report, by the 10th day of the month following the month the report covers. This report shall be delivered before or concurrent with the contractor’s monthly billed invoice so that it is available to check and approve each month’s invoice. At a minimum, in each monthly progress report, the contractor shall include information on the following aspects of the study and its operation:

The past month’s work, including a description of continuing work, new work, and completed work, with key dates on which tasks or activities were initiated or completed during the month;

The upcoming month’s work, including a description of the work and key dates by which time tasks or activities will be initiated or completed during the month;

Major accomplishments for the month;

List of products and milestones submitted or achieved during month, with a short description of each; and

Problems/Issues and how each has been or will be addressed by the contractor.

The contractor shall also submit to the COR and CS a regular monthly financial data report by the 10th day of the month following the month the report covers. This report shall be delivered before or concurrent with the contractor’s monthly billed invoice so that it is available to check and approve each month’s invoice. In each monthly financial data report, the contractor shall, at a minimum, include

(1) information on the following aspects of the study and its operation:

Past month’s cost of work completed, by task, subtask, and cumulatively;

Percent of contracted amount spent to date, by task;

Estimated cost to complete, total estimated cost, and total contracted cost;

Difference between the total estimated cost and the contracted cost; and

Staff hours spent on each task during the prior month;

(2) appropriate records and information to permit the COR and CS to certify that the services that are listed on monthly bills are actually used and are for official purposes; and

(3) a manpower report prepared and signed by the project director that summarizes actual personnel assignments for the month just completed, showing for each named individual the hours charged, by task. For individuals not named in the contract, the manpower report will show hours spent by position (e.g., data collectors). All hours billed to the contract will be summarized in the report.

The contractor shall include detailed information in the monthly financial data reports on any anticipated or actual cost overruns or cost savings, whether incurred within a single task or cumulatively. From this information, the contractor shall ensure that the COR is clear about the progress being made on tasks, the work that remains to be accomplished, the financial resources used and available, and any problems and proposed solutions. At the request of the

CO, if additional issues need to be addressed in either monthly report, the contractor shall submit an addendum within 1 week of the request.

The format for the monthly report will be consistent from month to month. The same format will be used to report on any subcontractor activity costs.

The monthly report shall serve as a justification for all work billed for the month. Therefore, the contractor shall take care to ensure not only that all work billed for the month is described in sufficient detail in the monthly report to support an audit, but that all detailed information

(on completed work) contained in the monthly report is transparently and easily associated with its respective cost or billing.

Deliverables: 1.2.3 Monthly study progress and financial reports

1.2.4 Internal Project management portal

No more than 1 month after the delivery of the first deliverable, the contractor shall establish, and thereafter maintain, an internal server to house all project reports, interim as well as final versions of deliverables, and other appropriate project documentation. The space shall be accessible to the COR, CO, and CS, and other appropriate contractor or outside staff. This will aid in ensuring that all project staff have the most up-to-date versions of materials and will complement (but not replicate) websites maintained by the international study centers.

The space shall be password protected. No data collected from the respondents shall be placed on the project management portal.

Deliverables: 1.2.4 Project portal for housing electronic copies of project materials

1.2.5 Maintenance of data files

The contractor shall maintain complete data files for all components of TIMSS 2015, and all supplemented files created and obtained in support of the study, for the life of the contract.

Maintenance of data files means keeping data file documentation available for reproduction on request of the COR or other authorized individuals. Certain data files that will be maintained will not be made available to the public because of privacy considerations. In these cases, file maintenance means that the data files will be kept intact, in a form that can be updated or copied. Data files will not be allowed to expire, to be released, overwritten, or otherwise destroyed. These files will be maintained with a software system capable of interlinking them for analysis (or other) purposes while, at the same time, preserving the necessary confidentiality requirements. If prior TIMSS datasets are required by the contractor, the COR will arrange for delivery for those datasets.

1.2.6 Relationship of the contractor to NCES and other agencies

The contractor shall report all activities for this contract and activities related to TIMSS 2015 to the COR. All data, publications, and products of the study are the sole property of the U.S.

government, and shall be provided to it as specified below and upon request. Under no circumstances will the contractor, or any subcontractors, distribute or release any data, publications, or products of the study in any form to any individual or party without prior permission of the COR. Any contractor or subcontractor conference presentations or papers, based on this study, shall be reviewed and cleared by NCES prior to any such presentation.

This restriction will remain in effect throughout the life of the contract, and until NCES officially releases the data, publications, or products to the public.

The contractor shall work through the COR in all relationships and discussions with the IEA, the International Study Center, foreign governments, foreign education professionals, and all other agencies and individuals related to the study. The contractor shall not convey to any of these entities information that may be construed as NCES, IES, or Department of Education policy, except where related to technical aspects of the study or where given explicit permission by the COR.

1.2.7 Readiness for work under conditions of unusual speed and flexibility

In the event that the study’s international plans and time schedules change rapidly due to various necessities, the contractor shall be prepared to work with changing requirements. For example, time schedules may become very tight because the IEA are unable to meet their projected study timelines. Whenever necessary, the contractor shall be prepared to perform activities under conditions of emergency speed. Emergency speed will be defined according to four priority levels:

Priority 1: within 24 hours of request

Priority 2: within 72 hours of request

Priority 3: within 5 working days of request

Priority 4: within 10 working days of request

In the event that changes to plans or time schedules require the completion of a task or activity under emergency speed, the COR will notify the contractor of the request, after consultation with the Contracting Officer (CO), and identify the timeframe (i.e., priority level) under which the task or activity will be completed by the contractor.

1.2.8 Quality Control Measures.

NCES is firmly committed to determining how well each project meets its goals, the quality of data collected (including response rates at the school and the student levels), and the implications of the experiences and outcomes of the project for future projects. Project quality control should be integrated into every phase of the project. For instance, potential problems should be identified early so that possible solutions are tested as part of the field test. Item reliability and validity tests for questionnaires and assessments shall be built into the data collection efforts, and results from these tests shall inform data collection and processing.

Nonresponse analyses shall be conducted as part of the weighting process, and information about the survey methodology and sample design effects shall be made available to data users.

Quality control also includes (1) managing staff in the field to ensure the high quality of their performance; (2) instituting procedures relevant to confidentiality and protection of data

(including the secure transmittal of confidential data); and (3) ensuring deliverables are submitted on time and are of sufficient quality (in terms of accuracy, coherence, clarity, and comprehensiveness) to meet NCES statistical standards, IEA standards, professional standards, and NCES and IES style guidelines. As noted earlier, NCES expects that only minor changes to drafts will be needed for products to become final deliverables.

Again, all work conducted under this contract shall, at a minimum, meet the standards and guidelines set forth in the publication NCES Statistical Standards, and amendments implemented since that time. The standards can be accessed at http://nces.ed.gov/statprog/.

The IES guidelines can be accessed at http://nces.ed.gov/statprog/style.asp.

1.3 Obtain OMB clearance for field test data collection

The contractor shall prepare a clearance package for obtaining OMB approval of the TIMSS

2015 field test and non-cognitive survey instruments that will be used in the study, and any other materials that will be distributed to districts, schools, teachers, students, and parents

(e.g., parent permission letters), in accordance with current OMB, ED, and NCES requirements for such packages.

http://nces.ed.gov/statprog/style.asp

Preparing an OMB clearance package entails assembling an extensive amount of information to (a) justify the data collection as a whole as well as individual background questions; (b) estimate the cost of the data collection to the federal government and the amount of burden (in both time and cost) on respondents; and (c) explain the data’s uses, efforts taken to minimize the burden on respondents, any provisions for payments or gifts to respondents, steps to assure confidentiality, and the statistical methods that are to be employed in collecting and preparing the data (e.g., dealing with nonresponse). OMB requires that all this information be written up in a Supporting Statement. An OMB clearance package for submission must include the Supporting Statement, copies of all materials that will go to respondents (such as recruitment materials, survey questionnaire, letters of consent, etc.), and information on any national adaptations of the instruments (see subtask 2.3). The contractor shall prepare a draft

OMB clearance package for the field test no later than August 1, 2013, and preferably earlier.

After COR review and internal NCES review, the contractor shall revise the draft OMB clearance package within 1 week of receipt of comments. NCES will submit the revised OMB clearance packages to the Department and OMB for review and approval. Failure to submit the OMB clearance package by the deadline may result in the contractor being unable to collect data in a timely manner. The contractor shall be aware that the time between submission of the OMB clearance package and receipt of a clearance number from OMB for inclusion on all survey documents takes a minimum of four months, and upwards of six months, sometimes more. The contractor shall take these time constraints into consideration when preparing and submitting the OMB clearance package. The contractor shall work diligently and expeditiously with the COR to seek OMB clearance prior to recruiting schools.

In preparing the OMB clearance package, the contractor shall follow instructions for completing the Information Collection Submission Worksheet and writing the Supporting

Statement required for Paperwork Reduction Act submissions (for information on the requirements for a clearance package, officially called at present an Information Collection

Review, see http://www.whitehouse.gov/omb/inforeg_infocoll/). The contractor shall prepare all written materials needed in the formal clearance and approval process according to OMB requirements as outlined in the Information Collection Submission Worksheet. The package will include the U.S.-adapted background questionnaires but not the cognitive (assessment) instruments for the study. This approval process is expected to involve the following series of steps (called out as deliverables):

Contractor shall provide a draft package to the COR and shall revise it within a week of receiving the COR’s comments (1.3.1).

The COR provides the package to the NCES OMB liaison and chief statistician, and the contractor shall revise package according to comments received within 5 business days

(1.3.2).

The NCES OMB liaison provides the package to the Department’s OMB liaison office

(RIMS) and the contractor shall revise the package according to comments received within 5 business days (1.3.3).

ED provides package to OMB; the contractor shall answer OMB questions (1.3.4) within http://www.whitehouse.gov/omb/inforeg_infocoll/

5 business days (there may be more than one round of OMB questions) and revise the package according to final comments received within 3 business days (1.3.5).

The contractor staff shall be prepared to join the COR in person or via conference call for a review of the materials with Department and OMB staff. The contractor shall ensure that authorization by OMB has been obtained before data collection begins. All data collection forms will bear the approval number assigned by OMB.

The contractor shall be aware that states, districts, and schools may not be contacted until

OMB has approved the study plan. The contractor shall take this into consideration when planning the field tests and main data collections.

Following approval, the contractor shall provide NCES/ED with one hard copy and one electronic copy of OMB’s official approval or clearance for the study (1.3.6).

Deliverables: 1.3.1 Draft OMB field test clearance package; 1.3.2 Revised OMB field test clearance package; 1.3.3 Revised package based on RIMS comments; 1.3.4 Responses to

OMB questions on package; 1.3.5 Revised final field test clearance package for OMB approval; 1.3.6 OMB clearance for field test

1.4 Obtain OMB clearance for the main study’s data collection

OMB may grant clearance for both the field test and the main study activities or may require that after the field test a new clearance be obtained for the study. In the case that new clearance is required, the COR will provide guidance on the type of materials needed to obtain this clearance, which may include either a memo describing any changes made in instruments or procedures from the field test to the main study, or a full new clearance package similar to that prepared for the field test. In either case, the requirements and steps for approval, as described above in task 1.3, shall apply to obtaining the OMB clearance for the main study. The contractor shall provide either the memo or new clearance package at least 7 months before the scheduled start of the main study data collection.

Again, the contractor may not begin contacting states, districts, and schools until OMB has approved the study plan. This point is of particular importance in planning for the main assessment as schools typically need to be recruited early in the school year for them to be able to participate in a late spring assessment like TIMSS. The contractor shall take this into consideration when planning the main data collection schedule.

Deliverables: 1.4.1 Draft OMB main assessment memo or clearance package; 1.4.2 Revised

OMB main assessment memo or clearance package; 1.4.3 Revised memo or package based on

RIMS comments; 1.4.4 Responses to OMB questions on memo or package; 1.4.5 Revised final main assessment memo or clearance package for OMB approval; 1.4.6 OMB clearance for main assessment

1.5 Prepare for briefings on project

1.5.1 Prepare project briefing materials

From time to time throughout the course of the project, NCES will provide interested individuals and agencies with information about the nature, progress of, and findings of the study. In consultation with the COR, the contractor shall be prepared to provide up to 5 briefings/presentations about the study over the life of the contract. The COR will notify the contractor at least 2 weeks in advance of each briefing or presentation. The materials for such a briefing/presentation will be no more than 10 typewritten pages of text (double-spaced) with accompanying graphs and tables suitable for display. Requested materials may include overhead slides for making presentations regarding the study. Within 1 week of a request, a set of materials will be submitted to the COR for review and comment. The contractor shall make necessary revisions to the materials based on this review and resubmit the materials within 3 days of receipt of comments.

Deliverables: 1.5.1 Make presentations on TIMSS project

1.5.2 Provide briefings or presentations

The contractor shall also be prepared to provide up to 10 presentations, briefings, or demonstrations about the study over the life of the contract. Examples of the kinds of meetings at which the contractor shall be asked to discuss the study are the annual meetings of the American Educational Research Association (AERA) and NCES’ Summer Data

Conference. They might also include briefings with other federal agencies, staff of the U.S.

Congress, or the media. The COR will notify the contractor at least 2 weeks in advance of each briefing or demonstration.

Deliverables: 1.5.2 Prepare project briefing materials

Task 2. Preparations for Assessments (Firm Fixed Price)

Develop field test and main study school sampling plans and select samples

The contractor shall develop school and student sampling plans for the field test and main study that meet established international requirements for sample size and design (including use of replacement schools; see the school sampling guidelines in appendix C). For the field tests, the contractor shall select the field test school sample as well as the student sample following international standards as closely as possible. For the main study, the contractor shall prepare and forward sampling documentation and data files, by the established international deadline, to the international contractor, who shall select the main study sample.

The contractor shall work with NCES and the IEA to finalize the sample plan for the field test and the main assessment. The contractor shall detail potential population coverage problems associated with any proposed sample design.

The contractor shall meet NCES and IEA standards when developing sampling plans. NCES standards are available at http://nces.ed.gov/statprog/. NCES standards may be revised during the life of the contract. The contractor shall meet NCES standards in effect at the time of data collection and data reporting.

2.1 Field test sampling plan and sample

The sample for the field test may be a purposive sample of schools chosen by the contractor.

Although not a probability sample, the following guidelines need to be followed:

a) For each target grade level, the sample size should be approximately 1,400 students from between 35-40 schools.

b) To the extent possible, the field test sample should cover a range of schools types

(e.g., public, private, charter, magnet); the samples must include schools that have students in the appropriate target grades (grades 4 and 8); the sample should include schools with different demographic and socio-economic groups in the population of fourth- and eighth-graders that are generally representative of the national school population (e.g., different geographic regions, urban and rural locales, low and high minority populations).

The field test sampling plan needs to be submitted to and approved by NCES, OMB, and the

IEA. Only the approved field test sampling plan is to be implemented by the contractor. The contractor shall also complete all field test sampling forms, in accordance with the international schedule, as…