PIAAC PWS draft 2012.pdf

DRAFT

Performance Work Statement Integrated Adult Assessment

U.S. National Data Collection Supplement

Contents Introduction

Purpose of Procurement

Enabling Legislation

Statement of Work

Task 1. Contract Administration (Firm-fixed Price)

1.1 Project Initiation and Planning

1.2 Contract management

1.3 Obtain OMB clearance for data collection

1.4 Briefings and descriptive materials

1.5 Relationship of the contractor to NCES and other agencies

1.6 Readiness for work under conditions of unusual speed and flexibility

1.7 Materials from PIAAC 2011 contractor

Task 2. Preparations for Assessments (Firm-fixed Price)

2.1 Develop sampling plan and select sample

2.2 Adapt and prepare survey instruments and procedures

2.3 Testing of instruments

2.4 Use expert advice

Task 3. Data Collection and Delivery (Cost-reimbursable)

3.1 Recruit and train data collectors

3.2 Collect data

3.2.1 Implement data collection

3.3 Produce dataset

3.4 Data scaling, and estimation

Task 4. Score assessment and code questionnaire data (Cost-reimbursable)

Task 5. Prepare and Disseminate Results {FFP}

5.1. Analysis plans for topical reports

5.2. Produce reports

5.3 Disseminate results

PIAAC U.S. National Data Collection Supplement PWS - 2

Task 6. Develop Plan for Bridging Study {FFP}

6.1 Preliminary design memo

6.2. Final design memo

Task 7. Develop Plan for a Functional Writing/PIAAC Literacy (firm fixed price)

7.1 Preliminary design memo

7.2. Final design memo

Contract Options

Task 8. (optional) Bridging Study

Task 9. (optional) Additional domain

Task 10. (optional) Additional sample

Period of Performance

Renewal of contract for up to an additional 60 months:

Appendix A. Tentative Timeline

Appendix B. Data Confidentiality, Security, Quality Control, and Contractor Staff Security

Clearances

B-1. Overview of IES/NCES Confidentiality Statutes

Confidentiality of Individuals and Institutions

Protection and Security of Data

B-2. Data Security Plan

Definition of Personally Identifiable Information and Direct Identifiers

Development of Data Security Plan

Implementation of Data Security Plan

B-3. Adherence to NCES Statistical Standards

NCES Statistical Standards

B-4. Quality Control Measures Requirement

Quality Control Measures

B-5. Contractor Employee Security Screening

Contractor Employee Security Screening Requirements

B-6. Requirement of Affidavit of Nondisclosure

Affidavit of Nondisclosure

B-7. Security and Confidentiality Training for Contractor Staff

Security and Confidentiality Training

B-8 Information Collection Request (ICR) Requirement

Information Collection Request (ICR)

Appendix C. Sampling guidelines

PIAAC U.S. National Data Collection Supplement PWS - 3

Appendix D. Hardware Requirements

Appendix E. Disclosure Analysis Plan

Information checklist for disclosure analysis plans: Public-use microdata files

Appendix

Appendix F. NCES Review Process

PIAAC U.S. National Data Collection Supplement PWS - 4

Introduction

The National Center for Education Statistics (NCES) in the Institute of Education Sciences (IES) of the U.S. Department of Education (ED) supports a variety of international activities to collect and report data on education in the United States and other countries. It is the goal of NCES to provide comparative information about the U.S. population‘s educational performance, competencies, and skills in an international context as well as about the relationships of such with the nation‘s economic and workforce development. NCES participates in international assessments of adult skills and competencies in order to:

provide comparable indicators on adult literacy, competencies, and education levels across countries in order to benchmark the U.S. population‘s literacy, competencies, and education levels; and within the United States, document relationships among adult literacy, educational enrollment and attainment, and occupational outcomes, and support correlational analyses of the factors that may influence these relationships.

The study covered by this procurement is part of the larger international program that NCES has participated in through collaboration with a variety of international organizations and partners, such as the Organization for Economic Cooperation and Development (OECD), the International

Association for the Evaluation of Educational Achievement (IEA), and ministries of education and other government agencies throughout the world.

Through its participation, NCES seeks to strengthen the quality, validity, consistency, and timeliness of international data. To continue this effort, the United States must follow through with well-organized and well-executed data gathering activities within our national boundaries.

These efforts will allow NCES to provide policy-relevant data for international comparisons of the U.S. adult population‘s competencies and skills, and to help inform decision making on the part of national, state, and local policy makers, especially those concerned with economic development and workforce training.

Purpose of Procurement

The Government requires a contract to implement a data collection supplement to U.S.

participation in the OECD Program for the International Assessment of Adult Competencies

(PIAAC). PIAAC is an international household study of literacy, numeracy, literacy components, and problem-solving in technology-rich environments. PIAAC is a cooperative venture managed by participating OECD countries. The project is steered by a Board of Participating Countries

(BPC), which is supported by staff of the OECD Secretariat. The goal of PIAAC is to enhance our understanding of the effectiveness of education and training systems in developing basic cognitive skills and key generic work skills. For younger cohorts, PIAAC will complement the results of the Program for International Student Assessment PISA by providing measures of skill following completion of initial education. For older cohorts, PIAAC will allow examination and analysis of the effectiveness of education and skill formation systems in supporting skills

PIAAC U.S. National Data Collection Supplement PWS - 5 development over the lifecycle. More information about PIAAC can be found at www.oecd.org/piaac.

The overall goal for the procurement is to supplement the PIAAC U.S. national data collection undertaken in 2011 with (1) an additional sample of U.S. adults to produce, when combined with the original 2011 U.S. PIAAC sample, a representative U.S. sample capable of supporting analyses of key U.S. subgroups outlined in this performance work statement; (2) a bridging study that establishes a statistical linkage between PIAAC and the 2003 National Assessment of Adult

Literacy (NAAL); and (3) administration of an assessment of additional adult literacy domains not included in PIAAC 2011. The work of this procurement will include

Design and implementation of a data collection to produce national PIAAC estimates for young adults (ages 16-24 and 25-34) and the unemployed, and to improve the representativeness of the sample by including incarcerated adults;

Design of a bridging study to establish a statistical linkage between PIAAC 2011 and

NAAL 2003 in literacy, as well as an option to implement the study; and

Design of an assessment of adult functional writing paired with the PIAAC Literacy

Component, as well as an option to implement the study.

To ensure alignment with the original PIAAC sample and high data quality, these studies must follow the established design and standards as set forth by the OECD, as well as NCES standards. These design and standards will cover, but are not limited to, administration procedures, sampling guidelines and requirements, survey instruments, data collection, calculation of response rates, data coding and processing. The contractor shall follow the international guidelines for PIAAC 2011 and shall meet established NCES standards and procedures (NCES standards can be found at http://nces.ed.gov/statprog/). NCES will provide the international guidelines.

The contractor shall be clear that NCES is the sole authority to manage PIAAC in the United

States and to negotiate with the OECD on any issues related to the study. The contractor shall not use PIAAC items for any other purposes without NCES‘ written consent. Under no circumstances shall the contractor conduct any activities related to PIAAC, including conducting the assessment in sub-national jurisdictions, without permission from NCES.

Enabling Legislation

The authority for this study can be found in NCES‘ longstanding mission to ―collect, and analyze and disseminate statistics and other information related to education in the United States and in other nations‖ (stated in section 406 of the General Education Provisions Act, as amended (20

U.S.C. 1221e-1) and in the Education Sciences Reform Act of 2002 (P.L. 279-107, Part C, Sec.

153):

(a) GENERAL DUTIES.-The Statistics Center shall collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including—

(1) collecting, acquiring, compiling (where appropriate, on a State-by-State basis), and http://www.oecd.org/piaac http://nces.ed.gov/statprog/

PIAAC U.S. National Data Collection Supplement PWS - 6 disseminating full and complete statistics (disaggregated by the population characteristics described in paragraph (3)) on the condition and progress of education, at the preschool, elementary, secondary, postsecondary, and adult levels in the United States, including data on…(D) secondary school completions, dropouts, and adult literacy and reading skills.

See http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=107_cong_public_laws&docid=f:publ279.107.pdf for the full Education

Sciences Reform Act.

Scope of Work

Task 1. Contract Administration (Firm-fixed Price)

The contractor shall work closely with the Contracting Officer‘s Representative (COR) to implement the design of the study included under this procurement and the approach presented in the proposal.

1.1 Project Initiation and Planning

The contractor shall work closely with the Contracting Officer‘s Representative (COR) to implement the design of the study included under this procurement and the approach presented in the proposal.

1.1.1 Meet with NCES and hand-off

Within 2 weeks of the contract start date, the contractor shall meet with the COR, CO, and CS to review the contract tasks and to discuss issues related to the conduct the study. The meeting will be held in Washington, DC, at the offices of NCES. This meeting will be attended by the overall project director and no more than two other key contractor staff.

1.1.2 Produce master schedule

The contractor shall produce a project schedule that details milestones (using critical dates specified in appendix A and the ―Schedule of Deliverables‖) such as Office of Management and

Budget (OMB) clearance, instrument revision and adaptation, assessment administrator training, data collection and analysis, as well as reporting. This schedule will separate the tasks throughout the contract period. This schedule is required with the submission of the proposal and will be reviewed by NCES before the contract is awarded.

The project schedule shall include the following information in an easy-to-understand format:

Start and end dates of all tasks;

Key milestone dates;

Completion dates of sub-tasks, drafts, or other events;

Key deliverable dates.

http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=107_cong_public_laws&docid=f:publ279.107.pdf%20 http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi?dbname=107_cong_public_laws&docid=f:publ279.107.pdf%20

PIAAC U.S. National Data Collection Supplement PWS - 7

In addition, the contractor shall make clear the primary leads and contact person for each task or deliverable (if applicable).

The contractor shall submit a revised master schedule within 2 weeks of contract award. The contractor shall be prepared to update the schedule throughout the contract upon request by the

COR. At the request of the COR, the contractor shall provide an up-to-date master schedule within 24 hours of the request.

1.1.3 Project Management Plan

The contractor shall prepare and submit a project management plan to the COR within 3 weeks of contract award. The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the project management plan is approved by the COR, the contractor shall manage the project in accord with the approved plan and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

The plan shall detail the management techniques that will be employed to ensure timely, efficient, and cost effective data collection, processing, analysis, and reporting. The plan shall reflect the requirements specified in the contract or describe and justify any revisions to the original plan specified in the contract and approved by NCES. These might include changes in data collection approach, data processing, plans for analysis and reporting, as well as changes in the proposed project organization, staffing plan, and schedule. The plan will fully describe the rationale for any proposed changes.

1.1.4 Staffing Plan

The contractor shall prepare and submit a staffing plan to the COR within 3 weeks of contract award. The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the staffing plan is approved by the COR, the contractor shall implement the plan and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

The staffing plan shall detail the project‘s overall staffing, including contractors and subcontractor staff dedicated to the study, as well as the training or preparatory activities to ensure that all staff are familiar with and abide by the requirements of the study (including, but not limited to, the study‘s need for high quality and timely completion of all activities and for strictly enforced confidentiality procedures).

The ability of the COR to obtain timely and accurate information and decisions from contractor staff is considered of the utmost importance, and will remain a high priority of the contractor.

The contractor shall be responsible for the timely and satisfactory completion of work carried out by subcontractors, and shall convey to subcontractors the need to comply with the rules, regulations, timelines, and deliverable schedules of all tasks and sub-tasks included under this procurement.

PIAAC U.S. National Data Collection Supplement PWS - 8

1.1.4.1 Staff Security Requirements: Affidavits of Nondisclosure, Security and Confidentiality Training, and Background Checks

All contractor staff to work on the project for 30 days or more must complete a series of security forms and undergo a federal background check per OMB requirements, with higher levels of security clearance required for those who visit homes or who may come in contact with confidential data. The contractor shall ensure that all staff who may come in contact with confidential data shall (a) be trained in the proper handling of confidential data and (b) first sign and then notarize an affidavit of nondisclosure before having access to any data with confidential information. The contractor shall execute these affidavits of nondisclosure and retain them in a secure location. The contractor shall provide a list to the COR of all the individuals who have signed affidavits, the position in the organization of the person, and the person‘s relationship to the study, within 4 weeks of the award of the contract.

As new persons are assigned to the project, the security forms shall be filled out and an affidavit shall be executed for them on the first working day of assignment to the study and an updated list provided to the COR, no less than on a quarterly basis. The contractor shall notify the COR if any contractor staff leave the project. The contractor shall be aware of and comply with all contractor personnel security regulations and all Department IT security policy requirements for personnel, as described in appendices B and E.

1.1.5 Data Security Plan

The contractor shall prepare and submit a data security plan to the COR within 3 weeks of contract award. The COR will provide comments on the plan within 2 weeks of receipt. The contractor shall incorporate the changes necessary to respond to the concerns raised by the COR and submit a revised plan within 3 weeks of receipt of COR comments. Once the data security plan is approved by the COR, the contractor shall implement the plan and, if a change to the plan is needed, shall notify the COR of such change(s) before implementing such.

The security of this study‘s data is of utmost importance to NCES and the security plan will be reviewed by NCES‘ Chief Statistician. For a detailed discussion of the data security requirements of the contract, see appendix B.

1.1.6 Transmittals of study products and quality of deliverables The contractor shall transfer and retrieve study-related documents and files through the Internet, to or from NCES, or other designated contractors. The ability to transfer and retrieve documents through the Internet is important to maintaining the integrity of electronic versions of documents that may eventually be posted on the NCES web site or distributed internally. Thus, it is imperative that the contractor works closely with NCES web site personnel to ensure that all transfer protocols are followed and that transfer and retrieval of documents (especially from the contractor‘s management portal, see task 1.2.4) is completed to the satisfaction of NCES. The contractor shall test the transfer/retrieval system prior to making use of the system to eliminate problems.

For the contract, the contractor shall transmit all correspondence directly to the NCES COR via e-mail that is compatible with Microsoft Outlook. Compatibility will extend to the ability to access attachments sent as part of e-mails. As of 2012, NCES uses Microsoft Office 2010: Word, PowerPoint, and Excel as its primary software packages for communicating documents, charts, PIAAC U.S. National Data Collection Supplement PWS - 9 and tables. NCES uses the Department of Education‘s Microsoft Outlook software as its communications package. All deliverables are to be delivered in the version of Word, PowerPoint, Excel, or other software packages used by NCES.

All final versions of reports that are to be assigned NCES publication numbers will be delivered in PDF format in accordance with published in ―Policies, Guidelines, Standards and Procedures for Web Publishing for IES and NCES‖ (available at https://members.nces.ed.gov/ webstandards/). Additionally, all files will be delivered in native file format (e.g., Microsoft

Word, Excel, etc.). These publications include the data documentation, the field test report, and the OMB clearance package.

Although the term ―draft‖ is used throughout this PWS, it is important that the contractor understand what the Government means, and does not mean, by ―draft.‖ A draft is a complete product of high quality that the contractor (and the government) can publish with confidence as to its accuracy and usefulness. A ―draft‖ is not simply what is available on the due date— regardless of completeness or quality. NCES anticipates that only minor changes to drafts will be needed for them to become final deliverables.

Every deliverable will be sent with a cover memorandum signed or initialed by the project director or someone of higher corporate stature than the project director. The cover memorandum will certify that the contractor’s organization stands behind the quality of the product. This holds for all products—even those prepared primarily by consultants or subcontractors. At no time will the government take direct delivery of a product from a consultant or subcontractor.

All draft deliverables will be due at the office of the COR and the CS on or before their due date.

Deliverables must be sent in both electronic copy and, if requested by the COR, hard copy format. Hard copies of deliverables must be sent to the COR such that they arrive on or before their due date. Unless otherwise specified, three copies of each hard copy deliverable will be required. Contractors must justify use of and obtain prior approval for use of any delivery service more expensive than USPS priority mail.

1.1.7 Study brochures

The contractor shall assist NCES in developing an informational brochure for the study that will describe the study‘s general goals. At a minimum, it will identify the purpose of the project, the sponsoring organizations, the types of data to be collected, and the data collection schedule. The brochure will be sent in advance to households in the sample and will be distributed to individuals and organizations requesting information about the study. The contractor is welcome to base the brochure on the brochure used for the PIAAC 2011 data collection. English-language and Spanish-language versions will be required.

The contractor shall submit a draft of the brochure 6 weeks after contract award. The COR will review the drafts and provide comments and suggestions within 2 weeks. NCES, or the OECD, will provide the contractor with an electronic version of the logo of the study, which the contractor shall use in the production of the brochure and other published materials. Based on initial COR comments, the contractor shall revise the draft of the brochure and will resubmit it for full NCES review (see appendix F for a description of the NCES review process). Once http://webstandards/ http://webstandards/

PIAAC U.S. National Data Collection Supplement PWS - 10 approved, an electronic version of the brochure will be delivered with the finished copies for inclusion on the NCES web site. The electronic version of the brochure will meet NCES web site design and formatting guidelines, which currently requires PDF and HTML formatting, with graphics as .jpg or .gif files. The contractor shall supply the COR with copies of the final, printed brochure for the study as requested.

1.2 Contract management

Because this contract includes a series of time-sensitive and complex activities, it is critical that it be monitored and managed effectively. To that end, NCES requests the following general study management activities:

1.2.1 Project update meetings with contractor and other meetings with NCES/ED Key contractor staff shall travel to Washington, DC for meetings with the COR throughout the contract period, on a quarterly basis. The contractor shall participate in the development of the goals and agenda for each meeting. The contractor shall be responsible for distributing the agenda and other meeting materials to no more than 20 meeting attendees. Materials to be distributed will be approved, in advance, by the COR.

1.2.2 General study management and progress reports

Under this task, the contractor shall perform its general study management functions. A routine aspect of this management will be the development and submission of regular reports for the study to the COR.

1.2.2.1 Monthly study progress and financial reports

The contractor shall submit to the COR and CS a regular monthly progress report by the 10 th day of the month following the month the report covers. At a minimum, the contractor shall include in each monthly progress report information on the following aspects of the study and its operation:

Past month‘s work description, including continuing work, new work, and completed work, with key dates on which tasks or activities were initiated or completed during the month;

Upcoming month‘s work description, including key dates by which time tasks or activities will be initiated or completed during the month;

Major accomplishments for the month;

List of products and milestones submitted or achieved during month, with a short description of each; and

Problems/Issues and how each has been or will be addressed by the contractor.

The contractor shall also submit to the COR and CS a separate monthly financial data report by the 10 th day of the month following the month the report covers. In each monthly financial data report, the contractor shall, at a minimum, include

PIAAC U.S. National Data Collection Supplement PWS - 11

(1) information on the following aspects of the study and its operation:

Past month‘s cost of work completed, by task and cumulatively;

Past month‘s budgeted cost for the month, by task and cumulatively;

Percent of contracted amount spent to date, by task;

Estimated cost to complete, total estimated cost, and total contracted cost;

Difference between the total estimated cost and the contracted cost; and

Staff hours spent on each task during the prior month;

(2) appropriate records and information to permit the COR and CS to certify that the services that are listed on monthly bills are actually used and are for official purposes; and

(3) a manpower report prepared and signed by the project director that summarizes actual personnel assignments for the month just completed, showing for each named individual the hours charged, by task. For individuals not named in the contract, the manpower report will show hours spent by position (e.g., data collectors). All hours billed to the contract will be summarized in the report.

The contractor shall include detailed information in the monthly financial data reports on any anticipated or actual cost overruns or cost savings, whether incurred within a single task or cumulatively. At the request of the CO, if additional issues need to be addressed in either monthly report, the contractor shall submit an addendum within 1 week of the request.

The format for the monthly report will be consistent from month to month. Also, the same format will be used to report on any subcontractor activity costs.

The monthly report shall serve as a justification for all work billed for the month. Therefore, the contractor shall take care to ensure not only that all work billed for the month is described in sufficient detail in the monthly report to support an audit, but that all detailed information (on completed work) contained in the monthly report is transparently and easily associated with its respective cost or billing.

1.2.2.2 Weekly data collection progress reports

During the data collection period, in addition to the monthly report, the contractor shall submit to the COR a weekly report on the data collection activities. The data collection period will be understood to include the time spent collecting assessment and/or questionnaire data. At a minimum, the contractor shall include the following information in each weekly data collection report:

The number of cases for which contact has been attempted and the number for which contact was established during the report‘s week period;

PIAAC U.S. National Data Collection Supplement PWS - 12

The number of cases that have completed the assessment, that have changed date for the assessment, that refused to participate, that are pending, and that are incomplete;

Cumulative statistics on response rates and follow-up actions, both overall, by individual data collector, and by mode of collection;

Assessment of progress in securing the cooperation of adults in the sample;

Issues that arise with non-responding and non-participating adults; and

Other difficulties or problems encountered in the field.

As participation rates in the study are critical to the overall success of the study, the contractor shall inform the COR of any emerging issues that could negatively impact participation rates. To make sure that this information is relayed in a timely manner, the contractor shall ensure that the

COR is aware of any pressing issues immediately rather than waiting to include the information in the weekly report.

These weekly data collection progress reports will be submitted to the COR by COB every other

Monday during the data collection period.

1.2.2.3 Bi-weekly data scoring and coding progress report

During the data scoring and coding period of the study, the contractor shall submit to the COR a bi-weekly report in addition to the monthly report. The data scoring and coding period will be understood to include the time coding and/or scoring assessments and background questionnaires. At a minimum, the contractor shall include the following information in each bi-weekly data scoring and coding report for the study:

The status of data scoring and coding efforts;

An estimate of the percentage of data scored or coded and how much data remains to be scored or coded;

The inter-rater reliability for scoring and coding to date, and how it was calculated; and

Problems encountered and how they were or will be resolved, including ones that have consequences for cost containment or the time schedule of the study.

These bi-weekly data scoring and coding progress reports will be submitted to the COR by COB every other Monday during the data scoring and coding period.

1.2.4 Internal Project management portal

No more than 1 month after the delivery of the first deliverable, the contractor shall establish, and thereafter maintain, space on an internal server to house all project reports, interim and final versions of deliverables, and other appropriate project documentation. The space shall be accessible to the COR and other appropriate contractor or outside staff. This will aid in ensuring that all project staff have the most up-to-date versions of materials. The space shall be password protected with access limited to NCES, contractor staff, and key subcontractor staff as

PIAAC U.S. National Data Collection Supplement PWS - 13 appropriate. No data collected from the respondents shall be placed on the project management portal.

1.2.5 Maintenance of data files

The contractor shall maintain complete data files for all components of the contract, and all supplemental files created and obtained in support of the study, for the life of the contract.

Maintenance of data files means keeping data file documentation available for reproduction on request of the COR or other authorized individuals. Certain data files that will be maintained will not be made available to the public because of privacy considerations. In these cases, file maintenance means that the data files will be kept intact, in a form that can be updated or copied.

Data files will not be allowed to expire, to be released, overwritten, or otherwise destroyed.

These files will be maintained with a software system capable of interlinking them for analysis

(or other) purposes while, at the same time, preserving the necessary confidentiality requirements.

1.3 Obtain OMB clearance for data collection

The contractor shall prepare the clearance package for obtaining OMB approval of the study and non-cognitive survey instruments, and any other materials that will be distributed to participants.

Preparing an OMB clearance package entails assembling an extensive amount of information to

(a) justify the data collection as a whole as well as individual background questions; (b) estimate the cost of the data collection to the federal government and the amount of burden (in both time and cost) on respondents; and (c) explain the data‘s uses, efforts taken to minimize the burden on respondents, any provisions for payments or gifts to respondents, steps to assure confidentiality, and the statistical methods that are to be employed in collecting and preparing the data (e.g., dealing with nonresponse). All this information must be written up in a Supporting Statement

(Supporting Statements include justification for the study and description of the data collection, as well as samples of materials used in recruitment). The OMB clearance package for submission must include the Supporting Statement, copies of all materials that will go to respondents (such as recruitment materials, survey questionnaire, letters of consent, etc.), and information on any adaptations of the PIAAC instruments and questionnaires. PIAAC instruments have previously been approved by OMB for the PIAAC 2011 data collection. However, revisions to the background questionnaire will be required for implementation of this study. The contractor shall prepare a draft OMB clearance package for the data supplement study not less than 8 months prior to the start of the data collection.

After COR review and internal NCES review, the contractor shall revise the draft OMB clearance package within 1 week of receipt of comments. NCES will submit the revised OMB clearance packages to ED and OMB for review and approval. Failure to submit the OMB clearance package by the deadline may result in the contractor being unable to collect data in a timely manner. The contractor shall be aware that the time between submission of the

OMB clearance package and receipt of a clearance number from OMB for inclusion on all survey documents takes a minimum of four months, and upwards of six months. The contractor shall take these time constraints into consideration when preparing and submitting the OMB clearance package. The contractor shall work diligently and expeditiously with the COR to seek

OMB clearance prior to contacting households. The contractor may not begin to contact households until OMB has completed its clearance process and approved the data collection.

PIAAC U.S. National Data Collection Supplement PWS - 14

In preparing the OMB clearance package, the contractor shall follow instructions for completing the Information Collection Submission Worksheet and writing the Supporting Statement required for Paperwork Reduction Act submissions (see http://edicsweb.ed.gov/edics_files_web/04031/Att_OMB%20Worksheet%20Part%201.pdf for an example of the worksheet). The contractor shall prepare all written materials needed in the formal clearance and approval process according to OMB requirements as outlined in the

Information Collection Submission Worksheet. The package will include the background questionnaires but not the cognitive (assessment) instruments for the study. This approval process typically involves the following series of steps:

Contractor shall provide a draft package to the COR and revise according to the COR‘s comments.

The COR provides the package to the NCES OMB liaison and chief statistician, and the contractor shall revise package according to comments received.

The NCES OMB liaison provides the package to the ED OMB liaison office and the contractor shall revise the package according to comments received.

ED provides package to OMB; the contractor shall answer OMB questions or revise the package according to comments received.

The contractor staff shall be prepared to join the COR in person or via conference call for a review of the materials with ED and OMB staff. The contractor shall ensure that authorization by

OMB has been obtained before data collection begins. All data collection forms will bear the approval number assigned by OMB.

Following approval, the contractor shall provide NCES/ED with one hard copy and one electronic copy of the clearance package for the study.

1.4 Briefings and descriptive materials

From time to time throughout the course of the project, NCES will provide interested individuals and agencies with information about the nature, progress of, and findings of the study. In consultation with the COR, the contractor shall support these activities by developing up to 5 sets of materials that describe different components of the study. Each set of materials will be no more than 10 typewritten pages of text (double-spaced) with accompanying graphs and tables suitable for display. Requested materials may include overhead slides for making presentations regarding the study. The contractor is welcome to use materials developed for PIAAC 2011

(which NCES will make available) as the basis for requested materials. Within 2 weeks of a request, a set of materials will be submitted to the COR for review and comment. The contractor shall make necessary revisions to the materials based on this review and resubmit the materials within 1 week of receipt of comments.

The contractor shall also be prepared to provide up to 10 presentations, briefings, or demonstrations about the study over the life of the contract. Examples of the kinds of meetings at which the contractor shall be asked to discuss the study are the annual meetings of the American http://edicsweb.ed.gov/edics_files_web/04031/Att_OMB%20Worksheet%20Part%201.pdf

PIAAC U.S. National Data Collection Supplement PWS - 15

Educational Research Association (AERA) and the NCES Summer Data Conference. They might also include briefings with other federal agencies, staff of the U.S. Congress, or the media.

The COR will notify the contractor at least 2 weeks in advance of each briefing or demonstration.

The contractor shall organize up to 3 panel/paper discussions and submit symposium proposals to appropriate conferences and organizations (e.g., AERA annual conference) over the life of the project, as requested by the COR.

1.5 Relationship of the contractor to NCES and other agencies

The contractor shall report all activities to the COR. All data, publications, and products of the study are the sole property of the U.S. government, and shall be provided to it as specified below and upon request. Under no circumstances will the contractor, or any subcontractors, distribute or release any data, publications, or products of the study in any form to any individual or party without prior permission of the COR. This restriction will remain in effect throughout the life of the contract, and until NCES officially releases the data, publications, or products to the public.

The contractor shall work through the COR in all relationships and discussions with any of the international coordinating centers, foreign governments, foreign education professionals, and all other agencies and individuals related to the study.

1.6 Readiness for work under conditions of unusual speed and flexibility

In the event that the study‘s plans and time schedules change rapidly due to various necessities, the contractor shall be prepared to work with changing requirements. Whenever necessary, the contractor shall be prepared to perform activities under conditions of emergency speed.

Emergency speed will be defined according to four priority levels:

Priority 1. within 24 hours of request

Priority 2: within 72 hours of request

Priority 3: within 5 working days of request

Priority 4: within 10 working days of request

In the event that changes to plans or time schedules require the completion of a task or activity under emergency speed, the COR will notify the contractor of the request, after consultation with the Contracting Officer (CO), and identify the timeframe (i.e., priority level) under which the task or activity will be completed by the contractor.

1.7 Materials from PIAAC 2011 contractor

The contractor shall work through the COR to obtain any and all materials from the PIAAC 2011

U.S. data collection contractor necessary to perform the work of this contract. Within 4 weeks of the award of the contract, the contractor shall provide the COR with a list of requested materials.

PIAAC U.S. National Data Collection Supplement PWS - 16

The COR shall determine which of these materials are necessary for the project and provide them to the contractor.

Task 2. Preparations for Assessments (Firm-fixed Price)

2.1 Develop sampling plan and select sample

The contractor shall develop a sampling plan that is based on PIAAC‘s sampling frame and meets NCES standards and established international requirements for sample size, sample design, and selection (see appendix C). The contractor shall develop a sample design that will, when combined with the PIAAC 2011 U.S. sample, ensure a representative sample of the U.S.

adult population ages 16 to 65, specifically including incarcerated adults in this age range. The contractor shall provide the necessary methods for obtaining accurate variance estimates from this combined sample (e.g., weights, including replicate weights).

The sample shall be designed to produce an achieved sample that, in combination with the

PIAAC 2011 sample, has sufficient statistical power to conduct the two topical analysis described in Task 5. As such, the sample shall be designed to provide results in literacy, numeracy, components of literacy, and problem solving in technology rich environments skill measures (average and in each proficiency level) for the following groups:

Unemployed adults ages 16 to 65

Gender (male, female) by

Race/ethnicity (white non-Hispanic, black non-Hispanic, Hispanic, other)

Educational attainment (less than HS, HS or equivalent, some college/certification/AA degree, BA or higher)

Enrollment in formal postsecondary education (yes, no)

Age groups in (approx.) 10-year increments

Young adults ages 16-24 and early career adults ages 25-34

Gender (male, female) by

Race/ethnicity (white non-Hispanic, black non-Hispanic, Hispanic, other)

Enrollment in formal postsecondary education (yes, no)

Employment status (out of the labor force, employed, unemployed)

Educational attainment (less than high school completion, high school completion or equivalent, some college/certification/associate‘s degree, bachelor‘s degree or higher)

Nativity (U.S.-born, foreign born)

Home language (English, Spanish, other)

In addition, NCES is interested in being able to compare adults over the age of 65 with younger cohorts. However, bidders should be aware of NCES interest in obtaining the smallest, most

PIAAC U.S. National Data Collection Supplement PWS - 17 efficient sample possible. Bids will be assessed in part on creativity and efficiency of their proposed sampling approaches. Moreover, NCES does not anticipate obtaining a sample higher than 5,000 cases to combine with the 5,000 cases collected under PIAAC 2011. The exception is that contractors may propose an option for an excess of 5,000 cases (see optional task 10).

Bidders should indicate in their proposals trade-offs associated with including an over-65 cohort within an overall 5,000-case-or-less sample design.

The contractor shall work with NCES to finalize the sample plan for the data supplement study.

The contractor shall detail potential population coverage problems associated with any proposed sample design. In addition, the contractor shall incorporate any recommendations from the

PIAAC 2011 main study‘s technical report or contractor for improving sampling.

The contractor shall submit the sampling plan to the COR 10 months before data collection is scheduled to start. Once the plan is reviewed and approved by the COR, the contractor shall include the approved sampling plan in the data collection OMB package (see task 1.3). OMB approval is critical; under no circumstances will the contractor contact survey participants prior to the approval by OMB.

Once approval has been received from NCES and OMB, the contractor shall implement the sampling plan and select the sample according to the approved plan and study timeline. The contractor shall not deviate from the final, approved sampling plan without approval from NCES and OMB.

Information about the sample plan will be delivered to NCES in standardized forms (i.e., electronic templates) supplied by NCES. Such information will include, for example, population information, sample design and approach, explanations of any oversampling or exclusions, sampling weights, and quality control procedures.

As part of implementing the sampling plan, the contractor may need to create a sampling frame of addresses, collect information about the household composition of a sampled unit, and systematically select eligible respondents within a sampled household. In addition, this task will require the contractor to provide an introductory letter and the study brochure (task 1.1.8) to explain the purpose of the survey to sampled participants as well as provide ―conversion‖ materials to reduce nonresponse, such as endorsements and conversion letters. To obtain survey participation and high response rates, the contractor may be required (with OMB approval) to offer to sampled households monetary and non-monetary incentives, which the contractor will also be responsible for providing. As a point of reference, for PIAAC 2011, participants were paid $50 for their participation (about 120 minutes).

2.2 Adapt and prepare survey instruments and procedures

The design of the PIAAC assessment is complex and takes into account the broad range of abilities among adults, including an interest in focusing on highly skilled adults and those segments of the population demonstrating limited abilities. This is accomplished in PIAAC through the use of different modes of administration (paper-and-pencil and computer-based) and multi-stage testing (with adaptive stages of difficulty).

PIAAC U.S. National Data Collection Supplement PWS - 18

Initial information from a background questionnaire shall be used to direct respondents to either the paper-and-pencil branch of the assessment or the computer-delivered branch. It is expected that the majority of adults in countries like the United States—with relatively larger proportions of higher skilled or computer-familiar adults—will be administered the direct assessment via computer.

The two stages of the adult assessment shall cover literacy, numeracy, and problem solving in a technology rich environment, and components of reading literacy. The entire direct assessment should take about an hour.

Besides the direct assessment in the paper- and computer-based formats, the survey instruments include a background questionnaire (BQ, about 45 minutes long), and the protocols and instructions for selecting and routing cases into the instrument as a whole and through different parts of it, and for ending interviews and closing out cases. These instruments, assessment protocols, and administration instructions have been developed by the OECD and adapted for use in the United States for PIAAC 2011. However, the background questionnaires will have to be further adapted for use with an incarcerated population (which was not included in the 2011

PIAAC design population).

The background questionnaires must be prepared as English-language and Spanish-language versions.

In responses to tasks 2.2.1 and 2.2.2, bidders should indicate any cost or schedule implications of adapting instruments to an over-65 cohort (costs would be captured under optional task 10).

2.2.1 Adapt and prepare survey instruments

Although the internationally-developed instruments, protocols, and instructions were adapted for the United States for PIAAC 2011, some further adaptations will be necessary for the incarcerated population portion of the current study because the PIAAC instruments were not designed for use with an incarcerated population. Thus the contractor shall review and recommend to NCES appropriate ways to further adapt the PIAAC main study‘s background questionnaires, protocols, and instructions for the supplemental study target population, including the U.S. incarcerated population. These recommendations shall take into consideration any recommendations from the PIAAC 2011 main study‘s technical report or contractor for improving the instruments.

The contractor shall submit a memo to the COR with proposed revisions to the background questionnaires, protocols, and instructions designed to ensure their cultural relevance and suitability. If deemed necessary by the COR, the contractor shall seek advice from the expert panels on adaptations.

In addition, the contactor shall adapt the instrument for administration as directed by the COR and NCES. For example, NCES may add additional items to the background questionnaire. The

COR shall work with the contractor to identify and adapt these additional items for inclusion in the PIAAC non-cognitive instruments.

PIAAC U.S. National Data Collection Supplement PWS - 19

Upon approval from the COR, the contractor shall implement necessary revisions to the questionnaires.

The revised questionnaires for the data supplement study will be delivered to the COR for review and comment no later than 4 months before scheduled data collection. The instruments to be used in the PIAAC study must be reviewed at several levels before they can be printed and distributed for use. The contractor shall be aware, and incorporate into its project schedule, the following levels of review (n.b., the levels of review indicated here are subject to change):

1. Contractor reviews and recommends necessary adaptations to questionnaires.

2. NCES reviews Contractor‘s recommendations

3. Contractor revises instruments based on feedback obtained in step 2.

4. COR submits OMB clearance package, prepared by the contractor, along with non-cognitive instruments to OMB for review and comment. (See task 1.3 for details on the extensive task of preparing an OMB clearance package)

5. Contractor revises instruments, based on feedback from step 4.

6. NCES and COR approve and finalize instruments for production.

7. NCES and COR submit final non-cognitive instruments (with OMB approval number, expiration date, and federal paperwork reduction act notice) to OMB for file.

After all reviews have been completed and appropriate revisions made, the contractor shall assemble for administration the assessment and questionnaire instruments according to the

NCES specifications and timeline for the study. Assembly shall include both installing on computers the instruments for computer-based administration as well as printing the instruments for paper-and-pencil administration.

The contractor shall supply sufficient, dedicated, contractor-owned laptops (one per interviewer plus spares in case of breakdowns), computer resources, and computer support (including IT support and a Help Desk) to administer the computer…