VCC Summary.pdf

PIAAC, PISA, TIMSS Pre-solicitation Conference Summary

This is a summary of the Pre-solicitation Conference held for three upcoming Department of Education solicitations for the PIAAC-Integrated Adult Assessment U.S. National Data Collection Supplement, the PISA 2015 U.S. Data Collection, and the TIMSS 2015 U.S. Data Collection contract. Below is a summary of remarks, as well as a list of companies that were present.

Attendees were asked to submit any questions in writing. No questions were received prior to the conference.

Remarks on the PIAAC Integrated Adult Assessment Supplement: Eugene

Owen, National Center for Education Statistics (NCES)

The goal for the Integrated Adult Assessment U.S. National Data Collection Supplement is to supplement the PIAAC U.S. national data collection undertaken in 2011 with (1) an additional sample of U.S. adults to produce, when combined with the original 2011 U.S. PIAAC sample, a representative U.S. sample capable of supporting analyses of key U.S. subgroups outlined in this performance work statement; (2) a bridging study that establishes a statistical linkage between PIAAC and the 2003 National Assessment of Adult Literacy (NAAL); and (3) administration of an assessment of additional adult literacy domains not included in PIAAC 2011.

PIAAC is the Program for the International Assessment of Adult Competencies, an international study sponsored by the Organization for Economic Cooperation and Development, the OECD, and developed through the input and coordination of the countries participating in the study.

PIAAC collected data over the fall of 2011and the spring of 2012. This original sample includes about 5,000 adults. The additional sample included in the work of the upcoming contract is, as I said, to make a large enough sample to support analyses of key U.S. subgroups. We expect this additional sample to be about 5,000 more adults.

The other two parts of the performance work statement call for the successful bidder to develop the design for a bridging study that establishes a statistical linkage between PIAAC and the 2003 National Assessment of Adult Literacy (NAAL) and to develop the design for an assessment of an additional domain, functional writing. Carrying out these additional data collections will take place as optional tasks in the contract.

Although I expect that you have read the draft performance work statement, I will read quickly through the list of tasks:

Task 1. Contract Administration (firm fixed price)

Task 2. Preparations for Assessments (firm fixed price)

Task 3. Data Collection and Delivery (Cost-reimbursable)

Task 4. Score assessment and code questionnaire data (Cost-reimbursable)

Summary of Pre-solicitation Conference for PIAAC, PISA, TIMSS - 2

Task 5. Prepare and Disseminate Results (firm fixed price)

Task 6. Develop Plan for Bridging Study (firm fixed price)

Task 7. Develop Plan for a Functional Writing/PIAAC Literacy (firm fixed price)

The following are optional tasks:

Task 8. (optional) Bridging Study

Task 9. (optional) Additional domain

Task 10. (optional) Additional sample

The primary challenges we see for the project are

(1) the need to ensure that the internationally established procedures for the data collection of the additional U.S. sample are followed faithfully,

(2) that the characteristics of the two assessments are considered when developing the bridging study, and

(3) that the field test results for the functional writing domain are used as the basis for the full scale assessment. Additional challenges for all three parts of the procurement include designing both the samples and the assessments.

Remarks on PISA 2015: Dana Kelly, NCES

The goal for the PISA 2015 procurement is data collection in the United States for PISA 2015 that meets all international and NCES guidelines and that makes available on a timely basis high-quality internationally comparative assessment data. The data collection will include computer-based assessments in mathematics, science, and reading literacy and questionnaires administered to students and schools. The study shall result in a report presenting U.S. data and the international comparisons from the perspective of an American constituency. This contract will also include a validity study to examine the relationship between performance on PISA and educational and workforce outcomes by recruiting students that participate in PISA 2012 and administering a questionnaire in 2015, when those students are 18 years old.

PISA is the Program for International Student Assessment, and it also is sponsored by the OECD and developed through the input and coordination of participating countries. PISA is an assessment of 15-year-old students in science, reading, and mathematics literacy, as well as, in 2015, of collaborative problem solving. The U.S. sample of schools and students in 2015 is expected to be on the order of 150 or 175 schools and 5,000 or 6,000 students.

In an effort to encourage more work for small businesses, we expect to let two contracts for PISA 2015 – one under full and open competition and the other as a small business set aside.

The winners of the two contracts will need to work as a team and so we expect teams of contractors to bid for these two contracts. We will make these arrangements more clear in the

Summary of Pre-solicitation Conference for PIAAC, PISA, TIMSS - 3 final performance work statement and other RFP documents, but, of course, you are welcome to submit questions and input about this in the mean time. For today’s purposes, we wanted to let you know what work we plan to set aside for a small business and give you a chance to meet each other and hear about each other’s capabilities.

I will read quickly through the list of tasks in the performance work statement:

Task 1: Contract Administration (Firm-fixed Price), including obtaining OMB clearance for field test and the main study’s data collection

Task 2: Study Design and Instruments (Firm-fixed Price), which includes sampling plans and preparing the instruments

Task 3: Data Collection (Cost-reimbursable), which includes recruiting schools, training data collectors, collecting data, scoring, and data file preparation

Task 4. Prepare and Disseminate Results (Firm-fixed Price) , which includes preparing the natioanl report as well as school-level reports.

Task 5: Use Expert Help (Cost-reimbursable)

Task 6: PISA Validation Study (Cost-reimbursable), which involves following up with students who participated in PISA 2012 and supplied contact information to administer an on-line questionnaire on their education and employment experiences at age 18. This task is a small business set-aside.

The following are tasks that may or may not be exercised by the government:

Task 7 (Optional): Manage Subnational Participation in PISA (Cost-reimbursable)

Task 8 (Optional): Conference for PISA 2015 Schools (Cost-reimbursable)

Task 9 (Optional): Additional Subject-Matter Assessments (Cost Reimbursable), which for 2015 could include financial literacy and/or environmental literacy.

Task 10 (Optional): Disseminate Results (Firm-fixed Price)

Task 11 (Optional): Item development (Firm-fixed Price)

The primary challenges we see for the project are ensuring that the U.S. data collection follows all internationally prescribed procedures, meets all internationally prescribed standards, and meets all international deadlines, and being able to adapt to changing circumstances, including evolving procedures and deadlines during the course of the study—this is always a challenge for the international studies, but we expect this to be even greater in PISA 2015 because of the shift to computer-based assessment and because the international contracting arrangements are new, with new players and new roles;

Summary of Pre-solicitation Conference for PIAAC, PISA, TIMSS - 4 specific to PISA 2015, a challenge is that PISA is transitioning from a primarily paper-based assessment to a fully computer-based assessment;

achieving the required response rates for schools and students which historically has been difficult for the United States to do; and if states opt to participate in PISA 2015 and optional task 7 is exercised, coordinating the implementation of PISA at the national and state levels will be a challenge.

Remarks for TIMSS 2015: Stephen Provasnik, NCES

The goal for the TIMSS 2015 procurement is data collection in the United States for TIMSS 2015 that meets all international and the National Center for Education Statistics (NCES) guidelines and that makes available on a timely basis high-quality internationally comparative assessment data. The data collection will include assessments in mathematics and science at the 4th and 8th grades (and possibly 12th grade) and questionnaires administered to students, as well as questionnaires for schools and teachers. The study shall result in a report presenting U.S. data and the international comparisons from the perspective of an American constituency.

TIMSS is the Trends in International Mathematics and Science Study. TIMSS is sponsored by the International Association for the Evaluation of Academic Achievement, the IEA, and developed through the input and coordination of participating countries. TIMSS is an assessment of mathematics and science in grades 4 and 8. In 2015, there is an option to assess students in grade 12 or the first year of college, as well. This is included as an option in the performance work statement. The U.S. sample of schools and students, not including the option, is expected to be on the order of 800 schools and 20,000 students.

We are interested in setting aside work for small businesses; however we have not identified any tasks in the performance work statement as small business set asides. We are very interested in your input on the work that could be done by a small business. Please send you input to our Contract Specialist, Brenda Jefferson.

Here, then, are the tasks as listed in the performance work statement (note that these tasks apply to a field test and the main study):

Task 1. Contract Administration (Firm Fixed Price), including obtaining OMB clearance for field test and the main study’s data collection.

Task 2. Preparations for Assessments (Firm Fixed Price), which includes developing a sampling plan, drawing the classroom samples, preparing the test instruments, and cognitive lab work.

Task 3. Recruiting, schools, teachers, and students (Cost Reimbursable)

Task 4. Data Collection and Delivery (Firm Fixed Price), which includes recruiting and training data collectors, collecting the data, producing datasets for the IEA, preparing responses to IEA information requests about the data collection, and preparing the National curriculum questionnaires and producing the U.S. chapter for the TIMSS Encyclopedia.

Summary of Pre-solicitation Conference for PIAAC, PISA, TIMSS - 5

Task 5. Scoring and Coding (Cost Reimbursable)

Task 6. Prepare and Disseminate Results (Firm Fixed Price), which includes preparing the national report, the non-response bias analysis reports, survey operations reports, TIMSS school reports, and TIMSS released item sets.

Optional tasks

Task 7 (Optional): Utilize expert help (Cost Reimbursable)

Task 8 (Optional): Comparison of NAEP/TIMSS (Firm Fixed Price)

Task 9 (Optional): State benchmarking (Cost Reimbursable)

Task 10 (Optional): TIMSS-NAEP mathematics linking study (Cost Reimbursable)

Task 11 (Optional) TIMSS Advanced assessment (Cost reimbursable)

Task 12 (Optional): Produce main study electronic public-use datafiles and user’s guides (Firm Fixed Price)

Task 13 (Optional): Disseminate Results (Firm Fixed Price)

The primary challenges we see for TIMSS 2015 are securing response rates over 85 percent;

managing a successful data collection in 2-month time frame that is in the spring when U.S. schools are in the middle of test prep or state testing;

producing a national report and getting through all the layers of NCES and IES review in less than 6 months.

In addition, if the optional tasks are exercised, two significant challenges that may present themselves are

Coordinating the TIMSS-NAEP linking study with the NAEP division of NCES and the NAEP state coordinators; and

Creating a sampling frame, drawing a sample, and conducting a successful data collection for TIMSS Advanced among 1st year college students.

Summary of Pre-solicitation Conference for PIAAC, PISA, TIMSS - 6

Pre-solicitation Conference PIAAC, PISA, TIMSS National Data Collections June 18, 2012

National Center for Education Statistics 8th Floor Conference Room #1

The following companies were represented at the meeting:

Company

Matrix Knowledge Group International

Ipsos Public Affairs

Juarez & Associates, Inc.

Mathematica Policy Research

Data Recognition Corporation

Washington Software, Inc.

NORC

NuCore Vision

Questions and Answers:

Two questions were received at the conference:

1. Who are the incumbent contractors for these solicitations?

Westat is the prime contractor for the PIAAC 2011 data collection contract, the PISA 2012 data collection contract, and the TIMSS 2011 data collection contract.

2. Will you make a list of attendees at the bidders conference available?

A list of companies who attended the meeting is provided above.