vcc_summary.pdf

Remarks for PIRLS 2016 Pre-solicitation Conference Summary This is a summary of the Pre-solicitation Conference held for the solicitation for the PIRLS 2016 U.S. Data Collection contract. Below is a list of the speakers and their remarks, as well as a list of registered vendor participants. Attendees were asked to submit any questions in writing. No questions were received prior to the conference. Three questions were received at the conference:

1. For Task 9, are the only things to be budgeted are the administrative costs of working with the states and getting NCES approval because the state will enter into contractors to pay the contractor? The answer is Yes.

2. What is the schedule for the RFP? The answer is that CAM is still working on the schedule as will get that out as soon as possible. This will be a 2014 award that is scheduled for the middle of October.

3. Has the decision been made to make this a small business set-aside or not? The answer is that there may be possibilities for teaming, but this is not set-aside for small businesses.

Opening remarks: Sylvia Reid, Contracts & Acquisitions Management

Good afternoon and welcome to the Pre-solicitation Conference for the upcoming solicitation: U.S. Participation in the Progress in International Reading Literacy Study 2016.

My name is Sylvia Reid and I am the Contracting Officer for this contract. The purpose of this conference is to provide potential vendors with a description of the work, to seek your input as we finalize the performance work statement, and to give vendors a chance to meet each other and consider possible teaming arrangements.

After the introductions, we will provide a brief description of the work on the project and the primary challenges we see in the work and we expect you to address in your proposals.

Thank you, and now let’s move on to introductions. We’ll start with the Education Department team and then to the vendors.

Then people in the room

[Participants introduced themselves. Please see below a list of registered participants.

International Vendor Communication Conference - 2

Remarks on the U.S. Participation in the Progress in International

Reading Literacy Study: 2016

The goal for the PIRLS 2016 procurement is data collection in the United States for PIRLS 2016 that meets all international and the National Center for Education Statistics (NCES) guidelines and that makes available on a timely basis high-quality internationally comparative assessment data. The data collection will include assessment in reading literacy at the 4th grade and questionnaires administered to students, as well as questionnaires for schools and teachers. The study shall result in a report presenting U.S. data and the international comparisons from the perspective of an American constituency.

PIRLS is the Progress in International Reading Literacy Study. PIRLS is sponsored by the International Association for the Evaluation of Academic Achievement, the IEA, and developed through the input and coordination of participating countries.

PIRLS is an assessment of reading in grade 4. The U.S. sample of schools and students, not including the option, is expected to be on the order of 200 schools and 5,000 students.

We are interested in setting aside work for small businesses; however we have not identified any tasks in the performance work statement as small business set asides. We are very interested in your input on the work that could be done by a small business. Please send you input to our Contract Specialist, Joe Gibbs.

Although I expect that you have read the draft performance work statement, I will read quickly through the list of tasks as listed in the performance work statement:

(note that these tasks apply to a field test and the main study):

Task 1. Contract Administration (Firm Fixed Price), including Project initiation and planning; Contract management; Obtaining OMB clearance for field test and the main study’s data collection; and preparing briefings or presentations on the project:

1.1 Project initiation and planning

1.1.1 Post-award conference

1.1.2 Produce master schedule

1.1.3 Project Management Plan

1.1.4 Staffing Plan

1.1.5 Data Security Plan

International Vendor Communication Conference - 3

1.1.6 Staff Security Requirements

1.1.7 Transmittals of study products and quality of deliverables

1.1.8 Study brochures

1.2 Contract management

1.2.1 Project update meetings with contractor

1.2.2 Other meetings

1.2.3 Monthly study progress and financial reports

1.2.4 Internal Project management portal

1.2.5 Maintenance of data files

1.2.6 Relationship of the contractor to NCES and other agencies

1.2.7 Readiness for work under conditions of unusual speed and flexibility

1.2.8 Quality Control Measures

1.3 Obtain OMB clearance for field test data collection

1.4 Obtain OMB clearance for the main study’s data collection

1.5 Prepare for briefings on project

1.5.1 Prepare project briefing materials

1.5.2 Provide briefings or presentations

Task 2. Preparations for Assessments (Firm Fixed Price), which includes developing a sampling plan, drawing the classroom samples and preparing the test instruments.

2.1 Field test sampling plan and sample

2.2 Main study sampling plan and sample

2.3 Prepare field test instruments

2.4 Prepare main study instruments

The contractor shall develop school and student sampling plans for the field test and main study that meet established international requirements for sample size and design (including use of replacement schools as shown in Appendix C of the PWS). For the field tests, the contractor shall select the field test school sample as well as the student sample following international standards as closely as possible. For the main study, the contractor shall prepare and forward sampling

International Vendor Communication Conference - 4 documentation and data files, by the established international deadline, to the international contractor, who shall select the main study sample. The contractor shall work with NCES and the IEA to finalize the sample plan for the field test and the main assessment. The contractor shall detail potential population coverage problems associated with any proposed sample design.

The contractor shall meet NCES and IEA standards when developing sampling plans. NCES standards are available on line at the NCES website. NCES standards may be revised during the life of the contract. The contractor shall meet NCES standards in effect at the time of data collection and data reporting.

Task 3. Recruiting, schools, teachers, and students (Cost Reimbursable)

This includes adhering to the standards of minimum response rate targets for the IEA and NCES; the submission of a recruitment plan; actual recruitment of schools, teachers and students; and collection of data.

3.1 Recruit Schools, Teachers, and Students for Field Test

3.2 Recruit Schools, Teachers, and Students for Main Assessment

3.3 Collect data for field test

3.4 Collect data for main assessment

Task 4. Data Collection and Delivery (Firm Fixed Price)

This which includes recruiting and training data collectors, collecting the data, producing datasets for the IEA, preparing responses to IEA information requests about the data collection, and preparing the National curriculum questionnaire and producing the U.S. chapter for the PIRLS Encyclopedia.

4.1 Recruit and train data collectors for field test

4.2 Recruit and train data collectors for main assessment

4.3 Produce field test dataset for IEA

4.4 Produce main assessment dataset for IEA

4.5 Prepare responses to IEA information requests about data collection

4.6 Prepare National curriculum questionnaires

4.7 Produce U.S. chapter for the PIRLS Encyclopedia.

Task 5. Scoring and Coding (Firm Fixed Price)

5.1 Score field test assessment and code questionnaire data

International Vendor Communication Conference - 5

5.2 Score main assessment and code questionnaire data (IDIQ, dependent on number of respondents)

For both the field test and main assessment, the contractor shall implement the guidelines (and timeline) established by the IEA/International Study Center for the selection and training of scoring staff and supervisors, and for scoring and coding of assessment materials. Appropriate contractor staff shall attend training by the IEA/International Study Center on scoring of test materials and will comply with any international re-scoring efforts by providing requested test booklets or other materials. The contractor is required not only to score and code data collected in the United States, but also to score and code data from other countries as part of the cross-country scoring reliability exercise established by the IEA/International Study Center. The cross-country scoring exercise is based on a limited number of items from other English-speaking countries. PIRLS open-ended items to be scored include both short-answer and extended-response items.

The contractor shall recruit and hire staff to complete scoring no later than 1 month before scoring begins, for both the field test and main assessment. The contractor shall give special consideration to recruiting staff with prior experience in similar efforts, particularly NAEP reading assessments.

Task 6. Prepare and Disseminate Results (Firm Fixed Price), which includes preparing the national report, the non-response bias analysis reports, survey operations reports , PIRLS school reports, and PIRLS released item sets.

6.1 National report analysis plans

6.2 Produce reports

Non-response bias analysis report Survey operations reports Field test data review memo National reports PIRLS school reports

PIRLS released item set

6.3 Produce main study electronic public-use datafiles and user’s guides

International Vendor Communication Conference - 6

6.4 Disseminate Results (targeted to Congress, the Department of Education, the general public, teachers, principals, policy makers, education researchers, the media, the business community, and/or other potentially interested parties.

Task 8: Comparison of NAEP/PIRLS (Firm Fixed Price)

The contractor shall prepare a report that compares the results of PIRLS 2016 within the context of U.S. national education concerns. The contractor shall plan for, implement, and prepare a working paper that compares the frameworks and content of PIRLS reading with that of NAEP reading.

Task 9: State benchmarking (Firm Fixed Price)

In the event that states indicate an interest in participating in PIRLS as a separate entity from the nation, the government may request that the contractor coordinate the implementation of the study in participating states. The contractor shall work with states and the IEA/BC to draw representative state samples thereby permitting estimates at the state level. The contractor shall not enter into a contract with a state or the IEA/BC for the implementation of PIRLS Benchmarking without the prior consent of the COR. The contractor, however, will be responsible to pay the IEA state participation costs and the BC data processing costs for each participating state. The contractor shall work with the state to implement the study exactly as required under international specifications, and using the exact same instruments and procedures as implemented in the U.S. national sample. States are allowed to augment the background questionnaires with additional questions of interest; states may not remove or reword items in the background questionnaires unless it is a matter of state law.

Optional tasks

Task 7 (Optional): Utilize expert help (Firm Fixed Price)

The purpose of using expert panels will be to give advice to the contractor, to ensure that national and professional educators‘ views are represented, and to provide NCES with a communication channel to various organizations involved reading education in the United States. Sample tasks in which the expert panels may be involved include but are not limited to:

International Vendor Communication Conference - 7

- reviewing item performance from the field test questionnaires and cognitive instruments and making recommendations about possible revisions;

- reviewing draft questionnaires and cognitive instruments for the main study and providing feedback on the need for U.S. adaptations;

- reviewing and commenting on analysis plans for presenting results from the study;

- assisting with dissemination plans and activities; and

- comparing PIRLS frameworks and items with that of other national assessments (e.g., NAEP).

Task 10 (Optional): Parent Questionnaire – Learning to Read Survey (Cost Reimbursable)

Internationally, IEA studies have found a strong positive relationship between students’ reading achievement at the fourth grade and home experiences that foster literacy learning. This survey is about students’ early learning experiences.

It assesses what parents (or primary caregivers) and students do together and what they think about different things related to their student‘s school. This information that is collected will be useful in helping understand how young children learn to read and for helping to improve the teaching and learning of reading for all children. Like the School and Teacher questionnaires, this instrument can be administered electronically or by paper and pencil. It should be administered to all students who have received parental permission to participate in PIRLS.

If this option is exercised, the contractor shall follow the same procedures as specified in sections 3.1 and 3.2, respectively for the field test and main study.

The primary challenges we see for PIRLS 2016 are:

1. Securing school response rates over 85 percent;

2. Managing a successful data collection in the 2-moth time frame that is in the spring when U.S. schools are in the middle of test preparation or state testing; producing a national report and getting through all of the layers of NCES and IES review in less than 6 months.

In addition, if the optional task of administering the Parent Questionnaire – Learning to Read Survey is exercised, the primary challenge is determining the

International Vendor Communication Conference - 8 methodology for administration. The method used internationally administers the questionnaire after the student assessment, with students taking the survey home to their parents and returning the survey to the school.

Closing remarks: Sylvia Reid

Ms. Reid thanked Ms. Helen Chang from the Small Business Office of the Department for her comments and stated that this will not be a small business set-aside. However, she noted that there may be opportunities for teaming. Ms.

Chang thanked the vendors and reminded them to send any questions by email input to the Contract Specialist, Joseph Gibbs.

International Vendor Communication Conference - 9

PRE-SOLICITATION CONFERENCE SIGN-IN SHEET

PIRLS National Data Collections June 6, National Center for Education Statistics 8th Floor Conference Room #3

Initials Name Company Phone E-Mail

Cammise McInnis Cammisells owner@cammisells.com

David Kastberg Westat 301-294-

Kastbed1@westat.com

Greg Binzer Westat 301-294-2036 Binzerg1@westat.com

Jason Hill

RTI

202-974-7872 jahill@rti.org

Debbie Herget RTI 919-485-7793 hergert@rti.org

Patricia Green RTI 312-456-5260 PGREEN@RTI.ORG

John Tanner

General Dynamics Information Services

703-292-3251

John.Tanner@vangent.

com

Kim Barnette Delta Decisions of DC 202-723-1919

xt. 1 kbarnette@deltadecisi onsdc.com

Shree Taylor Delta Decisions of DC 202-723-1919 X3 staylor@deltadecisionsdc.c om

Betsy Thorn Insight Policy Research 703-504-9488 bthorn@insightpolicyresear ch.cm

Stephane Baldi Insight Policy Research

Kristen Neishi NORC 301-634-9358 Neishi-kristen@norc.org mailto:Binzerg1@westat.com mailto:John.Tanner@vangent mailto:kbarnette@deltadecisionsdc.com mailto:kbarnette@deltadecisionsdc.com mailto:Neishi-kristen@norc.org