PWS.IEA_International_ED_Studies_V4_Draft_(07-01-16).pdf

Attachment A - Draft Performance Work Statement

Development and Implementation of the International Association for the

Evaluation of Educational Achievement (IEA) International Education Studies

Contents

I. INTRODUCTION

II. BACKGROUND

III. LEGISLATIVE AUTHORITY

IV. SCOPE OF WORK

Objectives

V. REQUIREMENT

Task 1: Overall International Project Coordination and Contract Management

Task 2: Development and Selection of Items for Study Instruments (Fixed Price)

Task 3: Create International Versions of Study Instruments (Fixed Price)

Task 3A: Create International Field Trial Versions of Study Instruments

Task 3B: Create International Main Study Versions of Study Instruments

Task 4: Adaptation, Translation and Translation Verification (Fixed Price)

Task 5: Review of Sampling Plans and Achieved Sample (Fixed Price)

Task 6: Scoring Training (Fixed Price)

Task 7: Identify and Train International Quality Control Monitors (Fixed Price)

Task 8: Oversee Full-Scale International Data Collection (Fixed Price)

Task 9: Database Creation, Data Editing/Cleaning, and Data Quality Control (Fixed

Price)

Task 10: Data Weighting, Scaling, and Estimation (Fixed Price)

Task 11: Data Analysis and Reporting (Fixed Price)

Task 12: Update and Maintain Public Web Site; Establish and Maintain Secure Web Site

(Fixed Price)

Task 13: Annual IEA Membership Meeting Activities (Fixed Price)

OPTIONAL TASKS

Optional Task 14: International Large-Scale Assessments and Surveys Database

Training (Fixed Price)

Optional Task 15: Upgrades and Extensions of Electronic Assessment Platform (Fixed

Price)

Optional Task 16: State, Territory or District Participation in IEA International Education

Studies (Fixed Price)

Optional Task 17: Updates and Maintenance of ILSA Gateway (Fixed Price)

SCHEDULE OF DELIVERABLES

Task 1: Overall International Project Coordination and Contract Management

Task 2: Development and Selection of Items for Study Instruments

Task 3: Create International Versions of Study Instruments

Task 4: Adaptation, Translation and Translation Verification

Task 5: Review of Sampling Plans and Achieved Sample

Task 6: Scoring Training

Task 7: Identify and Train International Quality Control Monitors

Task 8: Oversee Full-Scale International Data Collection

Task 9: Database Creation, Data Editing/Cleaning, and Data Quality Control

Task 10: Data Weighting, Scaling, and Estimation

Task 11: Data Analysis and Reporting

Task 12: Update and Maintain Public Web Site; Establish and Maintain Secure Web Site

Task 13: Annual IEA Membership Meeting Activities

OPTIONAL TASKS

Task 14: International Large-Scale Assessments and Surveys Database Training

(Optional)

Task 15: Upgrades and Extensions of Electronic Assessment Platform (Optional)

Task 16: State, Territory or District Participation in IEA International Education Studies

(Optional)

Task 17: Updates and Maintenance of ILSA Gateway (Optional)

APPENDIX A - COMPLIANCE WITH ED INFORMATION TECHNOLOGY (IT)

SECURITY POLICY

Draft Performance Work Statement

Development and Implementation of the International Association for the

Evaluation of Educational Achievement (IEA) International Education Studies

I. INTRODUCTION

For nearly three decades, the United States (U.S.) Department of Education has been working with a number of organizations to improve the quality and comparability of education data in order to report on U.S. education in comparison with education in other countries. One of the major organizations with which the United States works with is the

International Association for the Evaluation of Educational Achievement (IEA), based in

Amsterdam, the Netherlands (http://www.iea.nl). The IEA has a long and successful history of mounting major large-scale international education assessments and surveys, most recently the Progress in International Reading Literacy Study (PIRLS) 2016 and the

Trends in International Mathematics and Science Study (TIMSS) 2015. The IEA has established a regular cycle of assessments and surveys to provide policymakers, educators and the general public with information on the state of students’ knowledge in core academic subjects as well as emerging areas of policy interest, such as computer and information systems literacy and civic engagement and education. The IEA is now on schedule to implement the next round of data collection for several of its studies, including the International Computer and Information Literacy Study (ICILS) in 2018, TIMSS in 2019, PIRLS in 2021, and the International Civic and Civics Education Study

(ICCS) (to be determined (TBD)). ICILS is a computer-based international assessment of eighth-grade students’ computer and information literacy (CIL) skills. TIMSS focuses on fourth- and eighth-grade students’ mathematics and science knowledge and skills. PIRLS provides an opportunity for participating countries to gather information on fourth-grade students’ reading literacy skills. Both TIMSS and PIRLS are in the midst of a transition to a computer-based format. ICCS examines students' knowledge and understanding of concepts and issues related to civics and citizenship, as well as their beliefs, attitudes, and behaviors. The IEA also occasionally conducts international education studies in other areas, such as early childhood education and teacher preparation at the tertiary level. All

IEA studies allow participating countries to compare system level outcomes (for example, student achievement scores) to those in other education systems around the world.

NCES intends to contract with the IEA to develop and implement the next round of the

IEA’s international education studies. Through this contract, NCES will work with the

IEA, globe-spanning subject matter, measurement, and sampling experts, as well as partner countries to ensure the collection of high-quality data that can inform relevant

U.S. education policy questions. Each of the IEA’s international education studies identified above provides unique information that cannot be duplicated through existing surveys, particularly due to their international context.

http://www.iea.nl/

II. BACKGROUND

The National Center for Education Statistics (NCES) in the Institute of Education

Sciences of the U.S. Department of Education is the primary federal entity for collecting and analyzing data related to education. NCES’s longstanding mission is to “collect, and analyze and disseminate statistics and other information related to education in the United

States and in other nations.”

To fulfill its mission, NCES requires the development and international management of the next round of IEA international education studies, namely TIMSS, PIRLS, ICILS, and ICCS, to continue uninterrupted. The IEA maintains sole copyright of TIMSS, PIRLS, ICILS, and ICCS, including but not limited to the assessment items, survey questionnaires, and computer-based delivery platforms. The IEA is thus the sole source for continuing these important cross-national education studies in which the United States has participated since the early 1990s.

III. LEGISLATIVE AUTHORITY

The National Center for Education Statistics (NCES) in the Institute of Education

Sciences of the U.S. Department of Education conducts this study in compliance with the mandate stated in Section 406 of the General Education Provisions Act, as amended (20

U.S.C. 1221e-1) and in the Education Sciences Reform Act of 2002 (Public Law 107-279, Title I, Part C, Sec. 153)).

Further, NCES assures participating individuals and institutions that any data collected conforms to the NCES standards for protecting the privacy of individuals as required by

Section 183 of the Education Sciences Reform Act of 2002 (Public Law 107-279):

All collection, maintenance, use, and wide dissemination of data by the institute, including each office, board, committee, and Center of the

Institute, shall conform with the requirements of section 552A of title 5, United States Code [which protects the confidentiality rights of individual respondents with regard to the data collected, reported, and published under this title. [Section 183]

Under the law, Education Sciences Reform Act of 2002 (Public Law 107-279), all individually identifiable data about students, their families, and their schools shall remain confidential. To this end, the law requires that no person may:

Use any individually identifiable information furnished under the provisions of this section for any purpose other than statistical purposes for which it is supplied, except in the case of terrorism (see discussion of the Patriot Act);

Make any publication whereby the data furnished by any particular person under this section can be identified; or

Permit anyone other than the individuals authorized by the Commissioner to examine the individual reports.

Further, individually identifiable data is immune from legal process, and shall not, without the consent of the individual concerned, be admitted as evidence or used for any purpose in any action, suit, or other judicial or administrative proceeding, except in the case of terrorism. Employees, including temporary employees, or other persons who have sworn to observe the limitations imposed by this law, who knowingly publish or communicate any individually identifiable information will be subject to fines of up to

$250,000, or up to 5 years in prison, or both (Class E felony).

On October 24, 1994, Public Law 103-382 was passed by Congress which was the legislation reauthorizing the data collection authority of NCES. Section 405, Performance of Duties, indicates the following:

(a) Grants, Contracts, and Cooperative Agreements

(1) In General, in carrying out the Commissioner’s duties until this title, the

Commissioner may grant awards, and enter into contracts and cooperative agreements.

(2) Duration – Notwithstanding any other provision of the law, the grants, contracts, and cooperative agreements under this section may be awarded, on a competitive basis, for a period of not more than five years, and may be renewed at the discretion of the Commissioner for an additional period of not more than five years.

IV. SCOPE OF WORK

Objectives

The overall objective of this requirement is to produce data similar to those collected in prior implementations of PIRLS (2001, 2006, 2011 and 2016), TIMSS (1995, 1999, 2003, 2007, 2011, and 2015), ICILS (2013), and ICCS (2009 and 2016), and other occasional international education studies conducted by the IEA, such as early childhood education and teacher education/preparation. To do this, many of the materials, procedures and, now, survey delivery platforms used in these studies need to be the same—i.e., psychometrically equivalent—to the materials and techniques used in prior data collection rounds. This is especially important as these studies transition from paper-and-pencil only formats to computer-based delivery systems. In addition, the implementation of these international education studies need to be carefully and systematically monitored and documented to ensure that the participating countries carry out the assessments as required and that clear and consistent documentation is collected to assist in the evaluation of study results and in secondary data analyses.

To achieve these objectives, NCES requires the following tasks be conducted for PIRLS, TIMSS, and ICILS, among other IEA international education studies. It should be noted that this procurement does not include U.S. national data collection activities for any of these studies, as these are conducted through separate actions.

As each of the IEA’s international education studies is on a different timeline, the tasks described below are written to cover each study, with tasks applicable depending on the development stage of each study. ICILS will conduct a field trial in 2017 and a main study in 2018; TIMSS will conduct a pilot study in 2017, a field trial in 2018, and a main study in 2019; PIRLS will conduct a field trial in 2020 and a main study in 2021. While one study may be preparing for a field trial, another study may be in the earlier development stages (pre-field trial), and, as such, the tasks and deliverables will vary for each study over the course of this contract.

Independently, and not as an agent of the United States Government, the contractor shall supply all personnel, materials, services, and facilities necessary for the project and perform the tasks as described below. The tasks below are applicable to any of the IEA’s international education studies, whether the currently scheduled ones (TIMSS, PIRLS, and ICILS) or those still under discussion but that could become relevant during the course of this contract (e.g., ICCS). The contractor shall complete each of the tasks described below for TIMSS, PIRLS and ICILS, with due dates for deliverables based on the timelines developed for each study.

V. REQUIREMENT

The contract requires 13 tasks:

1. Overall International Project Coordination and Contract Management

2. Development and Selection of Items for Study Instruments

3. Create International Versions of Study Instruments

4. Adaptation, Translation and Translation Verification

5. Review of Sampling Plans and Achieved Sample

6. Scoring Training

7. Identify and Train International Quality Control Monitors

8. Oversee Full-Scale International Data Collection

9. Database Creation, Data Editing/Cleaning, and Data Quality Control

10. Data Weighting, Scaling, and Estimation

11. Data Analysis and Reporting

12. Update and Maintain Public Web Site; Establish and Maintain Secure Web Site

13. Annual IEA Membership Meeting Activities

Each time an IEA International Education Study is initiated in the contract base period or through the exercise of one or more optional tasks, the contractor shall repeat the tasks listed. In fulfilling these requirements, the contractor shall also refer to the additional requirements included in Appendix A.

Task 1: Overall International Project Coordination and Contract Management

(Fixed Price)

The contractor shall coordinate and monitor the quality of the activities of participating countries. This requires that the contractor monitor the progress of each participating education system in the preparation for, implementation of, and compilation of data for the PIRLS, TIMSS, and ICILS projects. Specifically, the contractor shall coordinate and monitor the following activities and processes that occur over the course of each study:

framework review and revision;

cognitive item development and field testing;

background questionnaire review, survey item development, and field testing;

instrument translation and quality control;

(where applicable) computer-based delivery system launch and updates;

sampling plans;

in-field quality control monitoring;

scoring and coding training;

data entry training;

data quality control measures and monitoring;

data base construction, editing, and documentation;

data weighting, scaling, and estimation;

data analysis; and data reporting and dissemination.

To accomplish this task, the contractor shall convene meetings of the country project directors, or National Research Coordinators (NRCs), approximately twice a year over the course of the next rounds of the PIRLS, TIMSS and ICILS projects. The contractor shall produce all materials necessary to conduct successful meetings with the NRCs, including but not limited to:

meeting agenda;

meeting arrangements (hotel/conference center) and information, including arrangements for remote participation in meetings (if available); and meeting materials necessary to carry out discussions of the agenda topics.

The contractor shall ask participating countries to volunteer to host meetings, and work with the host country to make hotel, meeting space, and remote technology arrangements

(if feasible).

To deal with ever-increasing costs related to holding international meetings, the contractor shall actively seek methods to reduce travel for NRCs and countries’ data collection contractors. The contractor’s proposal is expected to include alternatives to in-person attendance of meetings, such as webcasts or webinars of meetings. Other possibilities include a reduction in the number of in-person meetings, to be replaced possibly by more frequent remote-attendance meetings. Reduction of travel should not come at the cost of the quality of meetings however.

So that participating countries can plan appropriately, the contractor shall develop a detailed timeline for each study. The timeline shall include dates for all key milestones in the study, and shall take into careful consideration the effort required by NRCs to complete their required tasks, including reviews of cognitive items and publications. The contractor shall update each study timeline throughout the life of the contract if events require changes in key dates and deliverables.

Countries are responsible for the costs associated with their own participation (e.g., national data collection costs; quality control monitoring; participation fees).

To aid in the monitoring of the success of the TIMSS and PIRLS studies, the contractor shall submit to the Contract Officer’s Representative (COR) a progress report with each invoice, covering the same period as the invoice. A single progress report shall be submitted that includes information on the PIRLS, TIMSS, and ICILS projects, though detailed separately under each main heading. In the progress reports, the contractor shall succinctly and clearly report on the accomplishments achieved during the time period associated with each submitted invoice. The monthly report shall include a brief description of the following information for each study:

a summary of accomplishments by the contractor and its subcontractors;

descriptions of any problems encountered;

descriptions of solutions considered and implemented;

brief alert of issues on the horizon that could be problematic; and progress on each of the deliverables required in the subsequent 6 months covered under this contract.

Deliverables required under Task 1:

Deliverable 1.1: NRC meeting agenda (as scheduled for each study)

Deliverable 1.2: NRC meeting arrangements and informational letters (as scheduled for each study)

Deliverable 1.3: NRC meeting materials (as scheduled for each study)

Deliverable 1.4: TIMSS international timeline

Deliverable 1.5: PIRLS international timeline

Deliverable 1.6: ICILS international timeline

Deliverable 1.7: Monthly study progress report

Task 2: Development and Selection of Items for Study Instruments (Fixed Price)

The contractor shall identify leading subject matter and psychometric experts and then work with the identified experts to develop and make selections of replacement cognitive items as well as to refine the background questionnaires for each study. To accomplish this, in addition to working with the NRCs from participating countries, the contractor shall convene specialist subject matter committees to provide expert advice on the development and selection of items for inclusion in each study. In the recent past, these groups have included a Questionnaire Item Review Committee (QIRC); a Subject Matter

Item Review Committee (SMIRC); among others. As with the NRC meetings, the contractor is strongly encouraged to develop remote technology options to reduce the amount and cost of travel required of participants, when and where feasible.

It is expected that approximately one-half of the cognitive items will need replacement in each study that includes an assessment and that all background questionnaire items will require review and possible revision or replacement. Replacement cognitive items selected shall be similar in design and purpose to those used in the previous iterations of each study, appropriate to the ages/grades being assessed and shall conform to the parameters set forth in each respective study’s published Framework document. The contractor shall work with the experts in each committee to ensure that the replacement cognitive items will permit comparisons of achievement scores in the current round of studies with previous rounds and that the accompanying scoring rubrics are well devised and comprehensive. The same shall apply to the background questionnaire items (where feasible and advisable). The final selection of items shall be based, in large part, on results of the field testing of items and the expert opinion of the various expert committees as well as the NRCs.

The contractor shall produce a report on the selection of items following pilot or field testing, and shall document the procedures used to determine whether items, both cognitive and non-cognitive, functioned as intended and met the technical criteria established for each study. The contractor shall include in its review of the piloted or field tested items differential item functioning (DIF) analyses to determine whether there is evidence of bias at the country level and that the items functioned as expected. This report shall be made available to all participating education systems in each study, at their request, in either paper or electronic form.

As ICILS is already well underway, only a field test item functioning report is required under this task.

Deliverables required under Task 2:

Del 2.1: Establish the TIMSS expert committee(s)

Del 2.2: Establish the PIRLS expert committee(s)

Del 2.3: Field test item functioning report for TIMSS

Del 2.4: Field test item functioning report for PIRLS

Del 2.5: Field test item functioning report for ICILS

Task 3: Create International Versions of Study Instruments (Fixed Price)

This task is divided into two parts (3A and 3B) to cover the field trials and main studies.

Task 3A: Create International Field Trial Versions of Study Instruments

Once the items for the study instruments have been selected (under Task 2), the contractor shall produce standardized international sets of the instruments (cognitive and non-cognitive) for each study from which education systems will translate and make adaptations, as necessary, for the field trial. The contractor shall create forms of the survey instruments that will allow countries to easily produce their own versions of the instruments, in the language(s) of instruction, for administration in their own countries.

The contractor shall produce the field trial instruments for each study that allows countries to replace the source language text with the appropriate translated text while maintaining the preferred international layout of the instruments. In the past, this has been accomplished through the use of InDesign software. The contractor shall ensure that the digitally-based platform on which the instruments are to be administered (where applicable to a study) is amenable to allowing translations and adaptations across languages and cultures.

As ICILS is already well underway, The creation of field trial versions of the international study instruments is only required for PIRLS and TIMSS.

Deliverables required under Task 3A:

Del 3A.1: Create field trial version of PIRLS instruments

Del 3A.2: Create field trial version of TIMSS instruments

Task 3B: Create International Main Study Versions of Study Instruments

Once the items, instruments, and operations for the studies have been field trialed, the contractor shall produce international sets of the main study instruments (cognitive and non-cognitive) for each study, from which education systems will translate and make adaptations, as necessary. The contractor shall create forms of the main study survey instruments that will allow countries to easily produce their own versions of the instruments, in the language(s) of instruction, for reproduction in their own countries.

The contractor shall produce the main study instruments for each study in an electronic format that allows countries to replace the source language text with the appropriate translated text while maintaining the preferred international layout of the instruments. In the past, this has been accomplished through the use of InDesign software. The contractor shall ensure that the digitally-based platform on which the instruments are to be administered (where applicable to a study) is amenable to allowing translations and adaptations across languages and cultures.

Deliverables required under Task 3B:

Del 3B.1: Create main study version of PIRLS instruments

Del 3B.2: Create main study version of TIMSS instruments

Del 3B.3: Create main study version of ICILS instruments

Task 4: Adaptation, Translation and Translation Verification (Fixed Price)

The contractor shall work with participating countries to ensure the timely and accurate adaptation and translation of each study’s instruments into the language(s) of instruction of each country. The contractor shall provide participating countries with electronic files of the PIRLS and TIMSS 2011 instruments in a format that allows for easier translation into various languages, and which will result in uniformity across all countries’ instruments.

The contractor shall implement a standardized adaptation and translation verification system that ensures, to the greatest degree possible, the equivalency of items across country-specific study instruments. The contractor shall require countries to document each and every adaptation and translation so that this information can be easily reviewed and verified (such as through the National Adaptation Form, or NAF, used in previous administrations). Where warranted, the contractor shall utilize the services of a professional translation agency or a similar entity to assist in this effort. The contractor shall document the procedures put into place to obtain the highest possible cross-country instrument equivalency, and shall document any deviations from this plan and any cultural adaptations made by countries. This documentation shall be summarized in a chapter in the international technical report for each study.

Deliverables required under Task 4:

Del 4.1: Track adaptations/translations of PIRLS instruments through a standardized form

Del 4.2: Track adaptations/translations of TIMSS instruments through a standardized form

Del 4.3: Track adaptations/translations of ICILS instruments through a standardized form

Del 4.4: Produce chapter on PIRLS adaptation/translation verification process for international technical report

Del 4.5: Produce chapter on TIMSS adaptation/translation verification process for international technical report

Del 4.6: Produce chapter on ICILS adaptation/translation verification process for international technical report

Task 5: Review of Sampling Plans and Achieved Sample (Fixed Price)

The contractor shall implement a process for the review and approval of each country’s sampling plan, for the pilot, field trial and main study of each study (as applicable, depending on study timelines). The contractor shall create a system for documenting sampling deviations from the agreed upon international standards for each study, either in their initial planning or implementation in a country. In addition, the contractor shall clearly articulate the IEA Technical Standards for each study to participating countries, especially the minimum acceptable standards for participation rates at each unit of interest (e.g., school, teacher, and student). The contractor shall emphasize to participating countries the importance of meeting participation rates at all levels of sampling—including the possibility of deletion from final study reports--and to sample schools and students that are representative of the entire education system, rather than particular student subpopulations. Discussion of technical standards for each study shall be held at the first meeting of NRCs, or as soon thereafter as possible.

The contractor shall include the information collected in the review of main study sampling plans and the final, achieved samples of each country as a chapter in the international technical reports produced for each study. The chapters shall include the final unweighted and weighted participation rates for each participating country at the school, class, and student level for each study (or, as appropriate to each study).

Countries that do not meet the established minimum international standards for each study shall be clearly indicated in the international reports, including the source of the deviation from approved plans.

Deliverables required under Task 5:

Task 5.1: Produce chapter on sampling (i.e., characteristics of national samples) and final

(weighted and unweighted) participation rates for PIRLS

Task 5.2: Produce chapter on sampling (i.e., characteristics of national samples) and final

(weighted and unweighted) participation rates for TIMSS

Task 5.3: Produce chapter on sampling (i.e., characteristics of national samples) and final

(weighted and unweighted) participation rates for ICILS

Task 6: Scoring Training (Fixed Price)

The contractor shall provide training to those individuals in each participating country who will be responsible for overseeing or conducting scoring training on open-ended and short-response items. The scoring training shall be conducted at least twice to accommodate the dual data collection schedules created due to differences in how

Northern and Southern hemisphere countries organize their school systems.

The contractor shall develop a set of training materials that include actual sample student responses for all items that will need to be scored or coded. Included in these training materials shall be fully-developed scoring rubrics for all open-ended and short-response items.

The contractor shall work with participating countries to ensure that high quality scoring training and detailed documentation of scoring reliabilities is implemented in each participating country. Deviations from the approved scoring training, or levels of scoring reliability that do not meet IEA technical standards, shall be reported in the international technical reports for each study.

Deliverables required under Task 6:

Del 6.1: Scoring training materials for constructed-response PIRLS items

Del 6.2: Scoring training materials for constructed-response TIMSS items

Del 6.3: Scoring training materials for constructed-response ICILS items

Del 6.4: Publication of final main study scoring reliabilities, by participating education system, in the PIRLS international technical report

Del 6.5: Publication of final main study scoring reliabilities, by participating education system, in the TIMSS international technical report

Del 6.6: Publication of final main study scoring reliabilities, by participating education system, in the ICILS international technical report

Task 7: Identify and Train International Quality Control Monitors (Fixed Price)

The contractor shall identify and train a cadre of international quality control monitors

(IQCM) whose job is to conduct a quality assurance program to document the quality of the data collection for each study in each participating country. The contractor shall ensure, to the extent possible, that the quality control monitors are independent of the researchers, project directors, their families and friends in each participating country so as to avoid the appearance of bias.

The contractor shall conduct separate trainings of IQCMs to accommodate the data collection schedules (Northern and Southern Hemispheres) of countries in each study.

The contractor shall develop appropriate materials for the training sessions, including the development and implementation of a standardized protocol for the collection and reporting of the observations of the IQCMs. Information documented through the protocols shall be included in the international technical reports.

Deliverables required under Task 7:

Del 7.1: Quality control monitor (IQCM) training materials for each study

Del 7.2: Agenda for quality control monitor training meetings for each study

Del 7.3: Produce chapter on international quality control monitoring process and outcomes in PIRLS international technical report

Del 7.4: Produce chapter on international quality control monitoring process and outcomes in TIMSS international technical report

Del 7.5: Produce chapter on international quality control monitoring process and outcomes in ICILS international technical report

Task 8: Oversee Full-Scale International Data Collection (Fixed Price)

Because of differences in school year timing around the world, international data collection for each study will take place at different times of the year (as detailed in each study timeline; see Task 1). The contractor shall oversee and monitor the data collection process of participating countries at the international level. This means that the contractor shall ask for and receive periodic updates on the implementation of data collection procedures at the country level, and shall assist countries that may be experiencing technical or practical difficulties in implementing the data collection as required. The contractor shall document any difficulties or irregularities observed or reported, and include this information in the international technical reports for each study.

Deliverable required under Task 8:

Del 8.1: Annual summary update on progress of each study

Task 9: Database Creation, Data Editing/Cleaning, and Data Quality Control (Fixed

Price)

The contractor shall develop a set of materials, and a training program, to ensure that countries create country-level datasets for each study that conform to a standardized data structure, clearly document national adaptations and variations to the survey instruments, and ensure that all variables used for international comparisons are comparable and consistent across countries. The contractor shall apply a standardized set of editing and cleaning rules to each country-level dataset to ensure the accuracy and consistency of data, and shall document any deviations from the international database file structures for each study. The contractor shall work with countries to resolve any data inconsistencies discovered during the data cleaning/editing and verification process until all data are consistent and comparable between countries. Documentation of database construction and data quality control procedures shall be made available in the international technical reports for each study.

Deliverables required under Task 9:

Del 9.1: Produce chapter on database creation and data quality control in PIRLS international technical report

Del 9.2: Produce chapter on database creation and data quality control in TIMSS international technical report

Del 9.3: Produce chapter on database creation and data quality control in ICILS

Task 10: Data Weighting, Scaling, and Estimation (Fixed Price)

The contractor shall use weighting and scaling procedures comparable to, if not exactly the same as, those utilized in prior iterations of each study and as detailed in the technical standards for each study. The contractor shall develop sampling weights to ensure the proper representation of each unit of analysis (e.g., students, schools, and teachers) in each country’s respective dataset. The contractor shall use Item Response Theory (IRT) to estimate the measurement characteristics of each assessment question for each study

(i.e., scale the results). The contractor shall use the results of the scaling to produce estimates of student achievement (where applicable), as well as for content and cognitive domains subscales (where applicable and feasible). The contractor shall produce clean and error-free weighted data including scaled student scores for each study on schedule.

The contractor shall create a preliminary draft, a penultimate draft, and a final version of the international database for each study for participating countries to produce their own national-level reports, as scheduled.

The preliminary draft database should be made available to participating countries as early as possible so that preliminary analyses can be conducted in anticipation of national data release plans. The contractor is strongly encouraged to consider adding more time between the initial compilation of the international database and the release of the international reports to allow for countries to conduct analyses of national interest and to carefully review data.

Based on feedback from the countries, the contractor shall produce a penultimate draft database for each study (i.e., the final database with sufficient documentation but not yet finalized), and shall make them available to countries at least two months prior to the scheduled release of the international reports. Upon distribution of the penultimate draft of the databases, the contractor shall include a memo that specifies what changes, if any, have been made to the databases and their potential implications for any analyses. The availability of the databases prior to the release of the international reports is needed to provide countries with sufficient time to prepare and review their own national report materials.

Once the international reports have been finalized and released, the contractor shall produce final databases for each study, including all documentation necessary to successfully use the databases for secondary analyses at a later date. Database documentation shall be similar to documentation prepared for prior versions of the study databases. The contractor shall make the final international databases available for downloading on the international web sites, including all necessary database documentation and macros. The international databases shall be finalized and made available on the web within five months after the release of the international data reports from each study, respectively.

The contractor shall document all procedures used to produce weights and scaled estimates in the international technical reports for each study.

Deliverables required under Task 10:

Del 10.1: Produce chapter on the creation of sampling weights and the methodologies used to scale data in the PIRLS international technical report

Del 10.2: Produce chapter on the creation of sampling weights and the methodologies used to scale data in the TIMSS international technical report

Del 10.3: Produce chapter on the creation of sampling weights and the methodologies used to scale data in the ICILS international technical report

Del 10.4: Produce final international PIRLS database and documentation

Del 10.5: Produce final international TIMSS database and documentation

Del 10.6: Produce final international ICILS database and documentation

Task 11: Data Analysis and Reporting (Fixed Price)

The contractor shall analyze each study’s international database to produce graphs, figures, and tables similar in character and quantity to those produced in preparation of previously released international study reports. The contractor shall conduct thorough analyses of the cognitive and background data, and shall implement a procedure for the review of analyses to prevent errors in the international database and the publishing of erroneous results. The contractor shall refrain from making claims that cannot be supported by study data and the statistical comparisons made in the reports, including suggestions or hypotheses about the performance of individual countries or student subpopulations.

The contractor shall produce international reports on the findings from each study. The reports shall include estimates of achievement (where applicable) based on the newly collected data, as well as comparisons of achievement over time (where applicable). The contractor shall also produce and publish achievement subscales based on content areas and the cognitive domains (as defined in the Framework of each study; where applicable). Moreover, the international reports shall include estimates based on the non-cognitive items (i.e., background questionnaires) and, where feasible, comparisons to prior estimates. The contractor shall describe similarities and differences between education systems based on the outcomes of acceptable and sound statistical measures.

The contractor shall be responsible for composing the text to describe the data in the tables, graphs, and figures.

The contractor shall also produce technical reports that describe the processes and procedures that were put into place for each study. The international versions of the technical reports will include information on topics such as quality control, sampling, weighting and scaling, and the like. The technical reports shall be at least as thorough as previously produced technical reports for each study. The contractor shall make the technical report for each study available to the public, preferably online in Portable

Document Formant (PDF) or other appropriate formats.

To ensure the highest quality in publications, and to make vetting of publications more transparent, the contractor shall propose and implement a review process for the international reports included in this task that involves NRCs and independent expert reviewers. The contractor shall convene meetings with the NRCs to review the progress of data analysis, and to review anticipated publications. The contractor shall compile the comments of the NRCs, how the comments were addressed in the reports, and make them available for review by all NRCs.

Deliverables required under Task 11:

Del 11.1: Final, international report on PIRLS findings

Del 11.2: Final, international report on TIMSS findings

Del 11.3: Final, international report on ICILS findings

Del 11.4: PIRLS international technical report

Del 11.5: TIMSS international technical report

Del 11.6: ICILS international technical report

Task 12: Update and Maintain Public Web Site; Establish and Maintain Secure

Web Site (Fixed Price)

The contractor shall make updates to the PIRLS, TIMSS, and ICILS international websites to reflect activities and products related to the next data collection rounds of each study. The contractor shall also establish and maintain a secure website (or portal) for each study that will allow for the transfer of study materials and other documentation to countries in a secure and encrypted environment. The contractor shall provide login

IDs and passwords for the secure websites to the representative(s) from each participating country.

The creation of a secure website or portal is only required for PIRLS and TIMSS.

Deliverables required under Task 12:

Del 12.1: Update and maintain public PIRLS website

Del 12.2: Update and maintain public TIMSS website

Del 12.3: Update and maintain public ICILS website

Del 12.4: Establish and maintain PIRLS secure website or portal

Del 12.5: Establish and maintain TIMSS secure website or portal

Task 13: Annual IEA Membership Meeting Activities (Fixed Price)

The contractor shall conduct meetings of the following IEA committees and groups that are not study-specific to maintain the quality and continuity in the conduct of its studies.

The contractor shall hold annual meetings of the following:

Technical Group

Publications Group

Management Group

Standing Committee

General Assembly

The contractor shall prepare all documents necessary for the successful meeting of these groups. These documents shall include, but are not limited to, meeting agendas, briefing materials, and meeting registration materials. Meeting registration materials shall be made available at least 8 weeks in advance of each meeting. Meeting agendas shall be made available at least10 weeks in advance of each meeting. Briefing materials shall be made available at least 2 weeks in advance of each meeting. The contractor shall make all meeting arrangements, including selection of a locale and venue, blocking of hotel rooms, and a meeting space. The contractor shall prepare a summary of the discussion and outcomes of each meeting, to be made available within 1 month after the conclusion of a meeting. Meeting summaries shall be distributed to meeting participants and other members to whom such materials are normally distributed.

Deliverables required under Task 13:

Del 13.1: Meeting agenda for each IEA committee/group

Del 13.2: Meeting registration materials for each IEA committee/group

Del 13.3: Briefing materials for each IEA committee/group

OPTIONAL TASKS

The tasks outlined below (Optional Tasks 14-17) are optional tasks that the Government may exercise during the Base period (years 1 – 5).

Optional Task 14: International Large-Scale Assessments and Surveys Database

Training (Fixed Price)

The contractor shall, at the option of the Government, conduct database training seminar(s) for each study in the United States (locale to be determined) or other locations if deemed reasonable and feasible. The purpose of the database training seminars is to familiarize new and more experienced researchers and other interested parties in using the international education study databases. The contractor shall provide attendees of the seminar, at minimum, information on the following topics: overview of the studies and databases; sampling and weighting; plausible values; overview of item response theory;

overview of survey instruments and variables; appropriate statistical techniques to be used; limitations and other cautions when using the international databases; and other selected topics relevant to the successful analysis of each study database. Moreover, the primary focus of each seminar, beyond familiarizing attendees with the database, is to permit attendees to conduct a preliminary analysis of the international study databases during the seminar period.

The contractor shall be responsible for all aspects of the database training, including:

make venue arrangements, including arrangement of housing and seminar meeting space;

advertise training seminars;

oversee registration process; create and distribute training manual/materials;

arrange for presentations by experts knowledgeable in any of the seminar topics;

and provide access to the relevant IEA international education databases.

The contractor may also be asked to arrange for laptop/computer workstations for each registered attendee and appropriate statistical software on each laptop/computer. As a general guide, the contractor shall conduct database training for up to 20 training registrants. The contractor shall establish and clearly convey to applicants the criteria for the selection of seminar attendees, including familiarity with relevant statistical packages, experience in large-scale database analysis, and other relevant experience/expertise that may be relevant to the studies to be covered. The contractor shall recruit researchers at any stage of their career, from full professors to graduate students wishing to use the datasets for master’s or doctoral theses. The contractor shall also recruit individuals from multiple geographic areas and types of institutions. If necessary, a limited number of applicants from the same institution shall be admitted to the seminar by placing a lower priority on multiple applicants from the same institution. For graduate student/researcher in training applicants, the contractor shall request that they seek a letter of support from an institutional faculty/staff member who is familiar with the applicant, can attest to the intellectual and statistical abilities of the applicant, and expresses willingness to work with the applicant post-seminar training. Priority shall be given to those graduate students/researchers in training who express interest in using any of the IEA’s international education databases as the basis of a master’s or doctoral thesis. The contractor is strongly encouraged to schedule the training session to take advantage of the gathering of individuals for other education-related meetings and conferences (e.g., American Education Research Association (AERA); National Council on Measurement in Education (NCME)) to take advantage of planned travel by training registrants. U.S.

government employees may participate in the seminar but the contractor shall not reimburse U.S. government employees for any costs incurred related to their attendance.

Deliverables required under Optional Task 14

Del 14.1: Database training registration materials, including the application form

Del 14.2: Memo detailing the proposed selection process, including criteria for the selection of training applicants, reviewer names and timelines

Del 14.3: Memo detailing the results of the selection process, total number of applicants by sex, institutional affiliation, geographic location and the list of successful applicants

Del 14.4: Database training materials (i.e., User’s Guide materials)

Del 14.5: Memo summarizing the database training, issues that arose, issues to consider for any next training session, and any lessons learned

Optional Task 15: Upgrades and Extensions of Electronic Assessment Platform

(Fixed Price)

The contractor shall, at the option of the government, upgrade and extend the electronic assessment platform developed to deliver the IEA’s international education studies, such as TIMSS, PIRLS, ICILS, and ICCS. Possible areas for upgrade or expansion are:

A system enhance that delivers study instruments in paper format generated from the eAssessment system, i.e., a module than can generate PDF versions of the study instruments;

A research project to define which process information (‘click’ metadata) should be collected and how it can be used to validate or enhance current measures;

A system enhancement to collect process information identified a priori, i.e.

metadata. Metadata may enhance current measures or give a hint on respondent strategies used to solve instrument items, particularly cognitive items.

An enhanced item development interface for NRCs and other key staff that allow storage of different versions of items, review procedures, etc. (similar to the translation module). This module should also allow the creation of country-specific reports that are useful in the carrying out of studies (e.g., items, translations, national adaptations, data upload, and scoring, raw data).

The contractor shall identify reasonable and financially feasible options under this task, of which the Government may choose to implement one or more than one idea.