PWS - Task Order 09.pdf

Version Date: March 2, 2022

Task Order Title: International Activities Support Task

Task Order Type: Firm Fixed Price and Cost-Reimbursable

International Activities Support Task

Performance Work Statement

I. INTRODUCTION

The National Center for Education Statistics (NCES) within the Institute of Education Sciences

(IES) of the U.S. Department of Education (ED) has the following mission:

1. to collect and analyze education information and statistics in a manner that meets the highest methodological standards;

2. to report education information and statistics in a timely manner; and

3. to collect, analyze, and report education information and statistics in a manner that:

a. is objective, secular, neutral, and non-ideological and is free of partisan political influence and racial, cultural, gender, or regional bias; and

b. is relevant and useful to practitioners, researchers, policymakers, and the public.

P.L. 107-279, Part C, Sec. 151(b); H.R. 3801, Part C, Sec. 151(b), 2002.

Within NCES, the International Activities Branch or IAB (also known as the International

Activities Program or IAP) supports a variety of international activities to collect and report data on education in the United States and other countries. It is the NCES’ goal to provide valuable comparative information about the U.S. population’s educational performance, competencies, and skills in an international context as well as about the relationships of such with the nation’s economic and workforce development.

NCES participates in international student assessments in order to:

1. provide comparable indicators on student performance and schooling practices across countries in order to benchmark U.S. student performance; and

2. suggest hypotheses about the relationship between student performance and factors that may influence performance and about areas where students have particular strengths or weaknesses.

NCES participates in international assessments of adult skills and competencies in order to:

1. provide comparable indicators on adult basic skills and broad range of competencies, and education levels across countries in order to benchmark the U.S. working age adults’ literacy, numeracy, other competencies, and education levels; and within the United

States, document relationships among working age adults’ cognitive results, educational enrollment and attainment, and occupational outcomes, and support correlational analyses of the factors that may influence these relationships.

2. suggest hypotheses about the relationships between adult skills, learning and experience at work and outside of work, and economic and non-economic outcomes.

The work done through this procurement will be part of the larger international program that

NCES has participated in through collaboration with a variety of international organizations and partners, such as but not limited to the Organization for Economic Cooperation and Development

(OECD), the International Association for the Evaluation of Educational Achievement (IEA), ministries of education and other government agencies, and non-governmental organizations, as well as statistical and research institutes throughout the world.

Through its participation in these activities, NCES seeks to strengthen the quality, consistency and timeliness of international data. To continue this effort, the United States must follow through with well-organized and well-executed data gathering activities within our national boundaries. These efforts will allow NCES to provide policy-relevant data for comparisons of the U.S. education system with education systems in other countries not only related to characteristics of the education system, but also as the outcome of the education process related to participation in further education and training, readiness for the work force, and participation in society more generally. High quality data of this kind will help to inform decision making on the part of national, state, and local policy makers.

A. Background

The International Activities Branch within the Assessment Division of the National Center for

Education Statistics (NCES) has the responsibility to fulfill the Center’s mandate to “collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including acquiring and disseminating data on educational activities and student achievement (such as the Third International Math Study) in the United States compared with foreign nations.” To this end, IAB conducts various international projects, including international surveys and international large scale assessments (ILSAs). These have included, but are not limited to school-based studies, adult literacy assessments, and developmental projects.

International school-based studies

Trends in International Mathematics and Science Study (TIMSS)

Program for International Student Assessment (PISA)

Progress in International Reading Literacy Study (PIRLS)

Teaching and Learning International Survey (TALIS)

International Computer and Information Literacy Study (ICILS)

International Early Learning Study (IELS)

International adult literacy assessment and survey

Program for the International Assessment of Adult Competencies

(PIAAC)

Developmental projects through various international agencies

Asia-Pacific Economic Cooperation (APEC)

International Association for the Evaluation of Educational

Achievement (IEA)

Organization for Economic Cooperation and Development (OECD)

United Nations Education and Scientific and Cultural Organization

(UNESCO)

All of the projects listed contribute to the NCES goals of collecting and reporting data on education in the United States and other nations, and providing valuable comparative information about U.S. educational performance in an international context.

B. Purpose

The task supports the activities of the International Activities Branch by providing professional and technical services to carry out the work as outlined in this Performance Work Statement

(PWS). Support for international assessments and surveys entails a wide array of activities. It can include, but is not limited to, supporting the work of the Branch chief and NCES study leaders for the international studies; providing management support of international activities, as well as of NCES communication and policy initiatives with international agencies; providing expert and technical support for assessment development, analysis, and reporting; preparing analysis plans and conducting analyses; drafting or reviewing reports or parts of reports, including graphics;

revising manuscripts in response to editorial and technical comments; preparing documents for release in a variety of electronic formats; preparing documents in response to information requests, including from the IES Director, the Secretary, or Congressional staff; preparing conference papers and presentations; preparing responses to general public information requests;

coordinating dissemination activities; dataset review and preparation; website development; and communicating with other contractors and international organizations involved in the studies.

C. Legislative Authority

The authority for these studies can be found in NCES’ longstanding mission to “collect, and analyze and disseminate statistics and other information related to education in the United States and in other nations” (stated in section 406 of the General Education Provisions Act, as amended

(20 U.S.C. 1221e-1) and in the Education Sciences Reform Act of 2002 (P.L. 279-107, Part C, Sec. 153):

(a) GENERAL DUTIES.-The Statistics Center shall collect, report, analyze, and disseminate statistical data related to education in the United States and in other nations, including…(6) acquiring and disseminating data on educational activities and student achievement (such as the Third International Math Study) in the United States compared with foreign nations.

(See http://www.ed.gov/policy/rschstat/leg/PL107-279.pdf for the full Education

Sciences Reform Act.)

D. Quality Control

NCES is firmly committed to determining how well each project meets 1) the project goals, 2) the quality of data collected, and 3) implications for future projects. Project quality control should be integrated into every phase of the project. For instance, potential problems should be identified early so that possible solutions are tested as part of the field test. Item reliability and validity tests for questionnaires and student cognitive tests must be built into the data collection efforts, and results from these tests shall inform data collection and processing. Nonresponse analyses shall be conducted as part of the final weighting process, and survey methodology and sample design effects http://www.ed.gov/policy/rschstat/leg/PL107-279.pdf shall be documented for use by all data users. Thus, as in other NCES projects, statistical quality control shall be incorporated into each stage of this project as appropriate.

Quality control also includes (1) managing all staff to ensure the high quality of their performance; (2) instituting procedures relevant to confidentiality and protection of data

(including the secure transmittal of confidential data); and (3) ensuring deliverables are submitted on time and are of sufficient quality (in terms of accuracy, coherence, clarity, and comprehensiveness) to meet NCES statistical standards, OECD/PISA standards, professional standards, and NCES and IES style guidelines.

E. NCES Standards

NCES standards set guidelines to ensure the quality of NCES surveys, data, data products, and statistics. All work conducted under this contract must, at minimum, meet the standards and guidelines set forth in the most currently available statistical standards and guidelines for the

National Center for Education Statistics (NCES) and as amended by standards implemented since that time. The NCES Standards can be accessed at https://nces.ed.gov/statprog/2012/ /.

Publication style guidelines for IES products shall also be followed. The guidelines can be accessed at https://nces.ed.gov/statprog/style.asp.

All final deliverables that are to be assigned a NCES publication number or that will be made public through the NCES website as a product will be delivered in PDF format (or another appropriate format approved by NCES) in accordance with the NCES web publishing requirements (available at https://nces.ed.gov/pubs2003/secureweb/ch_2.asp), and must adhere to Section 508 requirements for accessibility. All final deliverables that are to be made public through the NCES website as webpages or in a web-based reporting format must also adhere to to Section 508 requirements for accessibility. More information on accessibility is available through the Federal IT Accessibility Initiative at: http://www.section508.gov.

F. Confidentiality

Section IV provides a guide to issues of confidentiality.

G. Contract Type

The contract type is a hybrid with both Cost Reimbursable and Firm Fixed Price tasks.

H. Period of Performance

The period of performance for this contract is a 12-month base period from May 1, 2022 through

April 30, 2023.

http://www.section508.gov/

II. OVERVIEW OF TASK SCOPE

This purpose of this task order is to support the activities of the International Activities Branch

(IAB). This section provides a description of the contract requirements (tasks) and a listing of deliverables associated with each task. The Performance Work Statement (PWS) will include tasks that fall under four general categories:

1. General management: communication and coordination with NCES IAB staff and planning and monitoring work, and contract transition activities.

2. Program study development and support: planning, monitoring, and technical support for the preparation and development of IAB surveys and assessments prior to data collection.

3. Analysis and reporting: development, quality review, and dissemination of analyses and reports using data from IAB surveys and assessments following data collection.

4. Other support: cognitive laboratory studies, website design, and support for international education organization surveys.

The contractor will support NCES in reviewing various plans, schedules, reports, data, and items, expert feedback and recommendations, and support to NCES in tracking schedules or project implementation. The contractor will not be conducting the same activities as the data collection contractors who conduct major instrument development, data collection, and, typically, the initial release of report findings.

Independently and not as an agent of the United States Government, the contractor shall provide all personnel, materials, services, and facilities necessary for the various studies and perform the tasks as described below.

The contractor shall be clear that NCES is the sole authority to manage these studies in the

United States and to negotiate with the international consortium on any issues related to the study. The contractor shall not use the items for any other purposes without NCES’ written consent. Under no circumstances shall the contractor conduct any activities related the school-based studies, adult literacy assessments, and developmental projects, without permission from

NCES.

The contractor shall report all activities for this contract to (a) the Contracting Officer’s

Representative (COR), and (b) the Branch Manager.

The contractor shall work through the appropriate NCES representatives in all relationships and discussions with the IEA, OECD, international contractors, foreign governments, foreign education professionals, and all other agencies and individuals related to the study. The contractor shall not convey to any of these entities information that may be construed as NCES, IES, or Department of Education policy, except where related to technical aspects of the study or where given explicit permission by the COR.

All data, publications, and products of the study are the sole property of the U.S. government and shall be provided to it as specified below and upon request. Under no circumstances will the contractor, or any subcontractors, distribute or release any data, publications, or products of the study in any form to any individual or party without prior permission of the COR. Any contractor or subcontractor conference presentations or papers, based on this study, shall be reviewed and cleared by NCES prior to any such presentation. This restriction will remain in effect throughout the life of the contract, and until NCES officially releases the data, publications, or products to the public.

Task 1. General Management (Firm Fixed Price)

The contractor’s Deputy Directors will meet with the NCES Branch Chief and COR within 5 business days of award for an initial kick-off meeting where the work activities will be reviewed, and the schedule of deliverables refined. The contractor will submit a summary of this initial meeting, including any revisions to the PWS, to the NCES Branch Chief and Task Leader within one week of the meeting. Historically, there has been a “lead” or Deputy Director for the school-based international study work and a “lead” or Deputy Director for the adult literacy assessments

(PIAAC); NCES is open to changing this type of organization structure and will discuss it at or prior to the kick-off meeting.

Subtask 1.1. Post-Award Project Meeting (Firmed-fixed price)

Within the first week after contract award, the contractor’s project director, study leads, and other key project staff as identified in the proposal shall hold a meeting with the NCES branch chief, COR or Task leader, and NCES study leaders (sometimes titled National Research

Coordinators (NRCs), National Study Leaders (NSLs), or National Program Managers (NPMs)), and other appropriate NCES staff to review the overall contract tasks, to identify potential problems and possible solutions, and to discuss areas of concern related to the proposed project staffing plan and other management requirements. The primary purpose of this meeting is to refine the proposed plans for project management, staffing and succession, data security, invoicing, and the master project schedule. These refinements are not to alter the specifications of the contract, but to provide management information for use by both the contractor and the government in monitoring the work to be performed. This meeting will also provide the opportunity for ED staff to share project experience and materials with the contractor.

The contractor will submit a detailed account of the initial meeting discussion to the NCES

Branch Chief and Task Leader within one week of the meeting.

Deliverable:

• Minutes and attendee list from post-award meeting

Subtask 1.2. Staff Security Requirements (Firmed-fixed price)

All contractor staff to work on the project for 30 days or more must complete a series of security forms and undergo a federal background check per OMB requirements, with higher levels of security clearance required for those who visit schools or who may come in contact with confidential data. The contractor shall ensure that all staff who may come in contact with confidential data shall

(a) be trained in the proper handling of confidential data, and

(b) first sign and then notarize an affidavit of nondisclosure before having access to any data with confidential information.

The contractor shall execute these affidavits of nondisclosure and retain them in a secure location. The contractor shall provide a list to the COR and Task Leader of all the individuals who have signed affidavits, the position in the organization of the person, and the person’s relationship to the study, within four (4) weeks of the award of the contract.

As new persons are assigned to the project, the security forms shall be filled out and an affidavit shall be executed for them on the first working day of assignment to the study and an updated list provided to the COR and Task Leader, no less than on a monthly basis. The contractor shall notify the COR and Task Leader if any contractor staff leave the project. The contractor shall be aware of and comply with all contractor personnel security regulations and all Department IT security policy requirements, as described in section V.

Deliverables:

• List of all personnel with signed Affidavits of Nondisclosure

Subtask 1.3. Monthly study invoices and progress and financial reports (Firmed-fixed price)

The contractor shall gather and incorporate comments from NCES on its proposed invoicing system to make the monitoring and approving of billing smoother and simpler for this contract by having a format that is dynamic rather than static and that eliminates redundancy and cross-referencing across pages. The format must clearly allow this contract’s CO, COR, and Task

Leader to identify the costs in whole and in parts, by study, task, and subtask and distinguish between IT and non-IT costs (should there be any ED-defined IT costs) as well as costs for which a fee or profit is assessed and those cost for which there is none. It must allow for easy checking of receipts and subcontractor invoices as well as the labor hours for corporate staff, field staff, subcontractors, and experts.

The contractor shall prepare a revised invoicing plan to respond to any concerns raised by the

Task Leader, COR or CO about the invoicing system proposed in the bid no later than two (2) weeks after the start of the contract. The Task Leader or COR will provide comments on the plan within two (2) weeks of receipt. The contractor shall incorporate any changes necessary to respond to the concerns raised by the Task Leader, COR, or CO and submit a revised plan within

1 week of receipt of comments. Once the project management plan is approved by the Task

Leader, COR and CO, the contractor shall invoice the project in accordance with the approved invoicing plan and, if a change to the plan is needed, shall notify the Task Leader, COR, and CO of such change(s) before implementing such.

The contractor shall submit to the Task Leader, COR, and CS a regular monthly activity progress and financial data report, by the 15th day of the month following the month the report covers.

This report shall be delivered before or concurrent with the contractor’s monthly billed invoice so that it is available to check and approve each month’s invoice.

The monthly report shall serve as a justification for all work billed for the month. Therefore, the contractor shall take care to ensure not only that all work billed for the month is described in sufficient detail in the monthly report to support an audit, but that all detailed information (on completed work) contained in the monthly report is transparently and easily associated with its respective cost or billing.

At a minimum, the contractor shall include, in each monthly report, information on the following aspects of the project and its operation, broken out by study:

1. The past month’s work, including a description of continuing work, new work, and completed work, with key dates on which tasks or activities were initiated or completed during the month;

2. The upcoming month’s work, including a description of the work and key dates by which time tasks or activities will be initiated or completed during the month;

3. Major accomplishments for the month;

4. List of products and milestones submitted or achieved during month, with a short description of each; and

5. Problems/Issues and how each has been or will be addressed by the contractor.

6. Monthly financial data

For the monthly financial data section, the following information shall be presented in summary both for the project as a whole and broken out for each study:

• Past month’s cost of work completed, by task and subtask (including a break-down by IT and non-IT costs where applicable);

• Percent of contracted amount spent to date, by task;

• Estimated cost to complete, total estimated cost, and total contracted cost;

• Difference between the total estimated cost and the contracted cost;

• Staff hours spent on each cost-reimbursement task during the prior month;

• Cost of work conducted by subcontractors (including a break-down by IT and non-IT costs if applicable); and

• A “manpower” report prepared and signed by the project director that summarizes actual personnel assignments for the month just completed, showing for each named individual the hours charged, by task. For individuals not named in the contract, the manpower report will show hours spent by position (e.g., data collectors). All hours billed to the contract will be summarized in the report.

The contractor shall include detailed information on any anticipated or actual cost overruns or cost savings, whether incurred within a single study, single task, or cumulatively. From this information, the contractor shall ensure that the Task Leader, COR, and appropriate study leader is clear about the progress being made on tasks, the work that remains to be accomplished, the financial resources used and available, and any problems and proposed solutions. At the request of the Task Leader, COR, or CO, if additional issues need to be addressed in either monthly report, the contractor shall submit an addendum within two (2) days of the request.

The format for the monthly progress and financial data report will be consistent from month to month.

The contractor shall prepare a draft model or mockup report for the Task Leader, COR, and CO to review no later than two (2) weeks after the start of the contract. The Task Leader, COR, or

CO will provide any comments on the plan within two (2) weeks of receipt. The contractor shall incorporate any changes necessary to respond to the concerns raised by the Task Leader, COR, or CO, and submit a revised version within one (1) week of receipt of comments. Once the format for the monthly progress and financial data report is approved by the Task Leader, COR, and CO, the contractor shall prepare such reports in accordance with the approved format and, if a change to the plan is needed, shall notify the Task Leader, COR, and CO of such change(s) before implementing such.

The monthly progress and financial data reports shall be submitted together each month and the contractor shall respond to questions regarding either document within one (1) day.

Electronic copies of all deliverables for the month to be submitted to the Task Leader, COR, and

CO. The mode of delivery will be negotiated with NCES and ensures that NCES controls access to deliverables.

• Draft invoicing plan

• Final invoicing plan

• Draft monthly study progress and financial report format

• Final monthly study progress and financial report format

• Regular monthly study progress and financial reports

Subtask 1.4. Project update meetings with contractor (Firmed-fixed price)

For the life of the contract, the contractor will hold regular meetings with all IAB staff and branch chief every other month to discuss various study and project progress. The branch chief may propose to hold these meetings via telephone conference calls, over the Internet, or face-to-face if the meeting is critical. The contractor will propose an agenda in advance of the meeting, revise with feedback from the branch chief, task leader, and distribute to all invitees in advance of the meeting.

Deliverables:

• Meeting Agenda

• Meeting Notes that summarize work activities, problems, decisions, and next steps

Optional Subtask 1.5: Contract transition activities (Firmed-fixed price)

The transition period would allow the contractor to attend information sessions to support a smooth transfer and/or handoff of contract work that must continue without interruption once a new task order begins. The contractor is expected to familiarize their staff on task protocols and current project schedules. The transition meetings are likely to take place beginning one month prior to the end of the task order. The transition period, if implemented, would not affect the contract periodicity described in this task order.

• Memo describing all deliverables

• Memo describing processes involved in carrying out each task

• Final deliverables

Task 2. Program Study Development and Support (Cost-reimbursable)

The contractor will provide research and technical support for U.S. participation in international surveys and assessments, including PIAAC, PIRLS, PISA, TALIS (including TALIS video and related studies), TIMSS, ICILS, IELS, and the cross-cutting activities related to the Condition of

Education, Digest of Education, and NAEP, as well as other international data.

These activities will include

1) support NCES national study leaders in management of the international studies;

2) reviews of materials developed to plan and undertake the surveys and assessments (study design documents, instruments/adaptations, analysis plans, and results/analyses); and

3) providing technical advice to NCES.

Subtask 2.1: Management and technical support of International Study/Research Leaders

The contractor will provide support for the organization and management of meetings for study teams and identified experts; and securing expertise on specific topics for limited periods of time to help NCES address immediate needs. The contractor will attend regular contractor meetings to facilitate the communication of work to be assignments, changes to schedules, issues, and decisions. The meeting periodicity will be determined by the NCES study leaders.

Support also includes participation in international meetings – both ad hoc and regularly scheduled meetings (e.g., NRC and NPM meetings) – and under normal circumstances the regularly scheduled international meetings require the contractor to travel internationally to attend in person. These international meetings are run by international education organizations, such as IEA and OECD, and typically convened outside the U.S., hosted by different countries in a variety of locations around the world. Traditionally, these international meetings are held in person since some host countries do not have the capacity to hold remote or hybrid (remote & in-person) meetings for large groups.

The international education organizations hosted all international NRC and NPM meetings online from March 2020 through all of 2021 due to the global coronavirus pandemic. When the international meetings are again conducted in person, NCES will require contractor attendance at these meetings because the government only allows one federal staff person to attend in-person, and NCES often needs multiple staff to cover the concurrent sessions that take place at international meetings. In general, the contractor should plan to attend the equivalent of thirty

(30) 5-day international meetings, as well as up to 56 local meetings during the contract period.

For remote attendance, there will be no additional costs for travel, meals, and lodging and the complete level of effort is assumed in subtask 2.1. For both in-person international and domestic local meetings, travel, and related costs (lodging, meals, parking, and incidentals associated with travel) will be reimbursed per the allowable per diem rates. Receipts associated with travel and related costs must be submitted to NCES to support reimbursement requests.

The contractor will prepare for the international meetings, attend, and then prepare summaries of the meetings. The government requires the contractor to review advanced material for the meeting and writing clarifying notes or preparing technical support materials. The government also requires meeting summaries to be concise, of roughly five (5) to 20 pages long, and to capture all decisions, next steps, and action items arising from the meetings. The meeting summary will be submitted to the NCES Task Leader within five (5) business days after the meeting.

The contractor will also attend local meetings to help manage activities of international data collections, help draft responses to International Education Organization requests, take notes, and conduct background research to provide recommendations and memos to the NCES study leaders.

• Recommendation memoranda

• Meeting Summary

Subtask 2.1.1: In-person attendance at 20 international meetings (Cost-reimbursable)

This task will require the contractor to participate in 20 face-to-face national, state, local, and international conferences or meetings, as described in Task 2. Preparation for the meetings will be assumed in subtask 2.1. The travel related to conferences, trainings, and other events will be cost reimbursable with appropriate receipts. Meals and lodging will be cost reimbursable, based on the allowable federal per diem rates. Travel and work time while in attendance in person should also be factored into the cost estimates.

• Trip report and receipts related to travel (flight, ground transportation, meals, lodging, and incidental expenses)

Subtask 2.2: Technical evaluations and reviews of standard operating protocols, manuals, forms, study design documents, instruments/adaptations, analysis plans, and results/analyses.

IAB typically requires approximately 70 reviews per performance period across the international studies. These are generally related to pre-data collection products. The format of reviews will be negotiated at the time of the review, but typical formats are tracked changes in text documents, mark ups of hard copies of materials, or memos (or emails) listing, and describing the findings of reviews.

Some consultant time is anticipated under task 2 to provide high-level subject-matter input from recognized experts in the field on topics such as mathematics education, teacher education, measurement, or adult literacy, and on data development tasks.

• Technical and substantive review comments on approved review template

• Responses to questions for clarification of review comments

Task 3. Analysis and Reporting

The t contractor will support the program's mission to provide and disseminate data that describe how well the United States is educating its students compared with other nations. The contractor will provide analysis and reporting support for U.S. participation in PIAAC, PIRLS, PISA, TALIS, TIMSS, ICILS, and IELS as well as other international branch data development efforts.

Task 3 comprises a number of different activities including developing content for web reports and other written material, technically evaluating the completeness and accuracy of products, developing, and maintaining analytic tools and applications, and preparing for and conducting trainings. The contractor will be expected to deliver documents, training material, and presentations that reflect a high level of subject matter input/expertise. As part of analytic and reporting activities the contractor will be required to draft content for release reports and other materials, conduct independent analyses of findings to confirm the accuracy of international results and produce statistics for the reports, monitor the data collection contractor’s analysis and reporting work for international studies and provide recommendations to NCES, review contractor prepared products and materials for release or dissemination, update dissemination tools, and respond to information requests. The contractor will also provide technical and analytic support for training seminars and present IAB study results at conferences. The conferences and trainings may be attended remotely during all or part of a given performance period (see Task 2.1 reference to virtual meetings due to the global pandemic).

The work of this task includes additional details described in the subtasks below.

Subtask 3.1: Content Development

The contractor will provide analytic and dissemination content for 1) web reports, 2) non-routine, ad-hoc, or urgent requests, and 3) brief reports, web pages, and updates to study pages.

Subtask 3.1.1: Web Report (Cost-reimbursable)

These activities will involve developing and updating new webpages for IAB survey results.

The contents will include a cluster of charts and tables with supporting text that describes key findings. Charts and tables may be produced using the International Data Explorer (IDE) or a similar application, or they may be provided by NCES for posting on the web. During this period of performance, the contractor should plan on one web report. For sample web report layouts from past releases, see the TALIS 2018 Results

(https://nces.ed.gov/surveys/talis/talis2018/), PISA 2018 Results

(https://nces.ed.gov/surveys/pisa/pisa2018/#/) or PIAAC 2017 Results

(https://nces.ed.gov/surveys/piaac/current_results.asp).

• Drafts and final versions of reports master plan, outline, wireframe, table shells, tables, report, and 508 compliant PDF version of the report

Subtask 3.1.2. On demand, quick turnaround, and urgent assignments (Cost-reimbursable)

The contractor will respond to the branch chief’s request for information or special analyses related to IAB’s work with various international planning teams and data collections. Activities may include data runs to answer a specific question, developing Q&As, and providing information to NCES to facilitate responses to queries from the public regarding data, methods, and study procedures, as needed (approximately 4 per month). Queries generally relate to https://nces.ed.gov/surveys/talis/talis2018/ https://nces.ed.gov/surveys/pisa/pisa2018/#/ https://nces.ed.gov/surveys/piaac/current_results.asp timing, procedures, instrumentation, or results and locating information. The contractor will plan for up to 48 such activities. For planning purposes, the time generally allotted for quick-turnaround analyses will be 3 full business days.

• Drafts and final versions of quick turnaround memoranda and other analyses

Subtask 3.1.3: Data Points/Stats in Briefs/Other Analysis Products (Cost-reimbursable)

These will be longer term, more in-depth analyses than the on demand or quick turnaround assignments, and often have a longer schedule for development and review then the scheduled initial release web or print reports. The contractor will plan on up to five (5) Data Points and

Stats in Briefs. Stats in Brief are short reports based on focused analysis of a specific topic. They are generally 4-to-15 pages in length and are designed to provide data on policy-relevant topics.

For a description of different types of NCES reports, please see https://nces.ed.gov/statprog/2002/std1_2lista.asp.) The contractor will plan for a full IES review of each. For description of key steps in the review protocol, please see NCES statistical standards and IES procedures for peer review of reports (https://nces.ed.gov/statprog/2002/std6_1lista.asp; https://ies.ed.gov/director/sro/report_review.asp).

The contractor will first submit a report development plan for NCES approval, including schedule for each phase of report development and plans for an IES review of each web report.

• Draft report development plan and schedule

• Final report development plan and schedule

• Drafts and final versions of reports master plan, outline, wireframe, table shells, tables, report, and 508- compliant report

Subtask 3.2: Technical and substantive review of reports (Cost-reimbursable)

Because IAB must be responsive to internal and public requests for information, as well as research pertinent data on which to base sound decisions within a prescribed time period, the contractor will be required to support NCES by performing quick-turnaround (typically five (5) business days), thorough branch level review (using the most current NCES review template) of

1) Up to eight (8) NCES reports (approximately 100 pages including appendices), web and

PDF of approximately 20 pages, up to three (3) rounds of review,

2) Up to five (5) reports produced by international bodies (e.g., a PISA report produced by the international PISA contractor; typically, 500 pages including appendices), with two (2) rounds of review.

3) Condition of Education indicators or other indicators (e.g., Education at a Glance) based on international study data (up to six (6) indicators in total of approximately 10 pages each), with two (2) rounds of review,

4) PIAAC Skills map updates, and https://nces.ed.gov/statprog/2002/std1_2lista.asp https://nces.ed.gov/statprog/2002/std6_1lista.asp https://ies.ed.gov/director/sro/report_review.asp

5) Other brief reports or papers in development, as needed (up to eight (8) documents of approximately 10 pages each).

In addition to the above reports, the contractor will also review technical documentation (see for example U.S. PIRLS and ePIRLS 2016 Technical Report and User's Guide

[https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2019113] and TALIS 2018 U.S. technical report and user guide [https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2021070]).

The contractor shall evaluate products using the following criteria: conformity to standards and policies (including the NCES statistical standards, IES tabular guidelines and stylistic guidelines, and NAEP stylistic guidelines); consistency, quality and editorial correctness, and statistical soundness. Publications must be reviewed, and author responses checked for implementation of the intent of the reviewers, on an often tight schedule that mandates initial release reports to be released according to the International Education Organization's timetable.

The number of products requiring review are based on master survey plans and products that are already in the review pipeline.

• Draft report and indicator technical review plan and schedule

• Final and indicator technical review plan and schedule

Subtask 3.3: Reviews of preliminary data files for quality and accuracy, including restricted-use files (Cost-reimbursable)

Part of the mission of NCES is to make available public- and restricted-use data and documentation after a data collection. The contractor will support NCES by conducting a technical review of data files, codebooks, and technical documentation for the following surveys during the Bridge period: TIMSS PSI, PIRLS, and possibly the TALIS pilot and field test data.

As a part of the data file reviews, the contractor will observe NCES data security requirements.

The contractor will submit a data file and documentation review plan for NCES approval that includes a schedule for completing the required rounds of review.

The objective of the data file review is to identify deviations and errors in the products in order to ensure that final products meet the following standards:

1. There is consistency between .DAT data files and corresponding codebooks, including variable labels, value labels, and frequency distribution for categorical variables, and descriptive statistics for continuous variables.

2. There is consistency of information among related products i.e., data files, codebooks, and data companion.

3. System control files (e.g., SAS, SPSS, and STATA) produce consistent system data files from the .DAT file.

4. All example programs provided as part of the data file package run without errors.

https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2019113 https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2021070

In addition, the contractor will replicate summary data tables using EDSurvey to support the

Centerwide Review for all data files. This may include writing or revising modules for

EDSurvey to ensure all data are valid prior to release.

Subtask 3.4: International Data Explorer (IDE) and EdSurvey v.2.7 Support (Cost-reimbursable)

To support the IES goal of sharing information in formats that are useful and accessible to educators, parents, policymakers, researchers, and the public, the government requires the contractor to maintain and update the International Data Explorer (IDE). The contractor will provide substantive support to IAB for the updating and development of the IDE. The IDE is an online application (https://nces.ed.gov/surveys/international/ide/) that produces tables, charts, and maps displaying results from the IAB surveys and assessments. This support will include activities such as programming and all development necessary to add a dataset to the IDE, updating the Help Guide, and reviewing the content and functionality of the IDE at various stages. The contractor will also work with NCES IAB staff and other contractors, as needed, to prepare a draft and final plan for coordinating the development, support, and reviews of the IDE, as well as carry out the updating and maintenance of IDE. For planning purposes during this performance period, the contractor will assume the addition of up to two (2) datasets (i.e., PIRLS

2021, PISA 2022) during the period of performance.

The contractor will also continue to enhance the EdSurvey R package to support analysis of

PIRLS 2021 and PISA 2022 surveys. Enhancements include developing codes and updating functions to support analysis of data and comparisons of results from other reliable statistical tools or publications. IAB requires EDSurvey modules be written or revised to ensure all data are valid prior to release.

• Draft IDE update plans and schedule

• Final IDE update plans and schedule

• Updated IDE with new data, reviews of IDE updates, and revisions to IDE

• Draft EdSurvey R package update plan and schedule

• Final EdSurvey R package update plan and schedule

• Memo summarizing EDSurvey update completion status

Subtask 3.5: Enhancements to the international large-scale assessments (ILSA) interactive data analysis application (InCompass) (Cost-reimbursable)

The contractor will enhance InCompass, IAB’s interactive data analysis application, by loading one data collection round of one (1) international dataset (e.g., earlier years of TIMSS) for all countries, subject areas, and grade levels. This task will involve modifying the interface in response to client and other stakeholder feedback. This subtask includes (1) developing https://nces.ed.gov/surveys/international/ide/ additional back-end analytical functions requested by the client, (2) developing modules to enable the system to run more efficiently with larger data files, (3) programming adaptations for

508 accessibility, (4) conducting additional quality control of analysis outcomes through replication with existing software (e.g., SPSS, IDE), and (5) creating and running automated tests to ensure the interactive web application runs uninterruptedly and produces consistent and correct results.

The contractor will assist in submitting revised versions of the data analysis application for

NCES Technical Review and responding to feedback from that review. The contractor’s deliverables will include R modules with documentation, regular meetings with the NCES to provide demonstrations and status updates and receive comments, and a full version of an ILSA interactive web application that is accessible to NCES on a developmental server for Technical

Review.

• R modules, code, and technical documentation

• Enhanced ILSA interactive web application populated with selected IAB studies

• Memo summarizing InCompass update completion status

Subtask 3.6: Desk-topping and Publications (Cost-reimbursable)

The contractor will provide desktopping support for products produced under Task 3 including products produced under training and conferences by the International Activities branch. Desk-topping refers to preparing documents for print using professional computer-based tools to layout text and graphical elements. An estimated eight (8) Datapoints and Stats in Brief may need to be desk-topped for publication in a PDF format. Materials for ancillary activities (e.g., glossy tri-fold brochures on upcoming IAB data collections) may also need to be desk-topped.

• Desk-topped products

• Memo or email summarizing completion status

Subtask 3.7: Training (Cost reimbursable)

The training of researchers and other stakeholders to report/use the program's data accurately is a goal of NCES. To support this goal, the contractor will prepare content for NCES’s Distance

Learning Data Training (DLDT) for international datasets (anticipate updating two (2) for performance period); and provide technical and analytic support for an estimated two (2) training seminars on the analysis of school-based international datasets and an estimated two (2) training seminars on the PIAAC dataset at national or regional conferences (such as AERA or CIES).

The contractor will also provide two to four (2-4) additional local or web-based trainings for school-based international datasets and likewise for the PIAAC dataset. Content of the trainings will be based on NCES’s Distance Learning Data Training (DLDT) modules for international datasets, with additional content as appropriate, and may make use of the IDE or similar data analysis application.

NCES has determined that trainings of this nature are best conducted in person to provide participants with individualized support, but many of the training seminars may have to be conducted remotely during this performance period, depending on Departmental guidance on the spread of COVID-19. If the trainings are conducted remotely, travel, parking, meals, and lodging costs will not be reimbursed. For budgetary purposes, the complete level of effort is contained within subtask 3.7.

In order to prepare and deliver the training, IAB will require the contractor to create a draft and final plan that include all steps necessary to attract participants and conduct high quality training such as:

1) reviewing the training notices and application material;

2) advertising the training;

3) reviewing instructional material and completed applications;

4) preparing select portions of the instructional material; and

5) presenting and otherwise assisting at the seminar.

• Draft content from IAB studies for the DLDT

• Final content from IAB studies for the DLDT

• Draft training plans and schedule

• Final training plans and schedule

• Accepted training proposals

• Draft materials for training, remote meeting capabilities, support at the trainings

• Completed trainings; participant satisfaction survey results; debriefing; debriefing summary memo

Subtask 3.7.1: In-person attendance at up to eight (8) training seminars (Cost-reimbursable)

The contractor will provide technical and analytic support by attending face-to-face training seminars at national or regional conferences, as described in subtask 3.7. Preparation for the trainings will be assumed in subtask 3.7. The travel related to conferences, trainings, and other events will be cost reimbursable with appropriate receipts. Meals and lodging will be cost reimbursable, based on the allowable federal per diem rates.

Task 4: Support Dissemination and Outreach Efforts (Cost reimbursable)

The overall goal of Task 4 is to support NCES in sharing the information that it collects and methodologies in formats that are useful and accessible to educators, parents, policymakers, researchers, and the public. In support of this goal the contractor will provide technical, logistical, and managerial support for international survey and assessment dissemination and outreach activities across the studies.

Subtask 4.1: Release activities and product development

The government requires contractor support to help carry out release activities according to a strict timeline. Activities that IAB typically has required to fulfill this subtask include the following:

1) provide content for presentations using existing presentations as models;

2) provide technical support for two (2) report release events (e.g., reviewing presentations, providing Q&As) and ancillary materials;

3) provide content for social media posts prior to report release events and at regular intervals (to be negotiated with NCES study leaders) post release;

4) review IAB home page [https://nces.ed.gov/surveys/international/] and study pages for accuracy and provide updated content on a monthly basis, and supply new content and graphics as needed;

5) produce videos and animations to support two (2) release events and to market IAB’s studies in general;

6) provide other marketing and communications content to alert the public to future report releases and study changes…