PWS - Task Order 14.pdf

Version Date: January 12, 2022

Task Order Title: Assessment Division Support Task Order Type: Firm Fixed Price

Performance Work Statement

Task Order 14: Assessment Division Support

Scope of Work

This NAEP Task 14, Statistical and Psychometric Methodology Support, and NAEP Task 15, Technical Support for Secondary Analysis and Coordination (i.e., Technical Support Task 15), are both required to support the full schedule of NAEP assessments and are separate requiring their own knowledge and skills. However, the individual tasks and related subtasks cannot be accomplished without support from the companion task. For example, a secondary analysis of extant data produced through the Technical Support Task may be required in a methodology report developed under this contract. While maintaining the highest level of quality of work across these two separate NAEP tasks is the primary goal of AD, it would be inefficient to build complete NAEP assessment capacity in each of the two tasks. For, example, since what we expect to report, drives the development of survey and cognitive item development, the staff working on the Assessment Operations Task (3) will need to work closely with the staff working on the Reporting Subtask (4) to insure that items support the final reporting of NAEP results.

Consequently, AD envisions that there will be deliberate and planned coordination of expertise and resources across the tasks and subtasks within and across these two NAEP tasks. To facilitate the coordination process, a requirement for the NAEP Statistical and Psychometric Methodology Support Task 14 is cooperation and coordination with the contractor for NAEP Technical Support Task 15.

Independently and not as an agent of the United States Government, the Contractor shall provide all personnel, materials, services, and facilities necessary for the project and perform the tasks as described below.

Task 1: Research Under this task, the contractor shall conduct research studies, disseminate the results of such studies, attend research meetings, support NCES in checking the quality of research conducted by several contractors, and support NCES with analyzing existing NAEP data. Research to better understand the impact of COVID on NAEP performance should be prioritized in the context of the 2022 LTT and Main NAEP results.

Statistical and psychometric research studies conducted by the contractor serve several purposes:

1. Develop the psychometric soundness and precision of NAEP assessments: Examples of past research activities include a study exploring the construct equivalence of the NAEP reading comprehension assessment in the presence of a new item type (meaning vocabulary items).

2. Adapt NAEP assessments to changing situations and different populations: For example, in 2009, three studies examined factors that might have presented challenges in NAEP’s 2007 Puerto Rico mathematics assessment. These studies were designed to improve the 2011 Puerto Rico mathematics assessment.

3. Enhance understanding of NAEP results: For example, in 2009, a statistical link was established between the NAEP and the ECLS-K grade 8 reading assessments. Linking NAEP to a longitudinal assessment will help explain how past performance in reading assessments are related to performance on NAEP’s grade 8 reading assessment.

4. Explore new ways of modeling NAEP data: An example is the research conducted in 2015 regarding the feasibility of creating and applying teacher weights.

In addition, since 2015, NAEP has collected “observable” or “process” data through the digital-based assessments, where each key stroke and interaction with the assessment is captured with a time stamp. NCES is interested in maximizing the potential of this data to deepen understanding of student behavior and engagement with the assessment, inform decisions on item, assembly, development, and operations, and to analyze and report data. Therefore, NCES expects a portion of resources for research to support the Center for Process Data established in during the previous contract period of performance. This Center would support the preparation and dissemination of process data files so that files are available and engaging to a broad audience.

Activities may include, but are not limited to: characterizing and helping set standards for the structure of process data including security of items and protecting personally identifiable information; implementing quality control procedures, identifying and producing meaningful derived variables; identifying procedures for preparing process data for dissemination;

supporting the release of process data files and supporting documentation; and providing training and resource materials for users of process data.

In order to make studies conducted by the contractor accessible to the research community and to enhance its own studies, the contractor shall participate in research conferences. NCES also provides data training for the larger research community, primarily at research conferences. The contractor shall support NCES in preparing and conducting such training sessions. The contractor shall attend NAEP-related research meetings and conferences in order to facilitate the exchange of knowledge within and outside the NAEP research community.

NCES requires other NAEP contractors to conduct innovative psychometric and statistical studies. The contractor shall support NCES in making sure that such studies are conducted in accordance with statistical standards, are of high-quality and yet accessible to the larger research community. In addition, the contractor shall support NCES in analyzing existing NAEP data in response to special investigations. Because of the complex nature of NAEP data, such analysis requires a deep understanding of how NAEP assessments are conducted and how NAEP data are scaled.

Under this task, the contractor shall provide statistical and psychometric expertise to assist NCES in the planning and quality control of NAEP research activities described in Subtasks 2.a - 2.e and to provide statistical and psychometric support to tasks 3 and 4 and to the Technical support Task contractor.

Subtask 1.a: Conduct Statistical, Psychometric, or Operational Studies The contractor shall conduct up to 5 studies on emerging NAEP issues to assess the value of potential changes in the technical design of NAEP, and the analysis of NAEP data.

Below is a list of potential areas of research studies (priorities for these studies will be discussed at contract kickoff meeting):

• How can NAEP contribute to the “learning loss” conversation given the NAEP design

• Studying the impact of school closures on NAEP performance.

• Analyzing context (“neighborhood” of targeted test item) and position (practice and fatigue) effects on item functioning as reflected in IRT parameters.

• Applying Rasch scaling methodology to high school transcript data so as to detect differential functioning of courses with the same (or similar) titles across SES and/or racial/ethnic groups.

• Exploring validation of the TEL construct using MIRT approach and ECD model.

• Exploring of mode effects in the Writing assessment using 15 “common” items for grade 4 (not much concern of position and context effects for writing assessment).

• Testing apart reading proficiency from other subject area knowledge and skills using MIRT.

• Examining of reporting options for an integrated social studies assessment.

• Exploring options for an NAEP-HSLS link study that relate 12th grade NAEP to outcome data.

• Conducting an independent evaluation of automated scoring engines offered by Pearson and ETS. Does one do a better job at one subject (writing vs. math or reading) than another?

• Examining the feasibility of extending NAEP or a NAEP-like assessment to grades kindergarten through 3rd given the President's early learning initiative

For each approved study, NCES shall request a study or work plan. The complexity and profile of the proposed study will determine the level of detail necessary for the study plan. The contractor shall deliver the study plan within fifteen calendar days of the request. The COR will approve the study plan within fifteen calendar days of receipt of the final draft plan from the contractor. The contractor shall submit final reports on a schedule determined with the NCES.

In addition, the contractor shall conduct up to ten research memos for short research explorations on emerging NAEP issues, assessment of the value of potential changes in the technical design of NAEP, and the analysis of NAEP data

Below is a list of potential areas for research memos (priorities for these memos will be discussed at contract kickoff meeting):

• Feasibility of analyzing non-state regional sub-samples of NAEP data

• Exploration of measures of SES

• Cometrix analyses of reading passages

• Review of external research

The contractor shall commit staff to a Center for Process Data. The Center will support the release of process data files and supporting documentation. The contractor will meet with the NCES contact within the first week of the contract start and provide an update to the 3-year plan prepared during the previous contract’s period of performance. The plan shall provide the steps and actions necessary to release the process data files, first look reports, and supporting documentation. The plan shall also include timelines and actions for training, dissemination, future data file releases, and other support activities. The contractor will revise the plan based on NCES feedback and submit deliverables on a schedule determined with NCES.

Subtask 1.b: Facilitate exchange of NAEP knowledge among major NAEP stakeholders, in research-focused meetings and panels.

Under this task the contractor shall attend up to 10 meetings per year. If there is a virtual option for these meetings, that will be the default way of participating UNLESS approved by COR.

These meetings could include:

1. As requested by the COR, attend research focused NAEP-related meetings such as the

Design and Analysis Committee (DAC) and take notes on technical discussions for later review.

2. Attend and present NAEP research findings on behalf of NCES or with NCES staff at key research conferences approved by the COR. Conferences include the American Education Research Association/National Council on Measurement in Education in San Francisco, the National Assessment Governing Board meeting in El Paso Texas, the Educational Data Mining conferences in Morocco, and the Society for Research on Adolescence in San Diego.

3. Provide or support NAEP data training at such research conferences as requested.

After attending these meetings, the contractor shall summarize relevant technical issues pertinent to the work of Subtask 2. In consultation with NCES, the contractor shall identify staff (as many as 8 staff) to attend these meetings. Other potential meetings include: (DAC) Design and Analysis Committee (4/year in Washington, DC), (NVS) NAEP Validity Studies Panel (4/year in

Washington, DC), the Printing Planning Meeting, NCES National Indian Education Study meetings, and (NAGB) National Assessment Governing Board (4/year, 2 in Washington, DC).

If needed, the contractor shall assist the COR in coordinating with the Technical support Task contractor to ensure appropriate levels of technical support.

Subtask 1.c: Research Dissemination and Management Support The contractor shall develop and manage a program area for coordination, summarization, and dissemination of NAEP research and development efforts. Activities include:

• Review and categorize research and development studies and reports on NAEP and enter these into a searchable research database.

• Conduct activities related to dissemination of NAEP research to a larger research community: Develop a NAEP Findings Web publication in which studies addressing key topics in large scale assessment are summarized. Each issue shall address one key research topic for the purpose of informing stakeholders and researchers of important findings in NAEP research that are relevant to policy and/or assessment practice. Formats might also include monographs, subject matter books, and book chapters, as peer-reviewed journal articles, and NCES special reports.

• Conducting outreach and training sessions (e.g., webinar). Activities include providing website materials and announcements, as well as in-person activities, mostly in the DC area, but also including AERA.

• Providing dissemination and support to EdSurvey.

Subtask 1.d: Provide statistical and psychometric technical support for Tasks 3 and 4 and for the Technical Support Task The contractor shall provide statistical and psychometric support on an as-needed basis for tasks 3 and 4 and for the Technical Support Task 2 under this procurement. The members of this subtask staff shall serve on report review panels for the reporting subtask and for the secondary analysis task. Other cross task activities under this subtask may include, for example:

• Support NCES in the review and evaluation of NAEP Framework requirements and item specifications/statistics for providing support to the operations task (3).

• Provide support to the reporting task (4) for ensuring the statistical and psychometric quality of all data and analyses in NAEP paper and electronic reports.

• Provide statistical and psychometric support to the secondary analysis task (i.e., the Technical Support Task) in ensuring the quality of quick turnaround analyses, technical documentation, quality assurance plans, and the preparation of technical memoranda.

• Provide other technical support to NCES as requested.

Subtask 1.e: Utilize operations and reporting support from tasks 3 and 4 and from the Technical Support Task into specific work assignments The contractor shall utilize, on an as-needed basis, resources and expertise from tasks 3 and 4 and from the Technical Support Task under this procurement to ensure the highest quality products. This subtask may, for example, access support from other NAEP support areas including:

• NAEP Framework requirements and item specifications/content expertise from task 3.

• NAEP analysis and reporting requirements from task 4.

• Plans for updating and improving data tools and other information storage products to make information more accessible from the Technical Support Task.

The contractor shall update the deliverables list as appropriate, and document all changes including reason for the change. The COR shall provide monthly notification to the CO when a deliverable does not meet its deadline. To facilitate communication of information between the NAEP Statistical and Psychometric Methodology Support task and The Technical Support Task contractors, the Technical Support Task contractor will meet with the Statistical and Psychometric Methodology Support contractor and NCES for weekly updates meetings.

Subtask 1.f: Continued Development of R package software for NAEP and NCES The contractor will provide enough programmer support to continue the development of the R package, EdSurvey, so it includes more advanced analysis function and the exploration of including more NCES data sets. Activities under this subtask include the following:

• Activity 1: Develop R package for all NCES data sets:

o Plan and develop analytic functions for approximately three additional NCES data sets.

• Activity 2: Expand EdSurvey Functions o Continue developing item response models for NAEP data o Develop a plausible value generation function for NAEP data o Continue improving mixed models for NCES data o Continue developing missing data imputation functions for NCES data

• Activity 3: Develop an R Interactive Interface for NCES Data Display o Continue developing item response models for NAEP data o Develop a plausible value generation function for NAEP data o Continue improving mixed models for NCES data o Continue developing missing data imputation functions for NCES data

• Activity 4: Outreach and Training o Training (up to 3) o At least one report or one journal article for the EdSurvey package o Provide documentations (vignettes) for new EdSurvey functions and the iNRD web-based app.

o Website materials and announcements

• Activity 5: Maintenance of the released R Modules o Update R functions for upcoming NAEP data o Bug fixes o Respond to user requests and questions

Task 2: Assessment Operations Under this task, the contractor shall provide subject area, operational, logistical and editorial expertise to assist NCES in the planning development, preparation review and quality control of NAEP assessment operations. AD monitors all item development, materials development, and scoring for the NAEP program. The goal of this task is to ensure that all NAEP survey and cognitive items, assessment materials, and scoring are of the highest technical quality and meet generally accepted NCES and assessment industry standards as valid and reliable measures of the constructs envisioned in the NAEP survey and content frameworks approved by the National Assessment Governing Board.

Under this task, the contractor shall specifically assist NCES in the planning development, preparation review, and quality control of NAEP assessment operations including survey and cognitive item development (subtask 3.a), materials preparation, review and scoring (subtask 3.b) activities.

Subtask 2.a: Provide Planning and Quality Control Review Support for Survey and Cognitive Item Development The contractor shall provide support, technical assistance content expertise, and when appropriate, translation support for ensuring the quality of NAEP cognitive items and survey variables for all NAEP surveys and assessment delivery platforms (Main National, State, and District NAEP, Long Term Trend NAEP, the NAEP High School Transcript Study, and special studies including the National Indian Education Study).

Development of the cognitive and survey items takes place at least three to four years prior to the operational assessment. Within this time period, as part of cognitive and survey item development, items are pretested via small-scale question tryouts and cognitive laboratories) and piloted one to two years prior to the operational assessment. The contractor shall provide support relevant to various stages in development.

Survey variable development involves collecting important relevant contextual information to inform the interpretation of NAEP assessment results. Survey item and questionnaire development includes work in three areas: student reporting categories, contextual and policy information, and subject-specific information. The contractor shall provide recommendations for improving the quality of NAEP questionnaire and survey variable development and shall provide assistance in planning and reviewing activities related to the development of NAEP survey questions and questionnaires, including online surveys, and quality control review of existing survey questions to ensure quality of NAEP survey questionnaires of computer-based assessments.

Cognitive item development includes selected-response and constructed-response digital assessment items, scenario-based tasks, and hands-on task formats. NAEP continues to transition from paper-based assessments to digital-based assessments which include new interactive item types. The contractor shall provide recommendations for improving the quality of NAEP cognitive items and shall provide assistance in planning and reviewing activities related to the development of NAEP cognitive items including computer-based assessments.

Assessment delivery platform development includes student response platforms for the delivery of survey and cognitive assessment items and response platforms for the teacher and school survey questionnaires. The contractor will provide recommendations for improving the quality of the NAEP assessment delivery platforms and shall provide assistance in planning and reviewing delivery platform features and content associated with all delivery system platforms.

Design and analysis activities include design plans, analysis plans and item development quality control meetings to review pilot and operational data. The contractor will provide recommendations for improving the quality and clarity of NAEP’s design and analyses, provide assistance in strategic planning and provide reviews of design and analysis plans for pilot and operational assessments as well as special studies.

More specifically, as part of planning and review activities for survey and cognitive item development and assessment delivery platform development the contractor shall:

• Provide support in strategic planning of assessment development and review processes, products and procedures.

• Conduct, coordinate and document cognitive items and survey questions reviews for pilot and operational assessment development, provide input and documentation to support the review of computer-based assessment interface and associated tutorials.

• As part of ongoing item development and review, review scoring rubrics for all new or revised pilot constructed response items for accuracy, completeness and clarity.

• Provide technical and content support for reviewing non-item deliverables (such as ancillary assessment support documents, development schedules, and FAQs), handoffs, or milestones from multiple NAEP contractors.

• Attend framework, specifications, and item and survey development NAGB sponsored meetings and document important issues for NCES to consider.

• Support NCES in the development or updating of item development framework and specification clarification documents for specified content areas.

• Serve as subject area task leaders or provide other support, such as item reviews, in helping NCES ensure the quality and timeliness of mathematics and science item development.

• Provide support in monitoring the review of scoring materials for all selected-response machine scored items and human-scored constructed-response items.

• Observe field test, pilot, operational and special study administration activities and provide field notes.

• Observe pilot and operational scoring activities and provide field notes.

• Convene an expert panel to conduct an independent content review of new items.

• Assist in the planning and review of the cognitive and survey item/question development and review schedule.

• Prepare for, attend and provide summary notes of the a) monthly Item Development (ID) Contractor Steering Committee meetings, b) weekly conference calls with the ID contractor,

c) all subject area standing committee meetings, d) survey questionnaire-related standing committee meetings and/or panel meetings, and e) translations of assessment/survey items and associated assessment materials.

• Review and prepare other meeting or briefing materials as requested.

• Support NCES in coordinating responses to requests for public or researcher access to secure NAEP items, tracking requests, maintaining the Secure Item Request Database and providing access to secure NAEP items under secure conditions.

• Provide technical assistance to states and TUDAs related to the content of the NAEP frameworks, specifications or item development activities through the NAEP State Service Center contract.

• Convene, prepare for and attend additional meetings of NAEP contractors or expert panels relating to assessment development, delivery platform, design and analysis, and scoring, as requested.

• Conduct literature reviews, small-scale investigations or cognitive labs related to cognitive items development, delivery platform, design and analysis, and scoring as requested.

• Plan, conduct or coordinate methodological studies of new item types as requested.

• Review items for potential use in special studies as requested by NCES.

• Support NCES in the review of all documentation for NAEP information clearance requests (ICR) to OMB, as specified by the COR

• Review all certified directions and blocks for pilot and operational cognitive and survey items produced for administration and document comments into a spreadsheet as requested.

• Provide content and operation staff to the Center for Process Data

• Support investigations using process data to help inform operational decisions

• Provide expertise and technical support to NCES regarding DAR activities and DAR deliverables.

Subtask 2.b: Material Distribution, Processing, and Scoring Support The contractor shall provide management support, technical assistance and content expertise for improving the quality of NAEP cognitive items and survey questions for all NAEP surveys (Main National, State, and District NAEP, and special studies including the National Indian Education Study). The contractor will provide support for ensuring quality of printed assessment booklets, supplemental paper-based questionnaires and ancillary materials. The contractor shall provide support for improving the quality of NAEP scoring and shall provide assistance in operational planning, document preparation and review, observation of training and scoring, and ongoing support for the progress and quality of scoring as these pertain to the scoring of paper and pencil and computer-based assessments. This will also include ongoing support for development, preparation, review and scoring activities related to bilingual items and assessment materials. The contractor shall utilize contractor staff from the Technical Support Task to assist with item and scoring activities, as needed.

More specifically, these activities require that the contractor shall:

• Provide support in strategic planning of assessment development processes, products and procedures.

• Assist in the planning of the printing schedule and conduct page-proof and booklet reviews for printed assessment materials

• Support the review and provide feedback to the annual Scoring Memorandum.

• Support the review and provide feedback to the Scoring Training Handbooks for each content area being piloted or administered in each assessment year.

• Support the review, and provide feedback to the annual Materials, Distribution, Processing and Scoring (MDPS) contractor Report of Activities and appendices.

• Review and recommend specifications for scoring and training procedures, monitoring statistics, and electronic (computer-based) scoring functionality and MDPS daily and summary scoring reports.

• Facilitate the development of the spreadsheets to detail item response categories for each constructed response item, in preparation for computer-based scoring.

• Assist in the planning of the annual training and scoring schedule.

• Support NCES to monitor scoring statistics discussed during daily subject area conference calls between scoring, item development and analysis contractors.

• Maintain records of daily scoring statistical reports (Completion Report, TRMC, and other contractor summaries) and phone calls.

• Review page proofs for blocks and booklets of printed assessment materials, computer-based assessment materials, and ancillaries.

• As part of ongoing item development and review, review scoring rubrics for all new or revised pilot constructed response items for accuracy, completeness and clarity.

• Assist with the coordination of open-ended item scoring training set selection and annotation, approval and assembly.

• Observe pilot and operational scoring training activities and NAEP scoring and make recommendations for improvement.

• Prepare for and attend the monthly MDPS Contractor Steering Committee meeting.

• Prepare for and attend the annual Scoring Debriefing meeting and track the completion of the

Report of Action Items, to completion.

• Prepare for and attend the annual Scoring Planning meeting and track the completion of any action items.

• Provide technical content support and tracking for reviewing deliverables, handoffs, or milestones from the multiple NAEP contractors

• Review and prepare meeting or briefing materials as requested.

• Maintain and review records of MDPS contractor schedules and tracking reports.

• Convene, prepare for or attend additional meetings of NAEP contractors or expert panels relating to MDPS activities as requested.

• Conduct literature reviews, small-scale investigations or cognitive labs related to scoring.

• Plan, conduct or coordinate scoring studies related to new or special item types.

Subtask 2.c: Provide conceptual and content support for tasks 2 and 4 and for the Technical Support Task The contractor shall provide conceptual and content support on an as-needed basis for tasks 2 and 4 and for the Technical Support Task under this procurement. The members of this subtask staff shall serve, as needed, on report review panels for the reporting subtask and for the secondary analysis (i.e., Technical Support) task. The Technical Support Task contractor shall coordinate support on an as-needed basis for this subtask. In addition, contractor staff from the Technical Support Task 2 staff shall assist with item, materials and scoring activities as needed.

Additional activities under this subtask may include support such as:

• Review and evaluate NAEP Framework requirements and item specifications/statistics for various linking studies conducted by the research task 2.

• Provide support to the reporting subtask 4 for ensuring the proper interpretation of NAEP survey questions and cognitive items in NAEP paper and electronic reports.

• Provide content and conceptual support to the secondary analysis task 2 for ensuring the quality and usefulness of quick turnaround analyses, technical documentation, quality assurance plans, and the preparation of technical memoranda.

Subtask 2.d: Utilize research and reporting support from tasks 2 and 4 and from the Technical Support Task in specific work assignments The contractor shall utilize, on an as-needed basis, resources and expertise from tasks 2 and 4 and from the Technical Support Task under this procurement to ensure the highest quality products. In addition, the contractor will provide psychometric, statistical, technical expertise/support in ensuring the quality of NAEP instrument design and analysis. This subtask will access and coordinate subject area, psychometric, statistical and reporting expertise across the task to support review of NAEP design and analysis activities. More specifically, as part of the NAEP design and analysis support activities, the contractor will:

• Provide statistical and psychometric expertise from task 2 for ensuring the quality of items based on item statistics from pilot and operational testing and scoring.

• Review NAEP analysis and reporting requirements to ensure that survey and cognitive items are sufficient to support reporting needs.

• Conduct reviews of design and analysis memos for NAEP pilot and operational assessments, PBA/DBA and other linking studies, and special studies conducted by the NAEP Alliance or other NAEP or NCES contractors.

• Prepare for, attend and conduct small-scale follow-up investigations from Item Development

Quality Control meetings and IDQC Plausibility meetings.

• Prepare for and attend Design and Analysis Committee meetings.

• Review DAR contractor deliverables related to responses to questions on NAEP data.

• Provide technical support and tracking or reviewing deliverables, handoffs, or milestones from the NAEP Design and Reporting contractor.

• Assist with plans for updating and improving NAEP item tools and other information storage products to make information more accessible from the Technical Support Task.

Task 3: NAEP Reporting and Dissemination Support The contractor will provide technical reviews of all NAEP reporting products. These include the web-based Nation’s Report Card for all subjects, web-based supplemental reports, web-based technical documentation for each operational NAEP assessment, all data files (in preparation for release to the public), and their accompanying codebooks and data companions Additionally, the contractor will provide a quality check of the data used to prepare the Nation’s Report Cards (replicate summary tables with EdSurvey).

The contractor shall provide support for the review for all NAEP reports and shall support specified research activities related to NAEP reporting. Other report-related products may include statistical analysis reports, one-page “snapshot” reports or webpages for each participating state and urban district, computer-generated state and district reports, web pages, related data files and documentation, and technical documentation that describes the process of administering and analyzing the year’s assessments.

Additionally, the contractor shall develop at least two NAEP reports. For example, a secondary analysis report of COVID variables that provides descriptive statistics across demographics.

Additionally, the contractor shall provide support for releases of reports and presentations of results, both for reports the contractor produces and other Assessment Division reports. Support may include development of PowerPoint presentations and/or scripts, analyses of findings to discover important relationships in the data and interesting patterns that would inform policy makers and stimulate public discourse, and design of graphics or other materials to enhance release of findings.

The contractor shall utilize contractor staff from the Technical Support Task to assist with report reviews and to serve on report review panels as needed. In addition, contractor staff from the Technical Support Task shall provide support to this reporting task for ensuring the statistical and psychometric quality of all data and analyses in NAEP paper and electronic reports.

Subtask 3.a: Assessment Division Review The goal of the Assessment Division Review project is to provide a division review of NAEP publications. The contractor shall provide technical, content, editorial review and coordination support for each stage of division review. The contractor shall evaluate products using the following criteria: conformity to standards and policies (including the NCES statistical standards, IES tabular guidelines and stylistic guidelines, NAEP stylistic guidelines, and NAGB reporting policy); consistency, quality and editorial correctness, and statistical soundness. Publications must be reviewed, and author responses checked for implementation of the intent of the reviewers, on an often-tight schedule that allows certain NAEP reports to be released within 6 months of the end of data collection, and others within 12 months.

In 2022-2023, NCES anticipates reviews to support 2 major report release

• Long term trend age 9

• Mathematics and Reading, National, State and TUDA

Activities require that the contractor shall:

• Conduct an initial division review; entering review comments and providing recommendations in tracker (IMS Content Forward [IMS CF]).

• Submit comments to the AD Review Chairs and the Program Director for Design, Analysis, and Reporting with a technical review memo memorandum which summarizes major issues identified from the initial review.

• Attend division reviews, and enter final decisions reached at revision review meetings in IMS

CF.

• Conduct a side-by-side division review, entering decisions in IM CF.

• If requested, conduct a post Center-wide or IES re-review.

• Coordinate review meetings for print publications and provide a summary of decisions reached at reconciliation meetings by attending meetings and synthesizing reviewers’ comments.

• If requested, coordinate other AD review meetings.

Reports expected to be reviewed in web formats as were prepared during the previous contract’s period of performance, (two separate reviews for each) include the following:

• Mathematics and Reading, National, State and TUDA

• Long Term Trend age 9

In addition to the above reports, the contractor will also review report generators and snapshots for the states and urban districts, and technical documentation (approximately eight chapters).

The contractor will also review data files, codebooks, and data companions for the grade Long Term Trend 9 year old and Mathematics and Reading, National, State and TUDA assessment.

Activities will be based on the NAEP Assessment Schedule and related special studies. As a part of the data file reviews, the contractor will observe NCES data security requirements Data file reviews shall include:

1. Load data file supplied by NCES to a secure restricted-access drive.

2. Modify and run a quality check program to check consistency between .DAT data files and corresponding codebooks, including variable labels, value labels, and frequency distribution for categorical variables, and descriptive statistics for continuous variables.

3. Check inconsistency between SAS scoring macro and corresponding answer keys and item parameter data in the codebook.

4. Check if SAS, SPSS, and STATA system control files produce consistent system data files from the .DAT files.

5. Run example programs as provided as part of the data file package to check if they run without errors.

6. Run NAEPEX to process raw data on selected files to make sure it is problem-free.

7. Estimate average scores and their standard errors for selected groups using plausible values methodology and direct estimation by running AM and compare the results with the corresponding estimates from NDE.

8. Submit summary memorandum with review comments and recommendations and an error report.

9. Conduct a side-by-side (SBS) review of data products, i.e., data files, codebooks, and data companion after authors respond to NCES/contractor comments and submit a technical review memorandum summarizing SBS review comments and decisions, and major outstanding issues.

In addition, the contractor will replicate summary data tables using EDSurvey to support Centerwide Review for all NAEP data files. This may include writing or revising modules for EDSurvey to ensure all data are valid prior to release.

The contractor will also provide review support for post-Division review for Technical Documentation on the Web for all outstanding years.

Subtask 3.b: NAEP Report Development Activities The contractor shall contribute to two NAEP report development activities by participating in planning, writing, and analyzing data for NAEP reports. The contractor shall plan for general report development activities and two reports (all to be determined by or pending approval from

NCES):

If these are not the priorities of NCES, the contractor shall work with NCES to reallocate the funds accordingly to other report efforts. As a part of report development activities, the contractor shall:

1. Hold publication planning meetings.

2. Develop plans for the publications.

3. Develop outlines for the reports.

4. Produce table and figure shells.

5. Produce initial layouts and designs in wireframes. Hold review meetings and obtain NCES approvals.

6. Produce drafts of the reports for NCES review.

7. Respond to reviewer comments (pre-division review).

8. Produce and submit a division review-ready report.

9. Respond to reviewers’ comments and revise the reports (Division review, Center-wide review, Commissioner Review, IES review).

Additionally, as requested, the contractor shall support NCES in finalizing ad hoc publications and peripheral products (e.g., Spanish translations). The contractor shall work with the COR to determine the appropriate priorities and schedules.

The contractor shall utilize staff from the Technical Support Task to provide support to ensure the statistical and psychometric quality of all data and analyses in NAEP paper and electronic reports.

Subtask 3.c: Ad Hoc Technical Review Support In accordance with the contractor role of providing ad hoc technical review services and support for NAEP reporting, the contractor activities shall include the following:

• Coordinate and train report review staff on tools, procedures, and guidelines.

• Prepare for and attend review coordination meetings.

• Prepare for and attend biweekly Technical Documentation Website Working Group meetings. Prepare for and attend monthly NAEP Data Products Coordination meetings with NCES and ETS

• Participate in Outreach Coordinating Council meetings.

• Participate in QC Plausibility meetings.

• Support NCES in presentations to and analyses for the Reporting and Dissemination Committee meetings of the Governing Board, attend the meetings, and provide summary notes of issues, tracking follow up assignments related to processes and products.

• Provide other technical support for reporting as requested.

Additionally, the contractor shall review up to 3 technical memoranda, working papers and research reports submitted by other NAEP contractors. For example, NCES expects that the contractor shall review papers prepared as task orders.

The contractor shall utilize staff from the Technical Support Task 2 to report and serve on report review panels and to provide support to ensure the statistical and psychometric quality of all data and analyses in NAEP paper and electronic reports.

Subtask 3.d: Provide analysis and reporting support for tasks 2 and 3 and for the Technical support Task The contractor shall provide analysis and reporting support on an as-needed basis for tasks 2 and 3 and for The Technical Support Task under this procurement. The members of this subtask staff shall serve on report review panels for the secondary analysis task (i.e., Technical Support Task).

Additional activities under this subtask shall:

• Provide support to the research task 2 for ensuring the statistical and psychometric quality of all data and analyses in NAEP paper and electronic reports.

• Review and evaluate NAEP reporting requirements for defining survey and cognitive item needs to the operations task 3.

• Provide support to the secondary analysis task (i.e., Technical Support Task) for NAEP and

NCES reporting standards and procedures for use in quick turnaround analyses, technical documentation, quality assurance plans, and the preparation of technical memoranda.

The contractor shall utilize contractor staff from the Technical Support Task to provide support to the reporting task (4) for ensuring the statistical and psychometric quality of all data and analyses in NAEP paper and electronic reports.

Subtask 3.e: Utilize operations and reporting support from tasks 2 and 3 and from the Technical Support Task in specific work assignments.

The contractor shall utilize, on an as-needed basis, resources, and expertise from Subtasks 2.b and 3.c and from the Technical Support Task under this procurement to ensure the highest quality products. This subtask may access support from the following other NAEP support activities:

• Review analysis procedures to ensure accuracy and efficiency of reporting including serving on report review panels.

• NAEP Framework requirements and item specifications/content expertise from task 3.

• NAEP paper and electronic reporting requirements from task 4.

• Assistance with plans for updating and improving data tools and other information storage products to make information from task 2 in the Technical Support Task more accessible.

Task 4: State Analysis Support The Department of Education (ED) requires technical support to its assessment development, methodological research preparation, and developing analytical models (statistical and psychometric) related to state NAEP.

Subtask 4.a. Generation of Full Population Estimates (FPE) The contractor shall produce FPEs of NAEP score averages for grades 4 and 8 in reading, mathematics, writing, and science for each year these assessments are administered. Although NAEP provides accommodations for some students who could not participate in NAEP under standard testing conditions, some students selected in the sample are identified as having disabilities or not being proficient enough to take the test in English and are excluded from the assessment.

Since exclusion rates vary widely across states, concerns have been expressed about the comparability of the estimated state NAEP distributions. The estimates of the NAEP equivalent scores are affected to the degree that comparability is impaired. NAEP estimates the proficiency of the excluded students based on the information collected by NAEP SD/LEP Questionnaire.

The contractor needs to produce FPE estimates for reading, mathematics, writing, and science results for the excluded students in each state, for the nation and for participating urban districts.

These estimates will be used as an alternative to what NAEP reports officially for each participating state and jurisdiction. These alternative estimates are also shared with the public by posting them on NAEP website along with the initial release of assessment results. The contractor will provide such data to Government contractor 6-weeks prior to the report release.

Find more about FPE at: http://nces.ed.gov/nationsreportcard/about/effect_exclusion.asp Subtask 4.b. Achievement profile analysis of various race/ethnic groups The contractor shall conduct analyses to provide an in-depth picture of the academic achievement and achievement gap-closing patterns. This study will provide a basis to profile states and state subgroup(s) that are performing on par with the nation, leading the nation or lagging behind, and the states that are making significant changes in the achievement gap between subgroups. The pattern of academic achievement and changes in the achievement gap will be studied for all students including high and low performing students using various NAEP data, for example, scale score, percent by achievement level and composite scores, and strand scores. This study will incorporate background information collected by NAEP and supplement the findings with other publicly available information as necessary.

Subtask 4.c. State mapping studies The contractor shall estimate NAEP equivalent scores for state assessments using 4th and 8th grade percent proficient reported by states in Reading and Mathematics. The purpose of this mapping is to show where the state standards fall in relation to NAEP proficiency standards and across the states. In addition to mapping the state standards, the contractor shall evaluate the stability of this mapping analysis and gauge the gains that the states report. These mappings http://nces.ed.gov/nationsreportcard/about/effect_exclusion.asp involve collecting state assessment data and meta-data from the state websites or other publicly available sources and collecting information on state assessments through a survey. The survey should be an efficient way to communicate between NAEP State Coordinators and State testing directors. All information needs to be verified by each state and through the NCES review process before considered final. Find more about the mapping study at:

https://nces.ed.gov/nationsreportcard/studies/statemapping/.

Subtask 4.d. Develop/design statistical linking studies This subtask is designed to test the feasibility of integrating or linking the NAEP state database with other state, federal, or international databases. One such project could be the link between the 2007 Third International Mathematics and Science Study (TIMSS) and the 2007 NAEP.

Other databases include the National Teacher and Principal Survey (NTPS) (previously known as the School and Staffing Survey (SASS)). Possible uses of linked databases could potentially:

• Provide decision makers with performance of American students relative to those in other countries;

• Enhance the analytic power of NAEP;

• Provide additional background information on American students related to NAEP scores;

and

• Increase the reliability and efficiency of the indicators of student academic achievement.

Review all international, federal, and state databases, paying particular attention to databases that collect information on the educational condition of states, districts, schools, and student achievement. Once the data elements in each of the databases are identified, the contractor shall review the data elements in each database, classify items as complementary or redundant to those that NAEP provides, and identify those elements that will enrich the NAEP achievement measures. This may lead to future modifications of data collection processes in future assessments.

In addition, the contractor shall identify the levels (that is, student/school/ district/state) at which data can be linked, the limitations inherent in linking the data, and determine strategies and methods for linking the sources of data. Moreover, the contractor shall evaluate the impact of differences in time of data collection for their potential impact on validity. In all data analysis activities, the contractor shall maintain data security and the confidentiality of individual students’ data.

Task 5: NAEP Research and Development Program This subtask consists of four interrelated activities: NAEP Training Workshops, NAEP Young Scholars Program, NAEP Post-doctorate Program, and NAEP commissioned work. The purpose of this task is to create a mechanism to support development and innovations in large scale assessment and move the NAEP program forward.

Subtask 5a. NAEP Training Workshops https://nces.ed.gov/nationsreportcard/studies/statemapping/

The contractor will support NCES in a series of NAEP Training Workshops. The training should include sessions on data available through the NAEP program (e.g., response and process data;

NAEP assessments; NAEP Special Studies); the operations of NAEP and its challenges;

methodological considerations of NAEP data; reporting and analysis of NAEP data; and other NCES data sets, such as the Civil Rights Data Collection (CRDC), Teaching and Learning International Survey (TALIS), and the High School Longitudinal Survey (HSLS). Training materials shall draw on those included in the NCES Distance Learning Data Training (DLDT) system. If new materials are developed, they shall be prepared for inclusion in the DLDT.

Subtask 5b. Young Scholars Program Under this subtask, a summer internship program will be created for quantitative…