Attachments_A_-_G_-_RFP_91995918R0002.pdf

RFP# 9199595918R0002

Attachment A: Statement of Work

UPDATE OF NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS

(NAEP) FRAMEWORKS FOR MATHEMATICS, READING, AND OTHER

SUBJECTS

RFP# 91995918R0002

ATTACHMENT A: STATEMENT OF WORK

MAY 11, 2018

TABLE OF CONTENTS

A. CONTEXT OF PROCUREMENT

A.1. Purpose

A.2. Overview of the Governing Board

A.3. NAEP Frameworks and Assessments in Mathematics and Reading

Current NAEP Mathematics Framework

Current NAEP Reading Framework

NAEP Mathematics and Reading Assessments

A.4. Role of the Board in Framework Updates

A.5. Milestones for Framework Updates

B. SCOPE OF WORK

Major Deliverables

Overview of Tasks

B.1. Task 1: Project Launch

B.2. Task 2: Project Management

Weekly Project Teleconferences

Monthly Progress Reports

Plans for Item/Data Security

Logistical Plans

B.3. Task 3: Governing Board and Other NAEP Meetings

B.4. Task 4: Monthly Meetings with COR and NCES staff/contractors

B.5. Task 5M: Design Document Development

A. Framework Panel Selection

B. Framework Panel Meetings

C. Resources for Framework Panels

D. Technical Advisory Committee

E. Public Comment

F. Facilitation

B.6. Task 6M: Framework Panel Selection

B.7. Task 7M: Technical Advisory Committee (TAC) Meetings

B.8. Task 8M: Initial Resource Compilation

B.9. Task 9M: Framework, Specifications, and Contextual Variables Update Recommendations

A. Updated Framework

B. Updated Test and Item Specifications

C. Updated Contextual Variables

B.10. Task 10M: Process Report

B.11. NAEP Reading Framework Update Process: Tasks 5R, 6R, 7R, 8R, 9R, and 10R

B.12. Additional NAEP Framework Update Process: Tasks 5T, 6T, 7T, 8T, 9T, and 10T

C. SUBMISSION OF DELIVERABLES

C.1. Submission Protocols

C.2. Deliverables Schedule

D. REFERENCES

APPENDIX A: PRINCIPLES FROM GOVERNING BOARD FRAMEWORK DEVELOPMENT

POLICY

APPENDIX B: GUIDANCE FROM THE GOVERNING BOARD ITEM DEVELOPMENT

POLICY FOR NAEP ASSESSMENTS

APPENDIX C: ACHIEVEMENT LEVEL POLICY DEFINITIONS

UPDATE OF NAEP FRAMEWORKS FOR MATHEMATICS,

READING, AND OTHER SUBJECTS

A. Context of Procurement

A.1. Purpose

Each National Assessment of Educational Progress (NAEP) subject area assessment is guided by a framework that defines the scope of the domain to be measured by delineating the knowledge and skills to be tested at each grade and subject, the format of the assessment, and the achievement level definitions. Test and Item Specifications elaborate on frameworks as guidance for item development conducted by the National Center for Education Statistics (NCES) and the NAEP test development contractor(s). The National Assessment Governing Board (Governing Board) determines the content to be assessed and makes recommendations for contextual variables, which supports NCES development of the questionnaires administered to students, teachers, and schools to help the public understand the achievement results in each subject.

The purpose of this procurement is to develop recommendations to the Governing Board regarding updates to existing frameworks, specifications, and contextual variables for the NAEP Mathematics and Reading assessments, respectively. Based on Governing Board deliberations and actions, an additional framework update for one other subject area will be planned as a tentative requirement for the contract option period.

This requirement is pursuant to the legislative authority of the National Assessment, Public Law 107-279, Section 302 (5). The Governing Board’s Framework Development Policy articulates the Board’s role with principles and guidelines for framework update processes. Each update process shall produce assessment objectives and test specifications through a comprehensive, inclusive, and deliberative process that involves active participation of stakeholders. The process shall also take into account state and local curricula and assessments, widely accepted professional standards, exemplary research, international standards and assessments, and other pertinent factors and information. This Statement of Work (SOW) elaborates on the Governing Board Framework Development Policy, recently revised in March 2018 to detail procedures for updating NAEP Frameworks.

A.2. Overview of the Governing Board

The Governing Board is an independent, bipartisan organization that sets policy for NAEP, commonly known as The Nation’s Report Card. The 26 members of the Governing Board include governors, state legislators, state and local school officials, educators, researchers, business representatives, and members of the general public, who are appointed by the U.S.

Secretary of Education. The Governing Board’s policy roles align with the purpose of NAEP, which is to provide fair and accurate measurement of student academic achievement. Together, https://www.nagb.gov/content/nagb/assets/documents/policies/framework-development.pdf https://www.nagb.gov/content/nagb/assets/documents/policies/framework-development.pdf https://www.nagb.gov/ members of the Governing Board communicate NAEP results to the diverse audiences they represent. NCES implements and administers the assessment.

Since its creation by Congress in 1988, the Governing Board has overseen and set policy for NAEP by identifying subjects to be tested, determining and approving the assessment content, setting achievement levels for each assessment (i.e., Basic, Proficient, and Advanced), improving the reporting of results, and planning and executing initial releases of NAEP Report Cards.

NAEP makes objective information on student performance available to policymakers and the public for the nation, states, and the 27 local school districts in the NAEP Trial Urban District Assessment.

In accordance with congressional mandates and Board policies, the Governing Board has a direct role in setting the content of NAEP assessments, ensuring NAEP assessments are valid and reliable, and assuring that the frameworks that outline the content of NAEP assessments are based on widely accepted professional standards for educational assessment. The Governing Board ensures that framework development and updates adhere to the highest standards for test development, thereby supporting the validity of educational inferences made using NAEP data.

The Board also has final authority on the appropriateness of all assessment items.

The Governing Board has long engaged a broad spectrum of stakeholders in developing recommendations for the knowledge and skills NAEP should assess in various grades and subject areas. From this comprehensive and deliberative process, the Governing Board develops a framework to outline the content and format for each NAEP assessment at grades 4, 8, and 12.

The Governing Board’s Framework Development Policy supports stable measurement of student achievement, with periodic Board reviews of whether to conduct an update or develop a new framework for an existing NAEP assessment. Factors for these Board deliberations include new research on teaching and learning, changes in state and local curricula and assessments, and other factors.

In November 2016, the Governing Board adopted a Strategic Vision to expand NAEP’s impact by providing information about how students are doing in the most innovative and effective ways. The vision, intended to focus the Governing Board’s efforts through 2020, includes two broad goals: inform and innovate.

A.3. NAEP Frameworks and Assessments in Mathematics and Reading

Using an appropriate mix of content, items, and other state-of-the-art measurement practices, the Governing Board is committed to a technically sound, digitally based, and efficient assessment, designed as a valid and reliable measure of what students know and can do in mathematics and reading. To the extent possible, the assessment shall reflect the nation’s future needs and desirable levels of achievement.

https://www.nagb.gov/content/nagb/assets/documents/policies/framework-development.pdf https://www.nagb.gov/content/nagb/assets/documents/newsroom/press-releases/2016/nagb-strategic-vision.pdf

Current NAEP Mathematics Framework The current NAEP Mathematics Framework has been in place since the 2005 NAEP Mathematics Assessment. Empirical analyses have supported continued reporting of NAEP mathematics student achievement trends extending back to 1990 for grades 4 and 8 and extending back to 2005 for grade 12.

The Governing Board has conducted extensive work to strengthen 12th grade NAEP and study its use as an indicator of academic preparedness for education and job training opportunities after high school, including an in-depth 2006 review of the grade 12 objectives in the NAEP Mathematics Framework, with a parallel review of the NAEP Reading Frameworks.

In 2017, the Governing Board commissioned a review of state curricular standards in mathematics, which provides detailed comparative information for all 50 states, the District of Columbia, and the Department of Defense Education Activity. The Governing Board has also commissioned white papers from experts for framework review discussions. These reports and papers will be made available on the Governing Board website in Spring 2018. The Governing Board will provide these documents and other relevant commissioned papers to the contractor, during the Project Launch, Task 1. As appropriate, the contractor shall use these documents as resources for the framework update process.

Current NAEP Reading Framework The current NAEP Reading Framework has been in place since the 2009 NAEP Reading Assessment, and empirical analyses have supported continued reporting of student achievement trends extending back to 1992 for grades 4, 8, and 12.

In preparation for updating the NAEP Reading Framework, the Governing Board has commissioned white papers from content experts to inform deliberations and to shape the Governing Board’s charge to the panels that will be convened as part of the framework update process. Papers will be made available on the Governing Board website in Spring 2018, and shared with the contractor during the Project Launch.

NAEP Mathematics and Reading Assessments In accordance with the Every Students Succeeds Act (ESSA), the NAEP Mathematics and Reading Assessments are administered every 2 years at grades 4 and 8. At grade 12, the NAEP Mathematics and Reading Assessments are administered every 4 years. It is important to note that by law and by design NAEP does not produce results for individual students or schools.

In 2017, the NAEP Mathematics and Reading assessments were administered operationally for the first time as digital-based assessments (DBA) at grades 4 and 8. This transition acknowledges the new generation of classroom environments in which digital technology has become an increasing part of students’ learning. The 2017 NAEP digitally administered mathematics and reading assessments at grades 4 and 8 were designed to continue reporting trends in student performance dating back to the first administration in the early 1990s.

https://www.nagb.gov/naep-frameworks/mathematics.html https://www.nagb.gov/focus-areas/reports/preparedness-research/about/timeline.html https://www.nagb.gov/content/nagb/assets/documents/commission/researchandresources/Achieve-Mathematics-Report.pdf https://www.nagb.gov/content/nagb/assets/documents/commission/researchandresources/Achieve%20Reading%20Report.pdf https://www.nagb.gov/naep-frameworks/reading.html

The Board has kept NAEP Mathematics and Reading frameworks steady to support content stability and trend reporting during a time of sweeping changes in assessments across states. The 2017 assessment content was developed using the same frameworks used to develop the 2015 paper-based assessments and prior assessments since 2005 for mathematics and since 2009 for reading. In extensive reviews of all NAEP items, the Board has provided direct oversight and guidance relative to NAEP content, supporting alignment with the frameworks in the context of

DBA.

The 2017 assessments at grades 4 and 8 were comprised of previous paper-based assessment questions, adapted to fit a tablet screen and address the same content. The goal of adapting questions was to retain the same measurement targets as the original version of the question.

New questions aligned to the framework were also developed to take advantage of the digital delivery system.

The first digital-based NAEP Mathematics and Reading assessments at grade 12 will be conducted in 2019.

A.4. Role of the Board in Framework Updates

The Governing Board, through its Assessment Development Committee (ADC), monitors all framework updates as part of the Board’s ongoing policy oversight of NAEP. Board reviews of existing frameworks determine whether an update is needed to continue valid and reliable measurement of the content and cognitive processes reflected in evolving expectations of students.

At least once every 10 years, the Governing Board, through the ADC, reviews the relevance of assessments and their underlying frameworks. In the review, the ADC solicits input from experts to determine if changes are warranted, while making clear the potential risk of changing frameworks to trends and assessment of educational progress. The Board may decide based on stakeholder input that the framework does not require revision, or that the framework may require minor or major updates.

To initiate updates, the ADC prepares a recommendation for full Board approval. Minor updates include clarifications or corrections that do not affect the construct defined for the assessment. Major updates include the convening of framework panels to develop the recommended updates. Given the affordances of DBA, the Board anticipates that a major update process will be needed to more fully capitalize the digital assessment platform. Content shifts for each subject area may also need to be reflected. Hence, the offeror shall assume a major update process is to be conducted.

In initiating a framework update, the Governing Board seeks to maintain a balance to achieve stable reporting of student achievement trends. In considering when and how an adopted framework update will be implemented, the Board also considers the NAEP Assessment Schedule, cost and technical issues, and research and innovations to support possibilities for continuous trend reporting.

Each framework update project results in recommendations for Governing Board action in the form of three key documents: the framework; assessment and item specifications; and contextual variables relating to the subject being assessed.

Recommended changes to NAEP frameworks are made by ADC to the Board and are subject to full Board approval. At the conclusion of each framework update process, the Governing Board shall take final action on the recommended framework, specifications, and contextual variables.

A.5. Milestones for Framework Updates

In accordance with the Governing Board’s Framework Development Policy, the following milestones summarize the key areas of work in executing the framework updates in the context of the Board’s responsibilities:

• ADC Discussion with External Experts in the Subject Area

• ADC Recommendation for Updating Assessment

• Board Action on Charge

• Framework Contractor Selection

• Trend Scan & Resource Compilation

• Panel Meetings

• Full Board Review & Public Comment

• Framework Draft Finalized

• ADC Final Review of Framework

• Board Action

• Assessment Administered

The milestones carried out by the framework contractor are identified in the box above.

As a first step, the ADC conducts a framework review, where content experts are invited to a Committee session to provide reflections on the state of the discipline and the extent to which the relevant NAEP framework should be updated. Studies and additional outreach is pursued, as needed, to inform the ADC’s recommendation about the type of framework update that is required. Next, the ADC brings its recommendation to the full Board for approval. In the case of an anticipated framework update, the recommendation includes a charge to stakeholders who will serve on the panels convened to draft recommendations for the ADC’s consideration. After Board discussion of the ADC recommendation, the Board will take action on the charge.

Concurrently, Board staff will identify a contractor to execute the framework update process.

The framework contractor will launch the project by identifying individuals to serve on the framework panels and by compiling and developing resources to support the meetings of these stakeholders. A subset of these resources will include the Governing Board’s charge to the framework panels as well as documents used to inform the Board’s development of the charge.

The first meeting of stakeholders will be for the Visioning Panel to discuss the major issues to be addressed in the framework. A subset of the Visioning Panel will continue on as the

Development Panel to develop an updated framework. This panel will also develop the recommended updates to the Test and Item Specifications, as well as the Contextual Variables.

The ADC monitors the framework contractor’s work via regular project updates. A draft of the panels’ recommended framework will be shared for full Board review and public comment, as well as review by the Board’s Committee on Standards, Design and Methodology. This feedback will allow the Development Panel to address concerns and finalize the draft framework, specifications, and contextual variables for the ADC’s final review and Board action. The adopted framework, specifications, and contextual variables are given to NCES to begin assessment development, piloting, and finally administration of the operational assessment based on the new framework.

B. Scope of Work

The scope of work requires undertaking framework updates in reading and mathematics, and one other subject as an option.

Major Deliverables

In executing the framework update process, which includes the specifications and contextual variables, five (5) deliverables shall be executed respectively for each assessment, resulting in fifteen (15) major deliverables. The ten (10) deliverables addressing mathematics and reading are required for this contract, while the five (5) deliverables addressing one other additional subject area addresses tasks that may be exercised during the contract option period.

The major contract deliverables for each framework, i.e., reading, mathematics, and the other optional subject, shall include:

1. Design Document

2. Updated Assessment Framework

3. Test and Item Specifications

4. Contextual Variables

5. Process Report

The Design Document shall present a detailed plan for carrying out the framework update processes for the NAEP Mathematics and Reading frameworks, respectively. The Framework shall be written for a broad audience to reflect recommendations for the content to be assessed and the design of the assessment, while Test and Item Specifications shall elaborate the framework with additional details for NCES and NAEP contractor(s) to use in developing items and constructing the assessment. Recommendations on contextual variables relevant to achievement in the subject shall be presented in the Contextual Variables Document. The Process Report shall be developed to document how the framework update process was implemented. The contractor leads development of the Design Document and Process Report, while the convened framework panels lead content development of the updated Framework, Specifications, and Contextual Variables, with the contractor facilitating the panels’ work.

The recommendations from this project are anticipated to be used by the Governing Board to determine the content of the 2025 NAEP assessments in mathematics and reading.

The Framework Development Policy for NAEP assessments cites references which express widely accepted technical and professional standards for test development. These standards reflect the agreement of recognized experts in the field, as well as the policy positions of major professional and technical associations concerned with educational testing. The requirements in this work statement reflect procedural protocols on how this policy is to be implemented. To view a list of the guiding principles in the policy, see Appendix A.

Via the Governing Board’s Contracting Officer’s Representative (COR), the ADC provides direction to the contractor on how the work shall be informed and guided, and how the framework panels will develop recommendations. This guidance ensures compliance with the NAEP law, Governing Board policies, Department of Education, and government-wide regulations.

Overview of Tasks

For each subject area, there are similar task requirements.

Overarching Tasks include:

Task 1: Project Launch Task 2: Project Management Task 3: Governing Board and Other NAEP Meetings Task 4: Monthly Meetings with COR and NCES Staff/Contractors

Tasks for Each Framework Update include:

Task 5: Design Document Development Task 6: Framework Panel Selection Task 7: Technical Advisory Committee (TAC) Meetings Task 8: Initial Resource Compilation Task 9: Framework, Specifications, and Contextual Variables Update Recommendations Task 10: Process Report

The Design Document (Tasks 5) specifies the detailed plan, while Tasks 6 through 9 involves execution of the plan. Panel and TAC member selections (Tasks 6 and 7) and the initial resource compilation (Task 8) provide the platform for developing recommendations (Task 9), which is central to the Major Deliverables for each framework area. The Process Report (Task 10) documents these proceedings, while Tasks 1 through 4 provide ongoing support for the project.

To distinguish subject areas, Tasks 5M through 10M address mathematics, Tasks 5R through 10R address reading, and Tasks 5T through 10T address the subject area to be determined for the contract option period. Altogether, the project is comprised of 16 tasks. Six (6) additional tasks address the contract option period.

Within the scope of each task, there is flexibility in how the task can be implemented. In addition to demonstrating a strong understanding of the work required to accomplish the tasks and the important challenges to be addressed, the successful offeror shall use the flexibility available within each task to propose an implementation plan that is both feasible and technically sound.

The proposed implementation plan shall reflect an efficient sequence of project activities that supports high-quality project outcomes overall and relative to the Quality Assurance Surveillance Plan (QASP) (Attachment B), which details quality expectations.

B.1. Task 1: Project Launch

The project will begin with an in-person kickoff meeting with the Contracting Officer (CO), the COR, and key project staff from both the Governing Board and the contractor. This meeting will provide an opportunity to review the contract terms and conditions, identify project expectations and milestones, discuss quality control requirements, review the project timeline, and establish protocols for communication between points of contact.

The kickoff meeting shall take place in-person within 8 business days of contract award. A parallel kickoff meeting shall take place via videoconference for each additional framework area.

While the timeline for each framework update process is staggered, offerors may suggest the next kickoff meeting date no earlier than June 2019 for reading and no earlier than June 2020 for the additional subject area to be determined.

The Governing Board project staff will clarify key project requirements and provide guidance needed to inform the project plan. The contractor shall describe plans and timelines for organizing and structuring the project to encourage effective participation of various individuals and groups and ensure appropriate timing of activities. These plans and timelines may be refined as a consequence of the kickoff meeting and development of the Design Document.

Minutes summarizing the meeting and an updated Project Plan shall be submitted to the CO with a copy to the COR within five (5) business days following the kickoff meeting. The Project Plan shall include a project schedule, e.g., a Gantt chart, which shall present a complete chronological listing of the activities planned for the project with start and end dates, including dates for COR reviews. The Project Plan shall also describe plans for establishing the mathematics framework update panels (Task 6M) and the technical advisory committee (Task 7M), as well as plans for compiling mathematics framework resources (Task 8M). To the extent that these initial activities will support finalizing the Design Document (Task 5M), the Project Plan shall note these connections.

Task 1 deliverables include: kickoff meeting minutes and an updated Project Plan for each framework area.

B.2. Task 2: Project Management

In order to continuously monitor project work, regular and periodic meetings shall take place throughout the contract period of performance. These meetings will provide an opportunity to discuss task plans and scheduling, while addressing contractor needs and ensuring proactive measures are in place for successful attainment of project goals. The Design Document shall guide all areas of work, and timelines specified in the Project Plan and Design Document shall guide all phases of each framework update process. QASP requirements, including protocols and documentation, shall be addressed in all areas of Project Management.

Weekly Project Teleconferences Project team teleconferences will be held weekly with the COR to address ongoing work and to identify any challenges or issues. The frequency of these meetings may be adjusted as needed, depending on project activities. The updates, issues, and challenges discussed in these teleconferences shall be documented via a brief set of notes to capture decision points and open issues, submitted via email to the COR within 2 business days of each meeting.

Monthly Progress Reports Monthly progress reports shall be submitted on or before the 15th of each month with respect to the previous month’s activities. The date can be based on the contractor’s budget cycle, but it must be set for the same date each month. Each monthly report shall provide a status of the project by task area, challenges encountered, and work plans for the next reporting month.

This report shall be accompanied by the monthly invoice that shows the financial status of each task. A template for invoicing shall be provided at the first kickoff meeting.

Plans for Item/Data Security If the proposed Design Document calls for contractor access to current NAEP assessment item information as part of the framework update process, the use must maintain the validity of the framework update process. The Governing Board is sensitive to the fact that many groups and individuals may have access to various parts of the secure NAEP item pool. The contractor shall ensure that secure materials and data are securely controlled as well as confidentiality maintained by employees at all times. Security is a serious concern and it is the contractor’s responsibility to guarantee security for NAEP materials once in the contractor’s control. It is a felony to disclose confidential NAEP data or materials.

It is the responsibility of the contractor to develop effective data security plans and demonstrate how security procedures will be employed and monitored at all times for the duration of the contract. The contractor shall specifically address security procedures for (1) item distribution,

(2) item review, (3) data review, (4) storage of computers/tablets containing secure materials, (5) server security and avoidance of distributed denial of service (DDoS), if applicable, and (6) hotel and other staff security maintenance. The contractor’s data security plans shall be incorporated into the Design Document.

NCES requires that any person(s) who will be reviewing or using secure data or materials sign compliance forms during the course of this contract. These forms will be supplied by the

Governing Board COR for the contractor to complete. The contractor shall forward all compliance forms to the COR, who will forward a copy to NCES.

Logistical Plans The contractor shall be responsible for all plans and costs of logistical arrangements related to implementation of the project. Timelines for these activities shall be provided to the COR well in advance of their occurrence, and in accordance with the timelines specified in the approved Design Document. Logistical arrangements include:

Developing meeting agendas Preparing briefing materials and disseminating the materials as directed by the COR Arranging for suitable meeting space and equipment to support all required meetings for the framework update process Coordinating review and approval of negotiated meeting facility costs estimations and contracts. (The contractor shall submit preliminary costs projections for each in-person meeting using the template provided in the appendix to the Instructions to Offerors, Attachment C, which will facilitate the required approval from the U.S. Department of Education, Conference Policy Office.)

Making all travel, lodging, and group meal arrangements (at per diem rates) and paying for or reimbursing expenses for meeting participants (exclusive of outside observers, Board members, and Governing Board and NCES staff). Reimbursement of lodging and meals shall be in accordance with Federal Travel Regulations (see www.gsa.gov for current per diem amounts and ED guidance on meetings at http://www2.ed.gov/policy/fund/guid/gposbul/gposbul.html).

Reimbursing the cost of substitute teachers for teacher panelists Paying honorarium to panelists based on contract approved rates Preparing and making available minutes or transcripts of all meetings and conferences following each event, according to the timelines specified in the Design Document Providing updates at the weekly meetings with the COR via teleconference and/or webinar Setting up a secure project workspace to share materials Conducting all aspects of the meetings for the framework panel activities to include registration and onsite logistics Conducting TAC meetings Preparing and presenting reports for ADC and the Board Providing courtesy responses to all participants, including preparation of individual letters of commendation and appreciation under the COR’s signature

Task 2 deliverables include: Weekly teleconference minutes, monthly progress reports, plan for items/data security, and meetings logistics details as described above.

B.3. Task 3: Governing Board and Other NAEP Meetings

Two key members of the contractor’s project staff shall attend each quarterly Governing Board meeting during the course of the contract and make presentations regarding project activities.

http://www.gsa.gov/ http://www2.ed.gov/policy/fund/guid/gposbul/gposbul.html

The projected dates and locations of the Governing Board meetings are posted at https://www.nagb.gov/governing-board/quarterly-board-meetings.html

In addition to preparing and delivering a presentation to the ADC during each quarterly meeting (and to the full Board once for each framework update), the contractor shall submit a written update describing the project status and intended presentation to the COR one month in advance of the meeting start date. The written update will be included in the meeting materials sent to Governing Board members and other interested parties in advance of each quarterly meeting.

Also, for each subject area, the contractor shall plan for at least two one-day meetings of the ADC when the contractor’s key staff will brief ADC on the framework update status and the draft framework update documents.

Tasks 3 deliverables include: quarterly progress reports, and draft and final slide presentations.

B.4. Task 4: Monthly Meetings with COR and NCES staff/contractors

In order to carry out the tasks outlined in this SOW, the contractor may require access to NAEP data and other materials produced as part of NAEP operations. All these requests must be coordinated via the COR.

Operationally, the NAEP program is carried out by NCES via multiple contracts. Monthly meetings shall be scheduled for technical reviews of past and future activities and requirements for framework updating. These meetings will include the Project Director, the Governing Board COR, and a representative from NCES. Additional staff and contractors may be included at the discretion of the COR and NCES liaison. These meetings will be conducted via teleconference or web-based conferencing.

The contractor shall submit brief written minutes, which are intended to capture the major discussion and decision points, within two business days after each call.

Task 4 deliverables include: Monthly teleconference minutes.

B.5. Task 5M: Design Document Development

Offerors shall propose a Design Document that chronologically outlines each contract task and subtask activity for the project. To the extent that special analyses are proposed to guide the project the Design Document shall describe specifications for the overall methodology and the analysis. A Gantt chart or similar chart depicting project schedules and resources shall be included in the Design Document. The Design Document shall be submitted within 12 business days after the kickoff meeting.

The CO and COR shall review the Design Document and return comments to the contractor within 4 business days of its receipt from the contractor. An electronic file version of the final Design Document shall be delivered to the Board within one month of the contract award.

https://www.nagb.gov/governing-board/quarterly-board-meetings.html

Upon request by the COR, the contractor shall provide an updated version of the Design Document to reflect changes as needed.

The Design Document shall specify a set of performance metrics to support quality assurance monitoring, e.g., addressing inclusiveness, panelists’ satisfaction, etc. These metrics shall be reported on an ongoing basis or when the relevant project milestones are reached.

The Design Document shall detail:

(1) How framework panel candidates will be selected to represent broad curricular perspectives, expertise, and state assessment consortia affiliations, including representation for states unaffiliated with an assessment consortium;

(2) How framework panel members’ collaboration will be led and facilitated, including how members will use the current framework to determine appropriate updates;

(3) How supporting informational resources will be identified and developed, including but not limited to reports, descriptive statistics from public NAEP data tools, profiles on selected countries’ assessment systems, and video interviews;

(4) How resources and collected feedback will be used at different stages of the process;

(5) How framework panel members’ collaboration will be configured by grade-level content, while also allowing for contribution by experts who can summarize patterns in curricular content across multiple grade levels;

(6) How the important factors for each framework will be balanced;

(7) Which outcomes and issues are central for the Visioning Panel’s deliberation versus the Development Panel’s later deliberations;

(8) How technical experts shall be relied upon and convened to support the project, particularly with respect to psychometric issues;

(9) How public input will be solicited and used by the panels;

(10) How the details for the Specifications and the Contextual Variables will be codified in an ongoing manner as the Framework Update recommendations are developed;

(11) How efficient strategies will be used to adjudicate differences within panel discussions and across collected feedback in an inclusive manner, including visual representations where helpful; and

(12) How to represent the content of the recommended framework update in ways that are accessible and useful to the full array of NAEP stakeholders.

While noting contingency plans to maintain the project schedule, the Design Document shall address these and other salient issues within the context of six broad categories:

A. Framework Panel Selection B. Framework Panel Meetings C. Resources for Framework Panels D. Technical Advisory Committee E. Public Comment F. Facilitation

A. Framework Panel Selection The framework update process shall actively engage:

• Teachers

• Curriculum Specialists

• Content Experts

• Assessment Specialists

• State Administrators

• Local School Administrators

• Policymakers

• Business Representatives

• Parents

• Users of Assessment Data

• Researchers and Technical

Experts

• Members of the Public

These constituencies are hereafter referred to as the Stakeholders.

Comprised of up to 30 individuals representing the Stakeholders, the Framework Visioning Panel shall formulate guidelines for developing a recommended framework, based on the state of the field. Fifteen (15) members of the Visioning Panel will constitute the Framework Development Panel, the group charged with developing the drafts of the three project documents and engaging in the detailed deliberations to determine how to reflect the Visioning Panel guidelines in an updated framework.

At least 20 percent of the Visioning Panel shall have classroom teaching experience in the subject area under consideration. The Development Panel shall have a proportionally higher representation of content experts and educators. Individuals should be leaders within their respective stakeholder groups. All panelists must be well qualified by content knowledge and familiarity with the knowledge, skills, and abilities in the respective subject, while addressing all grade levels designated for the assessment.

Panels shall be representative in terms of region of the country, and educators shall be drawn from schools across the nation, including individuals who work with students from high-poverty and low-performing schools, as well as public and private schools. Panels shall represent state assessment consortia affiliations, including representation for states unaffiliated with an assessment consortium. The framework panels must also be representative in terms of gender, race/ethnicity, expertise, and viewpoints regarding the content of the assessment under development.

Nominations for the panels shall emerge from a broad solicitation process. The contractor shall send requests for nominations to disciplinary and other education organizations that represent the involved stakeholders. All candidates selected to serve on the framework panels will participate in the Visioning Panel at a minimum. In the candidate solicitation process, the contractor shall also confirm each individual’s availability to serve on the Development Panel, including schedule considerations to enable attendance at all meetings.

B. Framework Panel Meetings The contractor shall support 4 in-person panelist meetings, with the initial meeting devoted to the Visioning Panel and subsequent meetings devoted to the Development Panel. Each meeting shall be up to two days in length. Meetings shall be held in Washington, D.C., in order to have maximum participation by all parties, including Board staff, NCES, and NAEP contractors. The remainder of panel meetings shall be conducted virtually.

The update process for each framework shall begin with the Visioning Panel reviewing major issues in the content area, at the first in-person meeting. A major resource for this deliberation will be the Governing Board Charge and the Task 8M Initial Resource Compilation.

Each agenda for the in-person and virtual meetings shall be carefully constructed to facilitate discussion of major issues to be addressed in framework update deliberations. The first framework panel meeting shall result in guidelines to be used in drafting updates to the framework, specifications, and contextual variables. Subsequent meetings shall be used to develop revisions pursuant to the guidelines established during the Visioning Panel meeting and responsive to the Governing Board Charge.

C. Resources for Framework Panels The framework panels shall consider a wide variety of resources as deliberations proceed, and reference materials shall represent multiple views.

• The Current Framework. Framework updates shall begin with an extensive review of the current framework. To guide recommended updates, the contractor is responsible for developing an approach for the Visioning and Development Panels to engage with the current NAEP framework in the subject area. The panels shall also carefully consider whether recommended revisions are most appropriate to reflect in the Test and Item Specifications, as clarification and guidance to NCES for assessment development.

• Charge from the Board. The Governing Board will develop a charge for stakeholders who will serve on the panels convened to draft content recommendations for the ADC’s consideration. A Board member will attend the Visioning Panel meeting to deliver this charge in person. The Governing Board will undertake travel arrangements for the Board members who attend each Visioning Panel meeting.

• Initial Compilation of Resources. An initial compilation of resources shall serve as a springboard for the framework panels’ deliberations. These resources shall take into account widely accepted professional standards, exemplary research, standards and assessments internationally and in other countries, key reports having significant national and international interest, other assessment instruments in the content area, and other pertinent factors and information. The compilation shall offer a summary of relevant research, advantages and disadvantages of the latest developments, and trends in state standards and assessments for the content area. This compilation shall also summarize how stakeholders have used previous NAEP student achievement trends in the assessment area. The initial compilation may be comprised of previous or new video interview or speech excerpts, existing or newly developed white papers, profiles of assessment programs in other countries, existing reports, standards, and/or some form of public comment. The contractor shall develop this compilation as a comprehensive and organized presentation of issues, particularly in connection with new developments in the discipline and in assessment, since the last framework.

• State and Local Curricula and Assessments. To inform development of NAEP frameworks, the Governing Board commissioned a review of state curricular standards in Mathematics, which provides detailed comparative information for all 50 states, the District of Columbia, and the Department of Defense Education Activity. This report shall be shared with the contractor to be included in the resources for the first meeting of the framework panels. For Reading, other resources may address new developments and the state of practice as reflected in state and local curricula and assessments.

D. Technical Advisory Committee Technical experts specializing in educational measurement shall convene as a Technical Advisory Committee (TAC) and respond to technical issues raised during panel deliberations.

Given that the expertise represented on the TAC will not necessarily be connected to the subject area, retaining the same TAC members across the mathematics and reading framework updates is desirable. The TAC shall:

• Discuss and suggest resolutions to technical issues as they arise.

• Review recommended updates to the Test and Item Specifications, prior to submission to the Governing Board.

• Uphold the highest technical standards for development of the NAEP framework and specifications.

The group shall be convened by webinar only and, as needed, up to two TAC members may attend framework panel meetings.

E. Public Comment Public comment is an essential element of each framework process, and the draft framework update must be shared broadly through the framework development process. Public comment shall be sought from various segments of the population to reflect many different views, as well as those employed in the specific content area under consideration. There are various mechanisms through which these comments may be sought. The offeror shall propose public comment solicitation strategies, forums, and timing that accommodate project timelines and federal requirements.

The contractor is responsible for organizing and presenting the feedback received through public comment for the framework panels, e.g., grouping and summarizing the issues raised. The contractor shall also establish a structure for working with the Framework Development Panel to adjudicate issues raised in comments.

F. Facilitation In accordance with Governing Board policy, all NAEP framework development and update processes shall be conducted to be inclusive, comprehensive, and deliberative, while providing thorough documentation to support both validity and transparency. The NAEP framework update processes shall be informed by a broad, balanced, and inclusive set of factors. The framework shall reflect current curricula and instruction, research regarding cognitive development and instruction, and the nation’s future needs and desirable levels of achievement. This delicate balance between “what is” and “what should be” is at the core of the NAEP framework development process.

A primary focus of the contractor shall be to structure the opportunity for robust deliberation to achieve consensus, where possible. It is critical that the Stakeholders be given the maximum opportunity to participate in the decision-making process. The contractor is responsible for orchestrating the framework panels’ collaboration. Panel members are engaged to consider all viewpoints and debate all pertinent issues in formulating the content and design of a NAEP assessment, including findings from research.

The contractor is responsible for maintaining an environment that is open, balanced, and even-handed. The contractor shall identify procedures to clarify positions and views, roles and responsibilities of panel members and all project staff, as well as how the process will work toward reaching an understanding of the scope, content, and design of the updates for the framework. These procedures shall support: 1) defining and describing the issues identified; 2) making explicit any differences that cannot be reconciled; and 3) developing alternative assessment options with the pros and cons articulated and priorities suggested accordingly. As part of effectively facilitating the Panels’ discussion, the agenda and advance materials for all panel meetings shall be carefully developed to support discussions.

The Governing Board charge shall be used by the contractor to determine the scope of revisions to be conducted. In considering the factors specific to Mathematics and Reading, respectively, the contractor shall consider implications for the NAEP framework in terms of the evolution of the discipline, relevance to students’ postsecondary endeavors, advances in digital-based assessment, trends in standards and assessment nationally and in other countries, as well as student achievement trends generally and in terms of contextual factors. Part of this deliberation shall address how the content and design reflected in NAEP frameworks and assessments can best complement the information provided by international assessments.

The Visioning Panel shall discuss the potential risk of changing frameworks to trends and assessment of educational progress, and whether framework revision recommendations warrant the possible disruption. Considering the convergence in curricular emphasis that has occurred in the last 10 years, the Visioning Panel shall also discuss how information from NAEP can best complement student achievement information available from other sources. Ultimately, framework update recommendations shall address all U.S. jurisdictions, maintain NAEP as a curriculum neutral assessment, and reflect widely accepted professional standards.

In balancing the relative importance of various sources of information, framework panels shall consider direction from the Governing Board, the role and purpose of NAEP in informing the public about student achievement, the legislative parameters for NAEP, constraints of a large-scale assessment, technical assessment standards, issues of burden and cost-effectiveness in designing the assessment, and other factors unique to the content area.

Task 5 deliverables include: draft and final Design Document.

B.6. Task 6M: Framework Panel Selection

The contractor shall design and conduct a procedure to recruit individuals broadly representative of the Stakeholders to serve on the Visioning and Development Panels for the framework, in accordance with the Design Document.

A list of at least 50 prospective members will be prepared for Governing Board review. The list should be organized to show the extent to which each candidate reflects the expertise needed for the overall composition of the panels. After review and consultation with the Governing Board, the contractor shall formally invite candidates approved by the Governing Board COR to participate.

Task 6M deliverables include: List of Candidates for the Visioning and Development Panels.

B.7. Task 7M: Technical Advisory Committee (TAC) Meetings

Task 7M is to select and convene a group of five to eight technical experts as external advisers, in accordance with the Design Document. A list of at least 15 prospective members will be prepared for Governing Board review from which up to 8 members will be selected. The list should be organized to show the extent to which each candidate reflects the expertise needed for the overall composition of the committee. After review and consultation with the Governing Board, candidates approved by the Governing Board will be formally invited to participate by the contractor.

Task 7 deliverables include: Submission of Names/Candidates, agenda and materials for TAC webinars and webinar minutes.

B.8. Task 8M: Initial Resource Compilation

Each framework update process shall begin with…