Attachment C Assessment Peer Review.pdf

A State’s Guide to the U.S. Department of Education’s

Assessment Peer Review Process

U.S. Department of Education Office of Elementary and Secondary Education

Washington, D.C. 20202

September 24, 2018

Other than statutory and regulatory requirements included in the document, the contents of this guidance do not have the force and effect of law and are not meant to bind the public in any way.

This document is intended only to provide clarity to the public regarding existing requirements under the law or agency policies.

The U.S. Department of Education (Department) has determined that this document is a “significant guidance document” under the Office of Management and Budget’s Final Bulletin for Agency Good Guidance Practices, 72 Fed. Reg. 3432 (Jan. 25, 2007), available at https://www.federalregister.gov/documents/2007/01/25/E7-1066/final-bulletin-for-agency-good-guidance-practices. The purpose of this document is to provide States with information to assist them in meeting their obligations under Title I of the Elementary and Secondary Education Act of 1965 (ESEA), as amended. This document does not impose any requirements beyond those required under applicable law and regulations. It does not create or confer any rights for or on any person.

This document represents the Department’s current thinking on the critical elements and best practices for State development and implementation of assessment systems, and it supersedes the Department’s previous guidance, entitled U.S. Department of Education Peer Review of State Assessment Systems Non-Regulatory Guidance for States for Meeting Requirements of the Elementary and Secondary Education Act of 1965, as amended (revised September 25, 2015).

On June 27, 2018, this document was posted for public review and comment. The Department received 17 public comments concerning the content of this document. Many commenters proposed language to clarify requirements or to improve the consistency of language throughout the document, which the Department incorporated into this revised version of the guidance. Others proposed changes that were not consistent with the ESEA, such as permitting separate academic content standards for some groups of students, and those changes were not incorporated in this revised document.

If you are interested in commenting further on this document, please e-mail OESE@ed.gov or write to us at the following address: U.S. Department of Education, Office of Elementary and Secondary Education, 400 Maryland Avenue, SW, Washington, D.C. 20202.

Paperwork Burden Statement According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1810-0576.

https://www.federalregister.gov/documents/2007/01/25/E7-1066/final-bulletin-for-agency-good-guidance-practices https://www.federalregister.gov/documents/2007/01/25/E7-1066/final-bulletin-for-agency-good-guidance-practices mailto:OESE@ed.gov

Assessment Peer Review Process U.S. Department of Education

TABLE OF CONTENTS

I –ASSESSMENT PEER REVIEW PROCESS 4

A. INTRODUCTION 4

Purpose 5 Background 5 Changes in the assessment requirements in the ESEA as amended by the ESSA 6

B. THE ASSESSMENT PEER REVIEW PROCESS 9

Overview 9 Requirements for Assessment Peer Review When a State Makes a Change to a

Previously Peer-Reviewed State Assessment System 12

C. PREPARING AN ASSESSMENT PEER REVIEW SUBMISSION 15

Content and Organization of a State Submission for Assessment Peer Review 15 Coordination of Submissions for States that Administer the Same Assessments 21 How to Read the Critical Elements 22

D. TERMINOLOGY 23

Key Terminology 23 Additional Terminology 26

II – CRITICAL ELEMENTS FOR STATE ASSESSMENT PEER REVIEW 29

Map of the Critical Elements for Peer Review of State Assessments 29 Section 1: Statewide System of Standards and Assessments 30 Section 2: Assessment System Operations 36 Section 3: Technical Quality – Validity 47 Section 4: Technical Quality – Other 53 Section 5: Inclusion of All Students 60 Section 6: Achievement Standards and Reporting 65 Section 7: Locally Selected, Nationally Recognized 74

High School Academic Assessments

I – ASSESSMENT PEER REVIEW PROCESS

A. INTRODUCTION

Purpose

The purpose of the Department’s peer review of State assessment systems is to support States in meeting statutory and regulatory requirements under Title I of the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the Every Student Succeeds Act (ESSA),1 for implementing valid and reliable State assessment systems. Under sections 1111(a)(4) and 1111(b)(2)(B)(iii)-(iv) of the ESEA and 34 CFR § 200.2(b)(4) and (5) and (d), the Department has an obligation to conduct a peer review of the technical quality of State assessment systems implemented under section 1111(b)(2) of the ESEA. Assessment peer review is the process through which a State demonstrates the technical soundness of its assessment system. A State’s success with its assessment peer review begins and hinges on the steps the State takes to develop and implement a technically sound State assessment system.

From 2005 through 2012, the Department conducted a peer review process for evaluating State assessment systems. In December 2012, in light of transitions in many States to new assessments aligned to college- and career-ready academic content standards in reading/language arts and mathematics, and advancements in the field of assessments, the Department suspended peer review of State assessment systems to review and revise the process based on current best practices in the field and lessons learned over the past decade. The Department resumed this process in September 2015. Subsequently, the ESSA was passed. While most of the requirements for State assessment systems were unchanged, there are a few new components (which are described below). This document supersedes the guidance released in September 2015 and is consistent with the new components of the ESSA. Throughout this document, we reference ESEA requirements. In some cases, the ESEA requirements we reference are found specifically in the ESEA’s implementing regulations at 34 CFR Part 200 and, where appropriate, we provide a citation to the applicable ESEA implementing regulations.

This document is intended to support States in developing and administering assessment systems that (1) provide valid and reliable information on how well students are achieving a State’s challenging academic content and achievement standards to prepare all students for success in college and careers in the 21st century; and (2) provide valid and reliable information about the English proficiency of all English learners (ELs) in the State. Additionally, it is intended to help States prepare for peer review of their assessment systems and help guide peer reviewers who will evaluate the evidence submitted by States.

The document includes: (1) information about the assessment peer review process both for academic content assessments in mathematics, reading/language arts, and science, and for English language proficiency (ELP) assessments; (2) instructions for preparing evidence for

1 References and statutory citations in this document are to the ESEA as amended by the ESSA unless otherwise indicated.

submission; and (3) examples of evidence for addressing each critical element for each type of required assessment.

Background

A key purpose of Title I of the ESEA is to promote educational excellence and equity so that all students master the knowledge and skills, by the time they graduate high school, that they need in order to be successful in college and the workforce. A State accomplishes this, in part, by adopting challenging academic content standards that define what the State expects all students to know and be able to do, developing and administering assessments aligned to those standards and adopting academic achievement standards aligned to the academic content standards to define levels of student achievement on the assessments.

Specifically, under Title I of the ESEA, each State is responsible for implementing a State assessment system that is coherent and consistent within the State. The ESEA requires a State to develop and implement (1) challenging academic content and achievement standards in at least mathematics, reading/language arts, and science, and to apply the same academic standards to all public schools and public school students in the State (ESEA section 1111(b)(1)(A)-(D); 34 CFR § 200.1(a)); and (2) ELP standards that (1) are derived from the four recognized domains of speaking, listening, reading, and writing; (2) address the different proficiency levels of ELs; and

(3) are aligned with the challenging State academic standards (ESEA section 1111(b)(1)(F)).

The ESEA also requires a State to annually administer State-determined academic assessments in reading/language arts and mathematics in each of grades 3 through 8 and once in high school, and to annually administer State-determined academic assessments in science at least once in each of three grade spans (3-5, 6-9 and 10-12) (ESEA section 1111(b)(2)(B)(v); 34 CFR §§ 200.2(a)(1), 200.5(a)).

The ESEA requires that the academic content assessments be aligned with the State’s academic content standards and address the depth and breadth of those standards; be valid, reliable, and of adequate technical quality for the purposes for which they are used; express student results in terms of the State’s academic achievement standards; and provide coherent information about student achievement (ESEA section 1111(b)(2)(B)(ii)-(iv); 34 CFR § 200.2(b)(2)-(5)). In addition, the ESEA requires that the same academic assessments be used to measure the achievement of all students in the State, including ELs and students with disabilities2, with the exception allowed under ESEA section 1111(b)(2)(D) for students with the most significant cognitive disabilities who may take an alternate assessment aligned with alternate academic achievement standards (AA-AAAS) permitted under section 1111(b)(1)(E) of the ESEA (ESEA section 1111(b)(2)(B)(i), (vii), (b)(2)(D); 34 CFR §§ 200.2(b)(1), 200.6).

2 The ESEA and Title I, Part A regulations use both “students with disabilities” and “children with disabilities.”

Section 602(3) of the Individuals with Disabilities Education Act (IDEA), 20 U.S.C §1401(3), defines the term “child with a disability,” and that definition is also included in ESEA section 8101(4). However, because a State’s assessment system must include children with disabilities under IDEA, as well as students who are individuals with disabilities as defined in Section 504 of the Rehabilitation Act of 1973 and Title II of the Americans with Disabilities Act, we generally refer to students with disabilities throughout this document.

The ESEA and its implementing regulations also require a State to ensure that its local education agencies (LEAs) provide an annual ELP assessment of all ELs in grades K-12 in schools served by the State (ESEA section 1111(b)(2)(G); 34 CFR § 200.6(h)). Specifically, the ESEA requires a State to develop a uniform statewide ELP assessment to measure the English language proficiency of all ELs in the State, including ELs with disabilities, with an exception for ELs who are students with the most significant cognitive disabilities who may take an alternate ELP assessment (AELPA) if they cannot participate in the regular ELP assessment even with accommodations (ESEA section 1111(b)(2)(G); 34 CFR § 200.6(h)(1), (5)). The ESEA and its implementing regulations require that a State’s ELP assessments, including the AELPA, be aligned with the State’s ELP standards, provide valid and reliable measures of the State’s ELP standards, and be of adequate technical quality (ESEA section 1111(b)(2)(G); 34 CFR §§ 200.2(b)(2), (b)(4), (b)(5), 200.6(h)(2)).

Within the parameters noted above, each State has the flexibility and the responsibility to design its assessment system. This responsibility includes the adoption of specific academic content standards and ELP standards and selection of specific assessments that assess those standards. A State is also permitted to develop alternate academic achievement standards for students with the most significant cognitive disabilities (ESEA section 1111(b)(1)(E); 34 CFR § 200.1(d)) and to administer an AA-AAAS for academic content assessments (ESEA section 1111(b)(2)(D); 34 CFR § 200.6(c)). Further, a State has the discretion to include in its assessment system components beyond the requirements of the ESEA, which are not subject to assessment peer review. For example, some States administer assessments in additional content areas (e.g., social studies and art). A State also may include additional measures in its State assessment system, such as formative and interim assessments, which would not be subject to assessment peer review.

Changes in the assessment requirements in the ESEA as amended by the ESSA

This document reflects changes made to the ESEA standards and assessment requirements by the ESSA. For the most part, the assessment provisions under the ESEA as amended by the ESSA remain similar to the prior assessment provisions under the ESEA as amended by the No Child Left Behind Act of 2001. However, the ESSA adds several new provisions. These include, in part:

• The requirement that a State demonstrate that its challenging academic standards are aligned with entrance requirements for credit-bearing coursework in the system of public higher education in the State and relevant State career and technical education standards (ESEA section 1111(b)(1)(D)(i));

• The requirement that a State conduct meaningful and timely consultation with State leaders, including the Governor, members of the State legislature, State board of education, local educational agencies (including those located in rural areas), representatives of Indian tribes located in the State, teachers, principals, other school leaders, charter school leaders (if applicable), specialized instructional support personnel, paraprofessionals, administrators, other staff, and parents when developing the challenging academic standards and assessment systems and the English language proficiency (ELP) standards and assessment systems (ESEA section 1111(a)(1)(A));

• The option to exempt 8th grade students, who take the mathematics course associated with the high school mathematics assessment a State uses for Federal accountability purposes, from the 8th grade mathematics assessment the State typically administers (ESEA section 1111(b)(2)(C); 34 CFR § 200.5(b));

• The option to provide native language assessments for Native American and Alaska Native populations (34 CFR § 200.6(j));

• The requirement that a State ensure that accommodations for all required assessments do not deny students with disabilities or ELs the opportunity to participate in the assessment and any benefits from participation in the assessment (ESEA section 1111(b)(2)(B)(vii);

34 CFR § 200.6(b)(3), (f)(2)(i)); and

• The option for a State to allow LEAs to select and administer a nationally recognized high school assessment in lieu of the statewide high school assessment in a given subject, provided the assessment meets certain statutory and regulatory requirements (ESEA section 1111(b)(2)(H); 34 CFR § 200.3).

The ESEA as amended by the ESSA and its implementing regulations strengthen the requirements for assessing students with disabilities. For example, the use of appropriate accommodations may not deny students with disabilities the opportunity to participate in the assessment or any of the benefits afforded to any other students who are not students with disabilities (ESEA section 1111(b)(2)(B)(vii); 34 CFR § 200.6(b)(1), (3)). Additionally, the ESEA requires a State to reinforce the accessibility of assessments through appropriate accommodations for students with disabilities, and, to the extent practicable, incorporate principles of universal design for learning (UDL) for all required assessments (ESEA section 1111(b)(2)(B)(xiii); 34 CFR § 200.2(b)(2)(ii)). Also, the ESEA prohibits a State from precluding students with the most significant cognitive disabilities who take an AA-AAAS in an academic content area from attempting to complete requirements for a regular high school diploma, as defined in ESEA section 8101(43) (ESEA section 1111(b)(2)(D)(i)(VII); 34 CFR § 200.6(d)(4)). Moreover, if a State administers an AA-AAAS in an academic content area for students with the most significant cognitive disabilities, the ESEA requires that the AA-AAAS be aligned with the State’s academic content standards for the grade in which the student is enrolled and yield results relative to the State’s alternate academic achievement standards (ESEA section 1111(b)(1)(E)(i)(I), (b)(2)(D)(i); 34 CFR § 200.6(a)(2)(ii)(B)). Additionally, the ESEA requires that a State’s alternate academic achievement standards reflect professional judgment as to the highest possible standards achievable by such students (ESEA section 1111(b)(1)(E)(i)(III); 34 CFR § 200.1(d)(3)), and be designed to ensure that a student who meets those standards is on track to pursue postsecondary education or competitive integrated employment consistent with ESEA section 1111(b)(1)(E)(i)(V) and 34 CFR § 200.2(b)(3)(ii)(B)(2)).

The ESEA as amended by the ESSA and its implementing regulations also strengthen the requirements for assessing ELs. For example, appropriate accommodations for ELs may not deny them the opportunity to participate in the assessments or any of the benefits afforded to any other students who are not ELs (ESEA section 1111(b)(2)(B)(vii); 34 CFR § 200.6(f)(1)(i), (2)(i)). The ESEA specifies that, to the extent practicable, academic content assessments (mathematics, reading/language arts, and science) must be administered in the language and form most likely to yield accurate and reliable information on what ELs know and can do in order to determine the students’ mastery of skills in academic content areas until the students have achieved English proficiency (ESEA section 1111(b)(2)(B)(vii)(III); 34 CFR § 200.6(f)(1)(ii)).

The ESEA further requires a State to make every effort to develop assessments in languages other than English that are present to a significant extent in the participating student population (ESEA section 1111(b)(2)(F); 34 CFR § 200.6(f)(2), (4)).

Under the ESEA, requirements for ELP standards and assessments are covered under Title I rather than Title III. As a result, they are subject to many of the same requirements that govern a State’s academic content assessments. They are subject to peer review by the Department and must meet all applicable requirements (ESEA section 1111(a)(4); 34 CFR § 200.2(d)). Each State must submit evidence for peer review that its ELP assessment provides valid and reliable results, is aligned with the State’s ELP standards, and is consistent with nationally recognized professional and technical testing standards (34 CFR § 200.6(h)(2)). The ELP assessments that are subject to peer review requirements are covered in section 1111(b)(2)(G) of the ESEA and 34 CFR § 200.6(h). In other words, a State’s annual ELP assessment and the AELPA are subject to peer review.

ELs with disabilities must be provided accommodations on the ELP assessment (e.g., accessible formatting) so that these students are afforded the opportunity to demonstrate what they know and can do (34 CFR § 200.6(h)(4)). A State must develop an AELPA for ELs who are students with the most significant cognitive disabilities who cannot participate in the regular State ELP assessment, even with appropriate accommodations (34 CFR § 200.6(h)(5)). A State may choose to implement an AELPA aligned with the grade-level/grade-band achievement standards, or it may choose to implement an AELPA aligned with alternate ELP achievement standards that the State has the option to develop.

Consistent with 34 CFR § 200.2(d), the following assessments must be submitted for peer review under ESEA section 1111(a)(4):

• General mathematics and reading/language arts for grades 3-8 and high school (ESEA section 1111(b)(2));

• General science administered at least once in each of these grade spans: 3-5, 6-9, and 10- 12 (ESEA section 1111(b)(2));

• AA-AAAS in mathematics, reading/language arts, and science for students with the most significant cognitive disabilities for the grades described above (ESEA section 1111(b)(2)(D));

• ELP assessments for grades K-12 (ESEA section 1111(b)(2)(G));

• AELPA for ELs who are students with the most significant cognitive disabilities in grades K-12 (34 CFR § 200.6(h)(5));

• If applicable, locally selected, nationally recognized high school academic assessments

(ESEA section 1111(b)(2)(H));

• If applicable, assessments used for the 8th grade mathematics exception (ESEA section

1111(b)(2)(C));

• If applicable, content assessments in a student’s native language for ELs (ESEA section

1111(b)(2)(F)); and

• If applicable, content assessments in a Native American language (34 CFR § 200.6(j)).

B. THE ASSESSMENT PEER REVIEW PROCESS

Overview

The Department’s review of State assessment systems is an evidence-based, peer review process for which each State submits evidence to demonstrate that its assessment system meets a set of established criteria, called critical elements.

Critical Elements. The critical elements in Part II of this document represent the ESEA statutory and regulatory requirements that State assessment systems must meet. This guide divides them into critical elements covered under seven sections: (1) Statewide System of Standards and Assessments, (2) Assessment System Operations, (3) Technical Quality – Validity, (4) Technical Quality – Other, (5) Inclusion of All Students, (6) Academic Achievement Standards and Reporting, and (7) Locally Selected, Nationally Recognized High School Academic Assessments (if applicable). The map of critical elements included in Part II provides an overview of the seven sections and the critical elements within each section.

Evidence-Based Review. The ESEA requires that each State submit evidence for its assessment system that addresses each critical element. Consistent with ESEA sections 1111(b)(1)(A) and 1111(b)(1)(G)(i), the Department does not require a State to submit its academic content standards or its ELP standards as part of the peer review. In addition, the Department does not require a State to include or delete any specific content in its academic content or ELP standards or require a State to use specific assessment instruments or items (ESEA section 1111(e)(1)(B)(ii), (iii)(II)). The assessment peer review focuses on the processes for assessment development employed by a State and the relevant documentation and evidence that confirm the technical quality of the State’s assessment system.

Scheduling. The Department will notify States of the schedule for upcoming assessment peer reviews. Specifically, the Department will notify States regarding the submission of evidence for the ELP assessment within six months of the publication of this document. The Department will also notify States regarding the submission of evidence for the AELPA within eighteen months of the publication of this document. A State implementing new assessments or a State that has made significant changes to previously reviewed assessments should submit its assessment system for assessment peer review approximately six months after the first operational administration of its new or significantly changed assessments, or the next available scheduled peer review and prior to the second administration of the new or revised assessments (see also Exhibit 1). If a State wishes to permit an LEA to administer a nationally recognized high school academic assessment in place of the State’s high school academic content assessment, then it must present evidence for peer review for each nationally recognized assessment demonstrating that the assessment meets the requirements in ESEA section 1111(b)(2)(H) and 34 CFR § 200.3(b) prior to allowing an LEA to administer such assessment.

Expert Peer Reviewers. To determine if a State has met ESEA standards and assessment requirements, the Department uses a peer review process involving experts in the field of educational standards and assessments. Based on the evidence a State submits, the reviewers evaluate the State’s assessment system against ESEA requirements and provide their evaluations to the Assistant Secretary for Elementary and Secondary Education (Assistant Secretary).

Selection of Assessment Peer Reviewers. Assessment peer reviewers are individuals who have strong technical expertise necessary for reviewing State assessment systems and practical experiences in applying that expertise to the operation of State assessment systems. Each assessment peer reviewer is selected by the Department based on the individual’s experience and expertise, with an emphasis on knowledge of technical aspects of large-scale assessments, experience with the operation of State assessment systems, and relevant specialized expertise (such as in the relevant content areas, with technology-based assessments or with AA-AAAS).

Assessment peer reviewers are selected from lists of individuals who have previously served as assessment peer reviewers for the Department or as reviewers and consultants for other assessment-related activities for the Department; recommendations by Department staff; and recommendations from the field. Assessment peer reviewers are screened to ensure they do not have a conflict of interest.

Role of Assessment Peer Reviewers. Using the critical elements in this document as a framework, assessment peer reviewers apply their professional judgment and relevant professional experiences to evaluate the degree to which evidence provided about a State’s assessment system addresses each of the critical elements. Their evaluations inform the decision by the Assistant Secretary as to whether the State has sufficiently demonstrated that its assessment system addresses each critical element.

Assessment peer reviewers work in teams to review evidence submitted by a State. The Department selects peer reviewers and assembles the peer review team, aiming to balance peer reviewer expertise and experience in general. Where applicable, the Department selects peer reviewers who have the expertise and experience needed for the particular assessments a State has submitted for assessment peer review (e.g., technology-based assessments, ELP assessments, the AELPA, or AA-AAAS). The final configuration of an assessment peer review team, typically three reviewers, is determined by the Department. To protect the integrity of the assessment peer review process, the identity of the assessment peer review team for a specific State will remain anonymous.

During the peer review, the first step is for each of the assessment peer reviewers to independently review the materials submitted by a State and record their evaluation on an assessment peer review notes template. Next, at an assessment peer review team meeting, the assessment peer reviewers discuss the State’s submitted evidence with respect to each critical element, allowing the peer reviewers to strengthen their understanding of the evidence and to inform their individual evaluations. If there are questions or additional evidence appears to be needed, the Department may facilitate a conversation or communication between the peer reviewers and the State to clarify the State’s evidence. Based upon each peer reviewer’s review of the State’s documentation, he or she will note where additional evidence or changes in a State assessment system may be necessary for the State to meet the ESEA requirements; assessment peer reviewers may also present suggestions for addressing the outstanding requirements or highlight best practices in their notes. Although the assessment peer reviewers on a team are expected to generate one set of assessment peer review notes that reflect their review and evaluation of the State’s evidence, they are not expected to reach consensus.

The assessment peer review notes serve two purposes. First, they serve as the record of the assessment peer review team’s evaluation of a State’s evidence for the Assistant Secretary.

Second, soon after the assessment peer review ends, the assessment peer review notes are provided to the State as technical assistance and preliminary feedback prior to a formal decision regarding the outcome of the review. The assessment peer review notes, however, do not necessarily identify the final set of additional evidence, if any, which a State may need to submit to demonstrate that its assessment system meets all of the critical elements.

Training and Support for Assessment Peer Reviewers. Assessment peer reviewers will be trained in interactive training sessions with other assessment peer reviewers. Training will be based primarily on this document. Prior to the assessment peer review team meeting, each member of an assessment peer review team will be sent the following items: materials submitted to the Department by a State; this document; and an assessment peer review notes template. This allows for a thorough and independent review of the evidence before the assessment peer review team meeting.

Role of Department Staff. For some critical elements that serve as compliance checks or checks on processes, Department staff will review the evidence submitted by a State, as shown in the map of the critical elements. Department staff will determine either that the requirement has been adequately addressed or forward the evidence to the assessment peer review team for further review. In addition, one or more Department staff will be assigned as a liaison to each State participating in an assessment peer review and to the assessment peer review team for that State throughout the assessment peer review process. The Department liaison will serve as a contact and support for the State and for the assessment peer review team.

Outcomes of Assessment Peer Review. Following a review of evidence through assessment peer review, a State first will receive feedback in the form of assessment peer review notes.

Assessment peer review notes do not constitute a formal decision by the Assistant Secretary.

Instead, they provide initial feedback regarding the assessment peer reviewers’ evaluation and recommendations based on the evidence submitted by the State. A State should consider such feedback as technical assistance and not as formal feedback or direction to make changes to its assessment system.

The Assistant Secretary will provide formal feedback to a State regarding whether the State has provided sufficient evidence to demonstrate that its assessment system meets all applicable ESEA statutory and regulatory requirements following the assessment peer review. If a State has not provided sufficient evidence, the Assistant Secretary will identify the additional evidence necessary to address the critical elements. The Department will work with the State to develop a plan and timeline for submitting the additional evidence for assessment peer review.

Assessment Peer Review and Civil Rights Compliance. The assessment peer review will not evaluate or provide recommendations regarding whether a State’s assessment system complies with Federal civil rights laws that prohibit discrimination based on race, color, national origin, sex, disability, and age. These laws include Title VI of the Civil Rights Act of 1964, Title IX of the Education Amendments of 1972, Section 504 of the Rehabilitation Act of 1973, Title II of the Americans with Disabilities Act of 1990, the Age Discrimination Act of 1975, and applicable requirements in the Individuals with Disabilities Education Act (IDEA).

ESEA Requirements for Assessment Peer Review When a State Makes a Change to a Previously Peer Reviewed State Assessment System

In general, a significant change to a State assessment system is one that changes the interpretation of test scores. If a State makes a significant change to a component of its State assessment system that the State has previously submitted for assessment peer review, such as the adoption of new standards or assessments, ESEA section 1111(a)(6) requires that the State submit evidence related to the affected component for assessment peer review. A State should submit evidence for assessment peer review before, or as soon as reasonable following, the first operational administration of its assessment system with the change, and no later than prior to the second administration of its assessment system with the change.

To provide clarity about the implications of changes to State assessment systems, this document outlines three categories of changes (see Exhibit 1 for further details):

• Significant change is a change that clearly changes the interpretation of test scores and requires a new assessment peer review.

• Adjustment is a change between the extremes of significant and inconsequential and may or may not require a new assessment peer review.

• Inconsequential change is a minor change that does not impact the interpretation of test scores or substantially change other key aspects of a State’s assessment system. For inconsequential changes, a new assessment peer review is not required.

A State making a change to its assessment system is encouraged to discuss the implications of the change for assessment peer review with its technical advisory committee (TAC). A State making a significant change or adjustment to its assessment system also is encouraged to contact the Department early in the planning process to determine if the adjustment is significant and to develop an appropriate timeline for the State to submit evidence related to significant changes for assessment peer review. Exhibit 1 provides examples of the three categories of changes.

However, as noted in the exhibit, the changes listed in Exhibit 1 are merely illustrative and do not constitute an exhaustive list of the changes that fall within each category.

In addition, the ESEA and its implementing regulations require that a State that wishes to permit an LEA to administer a nationally recognized high school academic assessment in place of the State’s high school assessment present evidence for peer review for each nationally recognized assessment demonstrating that the assessment meets ESEA requirements prior to allowing an LEA to administer such a test (ESEA section 1111(b)(2)(H)(iii); 34 CFR § 200.3(b)(2)(ii)).

The Department recognizes that some peer review of nationally recognized high school academic assessments may have occurred prior to a State selecting such a test for this flexibility. A State may inquire through its Office of State Support contacts as to the current peer review status of any nationally recognized high school assessment and whether the previous peer review impacts the State’s submission for peer review. A State may be able to build its peer review submission of a nationally recognized high school academic assessment upon evidence previously reviewed and approved through Department peer review.

Exhibit 1: Categories of Changes and Non-Exhaustive Examples of Assessment Peer Review Submission Requirements when a State Makes a Change to a Previously Peer Reviewed State Assessment System

New Assessments ESEA requires that submissions address sections 2, 3, 4, 5 and 6 of the appropriate critical elements Significant Always significant.

Adjustment Not applicable.

Inconsequential Not applicable.

Development of a Technology-Based Version of an Assessment ESEA requires that submissions address critical elements 2.1–2.3, sections 3 and 4 of the appropriate critical elements Significant • Assessment delivery is changed from entirely paper-and-pencil to entirely computer-based.

• The new computer-based version of the assessment includes technology-enhanced items that are not available in the simultaneously administered paper-and-pencil version.

Adjustment • Assessment delivery is changed from a mix of paper-and-pencil and computer-based assessments to an entirely technology-based administration using a range of devices.

Inconsequential • Not applicable.

Development of a Native Language Version of an Assessment ESEA requires that submissions address critical elements 2.1 - 2.3, sections 3 and 4 of the critical elements Significant Always significant.

Adjustment Not applicable.

Inconsequential Not applicable.

Changes to an Existing Test Design ESEA requires that submissions address critical elements germane to the change in test design, likely from sections 2, 3, 4 and 6 of the appropriate critical elements Significant • State’s approved reading/language arts assessment included multiple constructed-response items for each academic content standard and the State replaces these items with multiple-choice items.

• Change in the assessment purpose, use, design, or content (e.g., due to legislative or regulatory change or State policy).

Adjustment • State changes the number of items on its assessments (by more than a few items).

• Scoring conducted across an assessment consortium as a whole is changed to scoring conducted individually by consortium member States.

Inconsequential • State changes from on-site to virtual training for its scorers of extended-response items.

Changes to Test Administration ESEA requires that submissions address critical elements germane to the change in test design, likely from sections 2 and 5 of the appropriate critical elements Significant • State shifts scoring of its alternate assessments aligned with alternate achievement standards from centralized third-party scoring to scoring by the test administrator.

Adjustment • State shifts from desktop computer-based test administration to Statewide test administration using a range of devices.

• State participates in an assessment consortium and shifts certain practices from consortium-level to State-level operation.

Inconsequential • State combines its trainings for test security and test administration.

Assessments Based on New Academic Achievement Standards or New ELP Achievement Standards ESEA requires that submissions address section 6 of the appropriate critical elements Significant • Comprehensive revision of State’s academic or ELP achievement standards (e.g., performance-level descriptors, cut-scores).

Adjustment • Smoothing of cut-score across grades after multiple administrations.

Inconsequential • Implementation of planned adjustment to State’s achievement standards that were reviewed and approved during assessment peer review.

Assessments Based on New or Revised Academic Content Standards or ELP Standards ESEA requires that submissions address sections 1, 2, 3, 4, 5 and 6 of the appropriate critical elements Significant • Adoption of completely new academic content standards or ELP standards, or comprehensive revision of the State’s academic content standards or ELP standards to which assessments must be aligned.

Adjustment • State makes minor changes to its academic content standards or ELP standards to which assessments must be aligned by moving a few benchmarks within or between standards, but assessment blueprints are not affected and test results are comparable after equating.

Inconsequential • State makes minor formatting changes, non-substantive word edits or corrections to typos in its academic content standards or ELP standards, which do not change the interpretation of test scores.

Locally Selected, Nationally Recognized High School Academic Assessments Used in Place of State High School Assessments ESEA requires that submissions address ALL sections 1, 2, 3, 4, 5, 6 and 7 of the critical elements Significant • A State that allows LEAs to use a nationally recognized high school academic assessment in place of the State’s high school assessment must always submit evidence for peer review demonstrating that the nationally recognized assessment meets ESEA requirements prior to its use.

Adjustment Not applicable.

Inconsequential Not applicable.

C. PREPARING AN ASSESSMENT PEER REVIEW SUBMISSION

A State should use this document to prepare its submission for assessment peer review. The State Assessment Peer Review Submission Cover Sheet and Index Template (see most current version posted online at https://www2.ed.gov/admins/lead/account/saa.html) includes a checklist a State can use to prepare an assessment peer review submission.

Content and Organization of a State Submission for Assessment Peer Review

Submission by State. A State should send its submission to the Department according to the schedule for assessment peer reviews announced by the Department. The Department will notify States regarding the submission of evidence for the ELP assessment within six months of the publication of this document. The Department will also notify States regarding the submission of evidence for the AELPA within eighteen months of the publication of this document. For all statewide assessments, a State should submit its assessment systems for assessment peer review approximately six months after the first operational administration of new or significantly changed assessments. The ESEA requires that a State that wishes to permit an LEA to administer a nationally recognized high school academic assessment in place of the State’s high school assessment present evidence for peer review for each nationally recognized test demonstrating that the assessment meets ESEA requirements prior to allowing an LEA to administer such a test (ESEA section 1111(b)(2)(H)(iii); 34 CFR § 200.3(b)(2)(ii)). The Department encourages each State to plan for preparing its peer review submission according to this timeline. A State is expected to submit evidence regarding its State assessment system approximately four weeks prior to the scheduled assessment peer review date for the State.

For assessments administered by multiple States, the Department will conduct a single review of the evidence that applies to all States implementing the same assessments. This approach both promotes consistency in the review of such assessments and reduces burden on States in preparing for assessment peer review. This review of common evidence will also include nationally recognized high school tests, if States collaborate in submitting evidence for these tests.

https://www2.ed.gov/admins/lead/account/saa.html

What to Include in a Submission. A State’s submission includes the following parts:

1) State Assessment Peer Review Submission Cover Sheet;

2) State Assessment Peer Review Submission Index; and

3) Evidence to address each critical element that applies.

State Assessment Peer Review Submission Cover Sheet and Index Template. The State Assessment Peer Review Submission Cover Sheet and Index Template (see most current version posted online at https://www2.ed.gov/admins/lead/account/saa.html) includes the cover sheet that a State must submit with each submission of evidence for peer review. It also includes an index template aligned to the critical elements in seven sections as shown in the map of the critical elements. Each State should use this State Assessment Peer Review Submission Index Template to prepare an index to its submission to accompany the evidence that the State submits.

A State’s prepared index should outline the evidence for each critical element with the following:

1) Identification of the critical element;

2) List of the evidence submitted to address the critical element (e.g., relevant document(s) and page number(s);

3) Indication of where evidence that addresses the applicable critical element can be found in the State’s submission; and

4) As applicable, a brief narrative of how the evidence addresses the critical element or any explanatory notes relevant to the evidence.

Because a State will submit numerous pieces of evidence, the Department recommends that the State use a coding or naming scheme to identify the various pieces of evidence cited in its State Assessment Peer Review Submission Index. Exhibit 3, at the end of this part of this document, shows suggested formats for how a State might present its submission.

Preparing Evidence. The Department encourages a State to take into account the following considerations in preparing its submission of evidence. See Exhibit 2 for examples of the State submission of evidence.

The description and examples of evidence apply to each assessment in the State’s assessment system (e.g., general and alternate assessments, assessments in each content area). For example, for Critical Element 2.3 – Test Administration, a State must address the critical element for both its general assessments and its alternate assessments. In general, evidence submitted should be based on the most recent year of test administration in the State.

Multiple critical elements are likely to be addressed by the same documents. In such cases, a State is encouraged to streamline its submission by submitting one copy of such evidence and cross-referencing the evidence across critical elements in the completed State Assessment Peer Review Submission Index for its submission. For example, it is likely that the test coordinator and test administration manuals, the accommodations manual, technical reports for the assessments, results of an independent alignment study, and the academic achievement standards-setting report will address multiple critical elements for a State’s assessment system. A State should avoid submitting the same document multiple times but should simply reference specific page numbers appropriate to each critical element.

https://www2.ed.gov/admins/lead/account/saa.html

Similarly, if certain pieces of evidence are substantially the same across assessments, a sample, rather than the full set of such evidence, may be submitted. For example, if a State has submitted all of its grades 3-8 and high school reading/language arts and mathematics assessments for assessment peer review, sample individual student reports must be submitted for both general and alternate assessments under Critical Element 6.4 – Reporting. However, if the individual student reports are substantially the same across grades, the State may choose to submit a sample of the reports, such as individual student reports for both subjects for grades 3, 7, and high school and provide narrative explaining that they are not substantively different. Note: this would not be acceptable for a fundamental component of the system, such as the alignment of the assessment to a State’s academic content standards. The State needs to submit information for all grades and subjects, such as an independent alignment study, to demonstrate the validity of the assessment based on content.

For some critical elements, a State should consider providing two types of evidence: (1) evidence of State policy; and (2) evidence that the policy was carried out. For example, in Critical Element 2.4 – Monitoring Test Administration, a State should provide evidence of monitoring policies and protocols, as well as evidence that monitoring occurred. This could include a monitoring report or a list of recent monitoring activity.

A State should prepare its submission in electronic format and the files should be clearly indexed, with corresponding electronic folders, folder names, and filenames. For evidence that is typically presented in an Internet-based format, screenshots may be submitted as evidence.

Links to websites should not be submitted as evidence. Each State should consult the Department when planning its peer review submission to receive a specific timeline for its review. At that time, a State will receive detailed instructions on how to submit the electronic evidence and index files for that peer review session.

Exhibit 2: Examples of a Prepared State Index for Selected Critical Elements Critical Element 4.2 – Fairness and Accessibility (EXAMPLE) Evidence Notes

For all State academic content and ELP assessments, assessments should be developed, to the extent practicable, using the principles of universal design for learning. “Universal design for learning” means a scientifically valid framework for guiding educational practice that--

(A) Provides flexibility in the ways information is presented, in the ways students respond or demonstrate knowledge and skills, and in the ways students are engaged; and

(B) Reduces barriers in instruction, provides appropriate accommodations, supports, and challenges, and maintains high achievement expectations for all students, including students with disabilities and ELs.

For academic content assessments, the State has taken reasonable and appropriate steps to ensure that its assessments are accessible to all students and fair across student groups in the design, development and analysis of its assessments;

For ELP assessments, the State has taken reasonable and appropriate steps to ensure that its assessments are accessible to all EL students and fair across student groups, including EL students with disabilities, in the design, development, and analysis of its assessments.

General assessments in reading/language arts and mathematics:

Evidence #24: Technical Manual (2015).

The technical manual for the State assessments documents steps taken to ensure fairness:

• Pp. 30-37 discuss steps taken during design and development.

• Pp. 86-92 discuss analyses of assessment data.

Evidence #25: Summary of follow-up to differential item functioning (DIF) analysis.

Evidence #26: Amendment to assessment contract requiring additional bias review for items and added instructions for future item development.

Alternate assessments in reading/language arts and mathematics:

The Example State’s alternate assessments were developed by the ABC assessment consortium.

Evidence for the assessments was submitted on this State’s behalf…