Att K - Scope of Work - update term and link.docx

Detailed Scope of Services: I AM, ILEARN and IREAD Assessments

Section 1: General Information and Non-Negotiables

1.1 General Information Comment by Skinnari, Ian: FYI: Will review/update dates once IDOA confirms whether we have approval for 5+1 or 6+1.

The purpose of this Request for Proposal (RFP) is to secure a contractor to develop, administer, score, and report three of Indiana’s statewide assessments: ILEARN, IREAD, and I AM. These three assessments will be administered by the same vendor so that schools can experience continuity across assessments. This contract applies to assessments administered during school year 2027-2028, 2028-2029,2029-2030, 2030-2031, 2031-2032, and 2032-2033. One additional school year may be added to the contract as an extension upon mutual agreement.

· ILEARN: Comprises a through-year assessment system (TYA) in mathematics and English/language arts, which includes three interim Checkpoint assessments administered throughout the year, as well as summative assessments at the end of the year for accountability. Additionally, it includes grade-level specific summative assessments in science and social studies. This assessment has already been developed and administered previously, and it must maintain validity and comparability. Details about assessment design, scoring, student participation, and other assessment requirements are provided in detail in Section 2, ILEARN.

· IREAD: A foundational reading assessment measuring foundational reading standards administered to grades 2 and 3 in spring and summer administrations. This assessment has already been developed and administered previously, and it must maintain validity and comparability. Details about assessment design, scoring, student participation, and other assessment requirements are provided in detail in Section 3, IREAD.

· I AM: An alternate assessment which compliments the general assessments with the focus on a transition from a summative to a through-year assessment. This assessment will be newly developed and administered beginning with this contract. Details about assessment design, scoring, student participation, and other assessment requirements are provided in detail in Section 4, I AM.

Requirements that pertain to all three Indiana statewide assessments (ILEARN, IREAD, and I AM) are described in detail in Section 5, All Assessment Programs. Each requirement listed in Section 5 must be met for ILEARN, IREAD, and I AM assessments. Requirements listed in Section 2, Section 3, and Section 4 only need to be met for the specific associated assessment.

All Indiana assessments are administered online, with comparable paper test forms available for students with approved paper accommodations only.

1.2 Non-Negotiables

In order to be considered as a viable solution, a contractor must meet ALL of the following non-negotiable requirements.

· Application Programming Interface (API) connection to integrated intervention or curriculum provider resources, requirements listed in Section 2.8(e): The contractor must provide an API system.

· Through-Year Assessment Delivery Capabilities, requirements listed in Section 2.1 and Section 4.1: The contractor must be able to administer interim Checkpoint assessments at any time throughout the year as well as summative assessments at the end of the year.

· Combined Scoring Approach Capabilities: The contractor must be able to implement scoring approaches that combine data across multiple test occasions including:

· Predictive models that incorporate student performance across time and multiple opportunities, as described in Section 2.8(b).

· Domain scoring that supplements end-of-year performance with performance on interim assessments, as described in Section 2.8(b).

· Reports: The suite of reports for ILEARN and I AM must provide individual and group skill plans, standards, through-year and end-of-year reporting, and cumulative reporting, as described in Sections 2.8 and 4.8

· Dynamic Data Reporting System for Families: The contractor must have a reporting retrieval system for families to access individual student reports, skills information to support students, and links to resources as described in Section 5.7(e). The reporting system must support through-year and end-of-year reporting.

· Released Items Repository (RIR): The contractor must have a platform where items that have been released can be viewed and practiced for each assessment by the public (no sign-in required) as described in Section 5.8.

· Capability to perform cut score validations and standard setting using Principled Assessment Design research models (including ID-matching and Bookmark method) as described in Section 2.7(d).

· System capability to deliver multiple computer-adaptive tests using the same item pool to the same student without repeating items in the delivery as described in Section 2.1 and Section 3.1.

· This system capability is essential for test validity with multiple opportunity assessments, including IREAD and ILEARN Checkpoints.

· Ability to create, administer, score, and report numerous item types, including selected-response, constructed-response, technology-enhanced, and performance-based items as described in Section 2.3(b) and Appendix A.

· Ability to create, administer, and score item clusters (including simulation item types) with assertion-based scoring for science assessments as described in Section 2.3(a), Section 2.3(b), Section 2.7(a), and Section 5.4(e).

· Ability to administer two modes of assessment for I AM; one with full graphic supports and one with limited graphic supports as described in Section 4.2.

Section 2: ILEARN

2.1 ILEARN Assessment Overview

Beginning school year 2027-2028, IDOE will partner with the selected contractor to administer the ILEARN Through-Year Assessment (TYA) system. ILEARN TYA measures the 2023 Indiana Academic Standards for Mathematics, English/Language Arts, Science, and Social Studies. Test blueprints and item specifications are available on the Indiana Assessment Frameworks webpage. These assessments have been developed previously and are administered using existing item banks and scoring models. These assessments must remain stable and comparable. This short video provides a brief overview of the ILEARN TYA system. Comment by Skinnari, Ian: Provide IDOA link when available.

Within this scope of work, a computer-adaptive test (CAT) is defined as an assessment that adapts by independent item or sets of items (i.e., those items aligned to a stimulus) as a student interacts with the assessment. The test must adapt through student engagement to most precisely estimate a student’s mastery of the defined content. Fixed form assessments present all students with the same questions and tasks to measure achievement. An overview of the assessments provided to students in each content area is provided in Table 2.1.

Table 2.1: ILEARN Assessment Overview

English/ Language Arts

* For each summative assessment, an accommodated paper fixed form must be developed and administered for a limited number of students with paper accommodations. Less than 1% of the student population will participate in a paper assessment.

2.1(a) Anticipated ILEARN Test Windows Anticipated ILEARN test windows for each school year are provided in Table 2.1(a). The Indiana State Board of Education (SBOE) sets testing windows each December for the following school year.

Table 2.1(a): Anticipated ILEARN Testing Windows

* The ILEARN High School End-of-Course Assessments (ECAs) are available as a long window from late fall to late spring to support a variety of course schedules.

2.2 ILEARN Assessment Development and Design

ILEARN follows an established test design, documented by the Indiana Assessment Framework for each content area and supporting specifications. The contractor must explain how each test’s design will be maintained. This explanation must include (but is not limited to) a demonstration of the ability to deploy the current test design, experience with computer-adaptive testing (CAT), and clear steps that will be taken to ensure comparability is maintained across test forms.

2.2(a) Mathematics:

· Administer three computer-adaptive interim Checkpoint assessments (with two opportunities available for each Checkpoint).

· Ensure that the entry-level of each subsequent assessment is based on performance of the previous assessment.

· Administer one computer-adaptive summative assessment.

· Adhere to all specifications as outlined in the Indiana Assessment Framework: Mathematics for grades 3-8.

· Adapt at the item level for all assessments.

· Incorporate 1-2 handscored items per student for the summative assessment only. No handscored items will be administered in the Checkpoint assessments.

· Provide a testing-system embedded, grade-level specific calculator for items aligned to calculator-designated standards, while restricting the use of the embedded calculator for items aligned to non-calculator standards.

· Provide a testing system embedded, grade-level specific reference sheet (or formula sheet) for all students. The reference sheet must be available in Spanish for Spanish-translated assessments.

· Utilize a variety of item types including all item types within Appendix A: Example ILEARN Item Types.

· Support untimed testing.

· Administered using a combination of Indiana-owned items and other items owned by a third party. Appendix B: Estimated Operational ILEARN and IREAD Items Beginning School Year 2027-2028 provides an overview of the number of items within each bank that are projected as available for operational use.

2.2(b) English/Language Arts:

· Administer three computer-adaptive Checkpoint assessments (with two opportunities available for each Checkpoint).

· Ensure that the entry-level of each subsequent assessment is based on performance of the previous assessment.

· Administer one computer-adaptive summative assessment.

· Administer one performance task following the summative assessment, which can be adapted based on students’ performance on the CAT. The performance task includes a passage set with two items; one research item that is constructed response, and one essay. Performance tasks are entirely hand-scored.

· Adhere to all specifications as outlined in the Indiana Assessment Framework: English/Language Arts for grades 3-8.

· Adapt at the item level as much as possible, including within passage sets. Describe the contractor’s ability to employ computer-adaptive item selection within a passage set.

· Incorporate 2-3 total handscored items per assessment for the summative assessment only. Two of these handscored items will appear on the performance task. Zero or one will appear in each student’s summative CAT for summary standards only. No handscored items will be administered in the Checkpoint assessments.

· Utilize a variety of item types including all item types within Appendix A: Example ILEARN Item Types.

· Support untimed testing.

· Administer using a combination of Indiana-owned items and other items owned by a third party. Appendix B: Estimated Operational ILEARN and IREAD Items Beginning School Year 2027-2028 provides an overview of the number of items within each bank that are projected as available for operational use.

· Score items using an item scoring model based on scoring assertions.

2.2(c) Science:

· Administer one computer-adaptive summative assessment, comprised of one section of NGSS-Next Generation Science Standards aligned items and one section of computer-science-aligned items for grades 4 and 6 respectively.

· Administer one computer-adaptive summative assessment comprised of NGSS-aligned items for High School Biology in one continuous window allowing for students to participate at the completion of local trimester and semester school schedules.

· Adhere to all specifications as outlined in the Indiana Assessment Framework: Science for grades 4, 6, and High School Biology.

· Adapt at the item-set level for item cluster and at the item level for stand-alone items.

· Utilize a variety of item types including all item types within Appendix A: Example ILEARN Item Types associated with the science assessment. No handscored items will be administered in the science assessments.

· Support untimed testing.

· Administer using a combination of NGSS-MOU-owned items and Indiana-owned items. Appendix B: Estimated Operational ILEARN and IREAD Items Beginning School Year 2027-2028 provides an overview of the number of items within each bank that are projected as available for operational use.

2.2(d) Social Studies:

· Administer one fixed-form summative assessment for grade 5 and for high school U.S. Government respectively.

· Adhere to all specifications as outlined in the Indiana Assessment Framework: Social Studies for grades 5 and High School U.S. Government.

· Utilize a variety of item types including all item types within Appendix A: Example ILEARN Item Types associated with the social studies assessment. No handscored items will be administered in the social studies assessments.

· Support untimed testing.

· Administer using solely Indiana-owned items. Appendix B: Estimated Operational ILEARN and IREAD Items Beginning School Year 2027-2028 provides an overview of the number of items within each bank that are projected as available for operational use.

2.3 ILEARN Existing Items, Item Development, and Item Ownership

2.3(a) Existing Items ILEARN uses items from three sources. The contractor must provide assurances that items from all three sources may continue to be used for the life of the contract, under the terms presented for each item bank below. The number of items projected to be available from each source are provided in Appendix B: Estimated Operational ILEARN and IREAD Items Beginning School Year 2027-2028. Currently, all operational items have been tagged with text-to-speech, braille, and Spanish supports as appropriate for the content area.

· Indiana-Owned Items: These items are owned by the State of Indiana and will be provided at no additional cost to the contractor as part of the contract.

· NGSS Items: These items are owned by a state collaborative group called the NGSS Memorandum of Understanding (MOU). Indiana maintains a relationship with this group outside of this contract and ensures that items will be available for use so long as Indiana maintains the partnership. In addition, the contractor must work with IDOE to accept new NGSS MOU items into the item bank annually, and to update existing item content (e.g., accessibility tags) as needed when the source item receives updates.

· New items accepted into the Indiana item bank require an educator Item Acceptance Review. The contractor is responsible for planning and facilitating one Item Acceptance Review for NGSS items each contract year. Additional details about educator committee expectations are documented in Section 5.6.

· Summative Gap-Fill Items from Third-Party Item Bank: Indiana currently uses items from the Cambium Assessment, Inc. Clearsight item bank. These items fill identified gaps in locked operational item pools for mathematics and English/Language Arts and also increase the adaptability of ILEARN assessments. The contractor must provide a comparable solution for additional items to fill this need. Additional details about Indiana’s current use of these items can be found in Appendix B: Estimated Operational ILEARN and IREAD Items Beginning School Year 2027-2028.

· Additional Item Banks: The contractor may recommend additional item banks or sources for items in the proposal. If the contractor proposes additional item bank sources, the contractor must clearly define the process for using the items and the plan to refresh the assessments with the new items, along with a process for Indiana’s review and acceptance (or rejection) of new items into the ILEARN item bank. The contractor must also describe characteristics of the proposed item bank that assure the quality of the items proposed, include the process by which items are developed, reviewed, and operationalized in the bank and the level of security associated with the item uses.

The contractor shall include any applicable lease or licensure fees in the contract billing schedule. The licenses are intended for the duration of the contract plus one additional year (to maintain consistency in the event of a transition) and all subsequent renewal periods.

Cambium Assessment, Inc. (CAI) currently maintains Indiana’s item bank and will deliver items to the contractor via item export. The contractor must describe any steps that may be necessary to import items into the contractor’s item banking system for use in content maintenance and test delivery. All item style, accessibility tagging, and item metadata must be maintained during the import process.

2.3(b) Item Development The contractor will work with IDOE to develop items for all ILEARN assessments. This item development fulfills multiple purposes, including (1) continuing to build the adaptive capabilities of the item bank, (2) updating the item bank for test security purposes, (3) maintaining quality items and item parameters, and (4) allowing for the release of items for public use, as required by Indiana legislation.

Needed item development may vary year over year. The contractor should provide a per-item cost for development and confirm that the contractor and IDOE can work together, each year, to create an item development plan based on the program’s needs. When providing per-item costs, the contractor should be sure to include:

· Costs for developing different item types (e.g., science item clusters with scoring assertions, ELA performance tasks, stand-alone items, technology-enhanced items, passage-based items) as applicable (see Appendix A for a description of ILEARN item types).

· Costs for tagging items with accommodations (e.g., text-to-speech, braille, Spanish translation, Spanish text-to-speech) as applicable.

· All items developed for science are required to be tagged with Spanish text-to-speech per the NGSS MOU agreement.

The contractor is responsible for:

· Creating detailed content-area specific item development plans, addressing specific categories in which items are to be developed, in collaboration with IDOE, each year, based on a mutually agreed on total number of items to develop.

· Writing quality items aligned with Indiana’s item specifications, passage specifications, passage models, style guide, and content/sensitivity/bias guidelines to fulfill content-area item development plans.

· Providing IDOE with the opportunity to review and request revisions for items.

· Reviewing all newly developed items with educators in Content and Fairness committees (in collaboration with IDOE).

· Adjusting items (in collaboration with IDOE) based on changes suggested from these reviews.

· Tagging items for accommodations, as applicable.

· Ensuring items appear in the student test delivery system as expected.

· Performing quality assurance checks on items during and after field testing including (but not limited to) data review, range finding (for handscored items, see Section 2.7(b) Handscoring for additional details) and rubric validation.

· Field testing items prior to operational use through embedded field testing in the operational assessment.

· Calibrating items to Indiana’s scale using the current scoring model. See Section 2.7(a) Scoring Specifications for additional details about the scoring models applied to ILEARN.

The contractor should describe in detail their process for item development, especially quality assurance protocols that are in place to ensure all items are accurate, correct, and of high quality.

2.3(c) Item Ownership All items developed under this contract are the sole property of the State of Indiana and may only be used for other purposes with explicit written permission.

2.4 ILEARN Accessibility Features and Accommodations

All students in Indiana are required to participate in the ILEARN assessment unless they are students with significant cognitive disabilities. Students with disabilities, students with Section 504 Plans, and English learners will all participate in the assessments, unless they are students with significant cognitive disabilities for whom Indiana’s Alternate Measure (I AM) is appropriate. In order to ensure ILEARN is accessible to all students, the contractor must provide certain accessibility features and accommodations in accordance with students’ special learning plan requirements. These accommodations, if embedded in the online test delivery platform, must also be able to function across Microsoft, Apple, and Google operating systems and devices.

The contractor must provide a complete list of all accessibility features, tools, supports, and accommodations that will be available for use on ILEARN assessments. The contractor should clarify if accommodations can be provided at the program level, grade level, individual test level, item level, or any other combination. The accommodations and features listed in Table 2.4 are required at a minimum; however, additional tools and features can be recommended. For additional details about current accommodations use, the contractor may review the Indiana Accessibility and Accommodations for Statewide Assessments.

Table 2.4: Required Features and Accommodations for ILEARN Assessments Embedded (Online) Universal Features: Tools or supports that are available to all students.

Indiana uses a specific version of the Desmos calculators. In Grade 6, Indiana replaces the square root button with a fraction button. In Grades 7 and 8, Indiana turns off the “convert to fraction” button in the answer bar, and the letter keypad is ordered as “ABC” rather than “QWERTY”.

Embedded (Online) Designated Features: Tools or supports that can be turned on for any student, but are not turned on by default.

Many items used to administer ILEARN already have Spanish translations available as part of the item tagging. See Appendix B for details.

Embedded (Online) Accommodations: Tools or supports provided only to those students with officially documented accommodations.

Verbal student responses may be indicated in English for all subject areas or Spanish for mathematics, science, or social studies.

Text-to-Speech tracking (highlights on the screen that follow the read-aloud text) should be included as part of this accommodation.

Text-to-Speech tracking (highlights on the screen that follow the read-aloud text) should be included as part of this accommodation.

Text-to-Speech tracking (highlights on the screen that follow the read-aloud text) should be included as part of this accommodation.

Students with a text-to-speech accommodation may select a tool to allow the completion options to be read aloud.

Non-Embedded Accommodations: Tools or supports provided only to those students with officially documented accommodations outside of the test delivery system.

A braille booklet is not provided for ILEARN Checkpoints as the Checkpoints are exclusively available online.

ILEARN U.S. Government is not provided via paper. Students needing to access braille tests must use refreshable braille displays.

ILEARN U.S. Government is not provided via paper.

A print booklet is not available on the ILEARN Checkpoints as the Checkpoints are exclusively available online.

ILEARN U.S. Government is not provided via paper.

The contractor must describe the process for collaborating with IDOE on the tools, supports, and accommodations which will be tracked and utilized in the online platform. The contractor should describe its system’s capabilities to track and monitor student use of accommodations, tools, and features. Preference is given to contractors that can track accommodation activity at the item level.

The contractor is required to provide an ILEARN accommodations file to IDOE prior to and following each summative testing window. The accommodations file provides information about accommodations assigned to each student and includes unique student identifiers as well as school and corporation associations.

The contractor must explain how its systems are compliant with the following standards and guidelines:

· Plug and Play (PNP) standards compliance. See http://www.imsglobal.org/accessibility/accpnpv2p0/spec/ISO_ACCPNPinfoModelv2p0.html

· U.S. Rehabilitation Action Section 508, which requires that all website content be equally accessible to individuals with disabilities.

· Web Content Accessibility Guidelines 2.1, which will make content accessible to a wider range of individuals with disabilities, including blindness and low vision, deafness and hearing loss, learning disabilities, cognitive limitations, limited movement, speech disabilities, photosensitivity, and combinations of these.

2.4(a) Paper Tests (Print Booklets) The Print Booklet accommodation requires the contractor to develop, produce, and score a fixed-form paper test. All print booklets must adhere to the specifications found in the Indiana Assessment Frameworks and must be comparable to online forms. The contractor must describe the steps they will take to develop fixed forms of high technical quality. Fixed forms may be used for three consecutive years and then must receive a refresh at a rate of at least 15% of items. However, any issues discovered with a fixed form during test administration must be adjusted prior to any further use of the form. For ILEARN, print booklets are only developed and provided for summative assessments. Checkpoint assessments are administered online only. No paper test forms will be developed for ILEARN Checkpoint assessments.

ILEARN Summative Print Booklets to be produced include:

· One scorable regular print booklet for each grade and content area.

· One large print booklet for each grade and content area. This is the same test form and booklet as the regular print but printed in increased font size (18 pt.) on larger sized paper. Student responses in the large print booklet are transcribed into a scorable regular print booklet by the school.

· One braille print booklet for each grade and content area. This is the same test form as the regular print booklet but transcribed into braille. Student responses in the braille test booklet are transcribed into a scorable regular bring booklet by the school.

· The braille forms will be produced using UEB contracted braille with Nemeth math code where needed for math and science assessments.

Paper test kits must be created and shipped to schools upon order. The contractor must describe a proposed process for the order and shipping of paper test kits to schools, as well as the materials included in the kits. The contractor should plan to print and ship the following quantities of materials:

· 180 regular print test book kits per grade and content area

· 50 large print test book kits per grade and content area

· 15 braille test book kits per grade and content area

2.5 ILEARN Ancillaries (Manuals and Scripts)

The contractor must develop and publish accurate, high-quality manuals and scripts to support the administration of ILEARN assessments. These manuals and scripts must adhere to IDOE’s established style guides (provided to the contractor upon contract execution) and must be updated each school year as needed. All manuals and scripts must be approved by IDOE and published at least six weeks in advance of the start of the respective testing window. Additional requirements for quality assurance and review processes are included in Section 5.13.

2.5(a) Test Administrator’s Manual (TAM) The TAM provides all the information that corporation and school test coordinators and test administrators need to prepare for and conduct successful online testing. The TAM will contain no secure information and will be published electronically for easy access; it will not be printed or shipped to schools. Appendix C provides an example Table of Contents for the ILEARN TAM. IDOE will provide the most up-to-date ILEARN TAM to the contractor upon contract award. The contractor must review and update the TAM prior to the first test administration and each subsequent administration as needed. IDOE must approve the content of the TAM before it is posted.

2.5(b) Online Testing Scripts The online testing scripts contain specific instructions that test administrators read aloud to students and follow during online test administration. These scripts contain no secure information and will be published electronically for easy access; they will not be printed or shipped to schools. IDOE will provide the most up-to-date scripts to the contractor upon contract award. The contractor must review and update the scripts prior to the first test administration and each subsequent administration as needed. A list of current ILEARN scripts follows:

· ILEARN Online Practice Testing Scripts:

· ILEARN Grades 3-5 Online Practice Testing Script (English and Spanish versions)

· ILEARN Grades 6-8 Online Practice Testing Script (English and Spanish versions)

· ILEARN Biology ECA Online Practice Testing Script (English and Spanish versions)

· ILEARN U.S. Government ECA Online Practice Testing Script (English and Spanish versions)

· ILEARN ELA Performance Task Online Practice Testing Script (English and Spanish versions)

· ILEARN Checkpoints Online Testing Scripts:

· ILEARN Checkpoints ELA Online Testing Script (English and Spanish Version)

· ILEARN Checkpoints Mathematics Online Testing Script (English and Spanish Version)

· ILEARN Summative Online Testing Scripts:

· ILEARN ELA Online Testing Script (English and Spanish Version)

· ILEARN Mathematics Online Testing Script (English and Spanish Version)

· ILEARN Science Online Testing Script (English and Spanish Version)

· ILEARN Social Studies Online Testing Script (English and Spanish Version)

· ILEARN Biology ECA Online Testing Script (English and Spanish Version)

· ILEARN U.S. Government ECA Online Testing Script (English and Spanish Version)

2.5(c) Paper Testing Scripts The paper testing scripts contain specific instructions that test administrators read aloud to students and follow during paper test administration. These scripts do contain secure information (listening stimuli read aloud to students during their ELA assessment) and will be printed and shipped to schools along with their paper test materials. IDOE will provide the most up-to-date scripts to the contractor upon contract award. The contractor must review and update the scripts prior to the first test administration and each subsequent administration as needed. A list of current ILEARN scripts follows (each grade-level script contains instructions for all content areas administered for that grade level):

· ILEARN Grade 3 Paper Testing Script

· ILEARN Grade 4 Paper Testing Script

· ILEARN Grade 5 Paper Testing Script

· ILEARN Grade 6 Paper Testing Script

· ILEARN Grade 7 Paper Testing Script

· ILEARN Grade 8 Paper Testing Script

· ILEARN Biology ECA Paper Testing Script

· ILEARN U.S. Government ECA Paper Testing Script

2.6 Remote Testing for ILEARN Checkpoints

The Indiana school system currently includes a small number of online schools. For online schools only, IDOE allows remote proctoring of administrations of ILEARN practice tests and interim Checkpoint assessments. No remote proctoring is allowed for summative assessments. The contractor must:

· Describe its ability to provide remote proctoring for certified Test Administrators, students, and specific tests (e.g., Checkpoint and not summative) only.

· Provide a training and certification course for remote Test Administrators.

· Explain how the remote proctoring system provides features to Test Administrators that promote valid and reliable testing experiences (e.g., cameras on for students, ability to communicate with the entire test session or with individual students).

· Provide a recommended student-to-test-administrator ratio for remote proctoring test sessions, along with rationale.

· Explain system features that will disallow (protect from) administration of summative assessments through the remote proctoring system (e.g., allow Checkpoint assessments to be administered through specialized remote proctor settings, but not allow summative assessments to be administered through those specialized remote proctor settings).

· Provide a way to only allow approved schools to administer the appropriate assessments remotely.

2.7 ILEARN Scoring

The contractor is responsible for scoring ILEARN assessments accurately and consistently with current practice, as defined within this section. The contractor must demonstrate the ability to employ the research-based scoring models described.

2.7(a) Scoring Specifications and Item Response Theory (IRT) Models ILEARN is scored using a specified process. The contractor must maintain current scoring models. The contractor will use previously established item parameters to score the assessments. ILEARN Summative assessments and ILEARN Checkpoint assessments use the same reporting scale and underlying theta scale (specific to each content area). A vertical scale is employed across grade levels for Mathematics and English/Language Arts.

· ILEARN ELA Grades 3-8, Mathematics Grades 3-8, and Social Studies Grade 5 are all scored using two Item Response Theory Models. For one-point items, the two-parameter logistic model (2PL) is used. For multi-point items, the generalized partial credit model (GPC) is used. Maximum likelihood estimation (MLE) will be used to estimate student ability.

· ILEARN Science Grades 4, 6, and Biology ECA are scored using Marginal Maximum Likelihood Estimator (MMLE) within the bifactor IRT framework using the Newton Raphson method. This method addresses the nuanced design of the item clusters that are specific to the science assessments.

· ILEARN U.S. Government is scored using the Rasch model.

The contractor must describe their experience scoring large-scale assessments using each of these statistical models.

The contractor must produce scoring specifications and update them annually, providing them each year to IDOE for review, discussion (as needed), and approval. IDOE will provide the contractor with current scoring specifications upon contract award. The scoring specifications must include (but are not limited to):

· Details about the IRT model(s) being used to score the assessment

· Scoring of “all correct” and “all incorrect” cases

· Test attemptedness and participation business rules

· The student-level scores that will be reported

· Information about method of transforming theta scale to reporting scale, as appropriate

· Methodology of calculating the standard error of measurement

· Other scoring business rules that are in place, as appropriate

2.7(b) Handscoring Handscoring occurs only for ILEARN Mathematics and English/Language Arts summative assessments. All other Indiana assessments do not contain any handscored items by design.

· ILEARN Mathematics Summative Grades 3-8: Each student will receive one or two operational handscored items (depending on computer adaptive algorithm). All handscored items will be short constructed response item types.

· ILEARN English/Language Arts Summative Grades 3-8: Each student will receive two or three operational handscored items (depending on computer adaptive algorithm).

· All students will receive one short constructed response research item in their Performance Task.

· All students will receive one writing prompt (e.g., essay/long response) item in their Performance Task.

· Students may receive up to one short constructed response summary item in their CAT.

While each test form includes the same range of handscored items, students may receive different handscored items based on the computer-adaptive nature of the assessment. Currently, IDOE maintains 12 research items and 6 essay items for each of three genres within each grade level. This totals to 18 writing prompts (each with 2 research questions associated) per grade level. IDOE also maintains handscoring anchors and training materials for these items.

The contractor is responsible for all aspects of handscoring ILEARN open-ended items accurately and reliably, including:

· Recruiting Qualified Handscorers: The contractor must provide IDOE with the qualifications required for handscorers and for handscoring directors. These qualifications may be subject to change based on IDOE approval.

· Indiana Code 20-32-5.1-12 requires Indiana educators to be included in the handscoring of ILEARN items to the extent possible. Handscoring may occur either in person or virtually. At least three months prior to the first spring administration, the contractor must submit to IDOE a detailed plan outlining how Indiana educators will be recruited and included in the scoring process. The plan shall include opportunities for IDOE to publish recruitment materials through established communication methods directly to Indiana educators.

· When hiring applicants for ILEARN handscoring, preference should be given to current Indiana educators.

· Training Handscorers: The contractor must provide a description of the training provided to handscorers, including qualification sets and thresholds of accuracy that must be met prior to scoring operational items.

· Handscoring: All Applicable Items in a Timely Manner: The contractor must provide a plan for handscoring that maintains the reporting timelines established in Section 2.8. The contractor must provide a mitigation plan for monitoring and responding to any risks associated with the handscoring timeline.

· Maintaining the security of all test items during all portions of the handscoring process.

· Second-Scoring: At least 10% of the handscored items (both online and paper) will be second-scored for reliability checks. The contractor must describe how they will define a second-score sample, perform data analysis as needed, and use these to monitor item performance and scorer performance. The contractor will develop a second-scoring plan and provide it to IDOE for review and approval.

· Monitoring the Accuracy and Reliability of Scorers: The contractor must provide documentation of the use of established procedures to monitor handscorers in real time, including (but not limited to) the use of validity papers during scoring and real-time monitoring of score distributions by scoring directors.

· Checking for drift of scores by using pre-scored samples at specified intervals.

· Meeting reliability standards during all phases of handscoring, as agreed on by IDOE and the contractor. Any handscorer that does not meet minimum reliability expectations will be retrained (or dismissed) and student work assigned to them will be rescored.

· Generating reports of representative samples for all operational assessments.

· Reporting issues to or seeking guidance from IDOE as needed.

· Developing Training Materials for Field-Tested Handscored Items: The contractor must develop annotated anchor sets, training/practice sets, and qualification sets for field-tested handscored items and provide these sets to IDOE for review. The contractor will meet with IDOE as needed to discuss these training sets and ensure full calibration of hand-scoring expectations for any individual item.

· Annotated anchor sets must include a minimum of 10 student responses that represent the variety of score points across the rubric as well as the variety of student responses received in the field test.

· Training/practice sets must include a minimum of 10 practice papers across a variety of score points.

· Qualification sets must include a minimum of 10 qualification papers across a variety of score points.

· Check sets to monitor scorer drift must include a minimum of 20 monitoring papers across a variety of score points.

· Score point designations must be clearly delineated for materials used in the training and handscoring process.

· Maintaining anchor sets and training materials for handscored items. Anchor sets and training materials exist for all operational handscored items and will be provided to the contractor by IDOE.

· Implementing a scoring reliability (auditing) system that continuously monitors score quality and reports statistics for representative samples of student responses.

· Flagging student responses with troubling content (e.g., suicidal thoughts, violent intentions) and providing those responses (redacted as appropriate to preserve the integrity of test items) to permissioned school users for follow up with students.

2.7(c) Rescores for Handscored Items Indiana Code 20-32-5.1-13 requires that each student and the student’s parents be given the opportunity to inspect the student’s scored responses and scoring rubrics for handscored items. Additionally, a student’s parent or student’s principal may request a rescoring of the student’s responses to a statewide assessment, including the essay portion. The contractor must establish a secure online means by which schools will facilitate the rescore process of handscored items as requested by a parent or school administrator upon the conclusion of each ILEARN summative test administration. (Rescores only apply to ILEARN summative assessments. They do not apply to ILEARN Checkpoint assessments, IREAD, or I AM because these assessments do not contain any handscored items.)

Each rescore window will open on the same day as preliminary results become available and have a duration of at least two weeks. If the student’s score changes (increase or decrease) as a result of the rescore, the student’s results will be updated in the online scoring system. The rescore application must require user acknowledgement of a confidentiality statement to protect secure items from being released publicly. The rescore platform must have the capability of showing parents and administrators items eligible for rescoring, as well as the related passage or stimulus (if applicable), associated student response, corresponding anchor set (with the assigned score point clearly delineated on each sample response) and the corresponding scoring rubric(s).

The contractor must develop two training modules to provide an overview of the online rescore process: one for schools and one for parents/families. These modules are intended to be short (3-7 minutes) and should provide each user with the steps that they can take to view rescore eligible items…