Att F - Tech Proposal_041626.docx

RFP 26-86999

TECHNICAL PROPOSAL QUESTIONS - Detailed Scope of Services I AM, ILEARN, and IREAD Assessments

ATTACHMENT F

Instructions: Vendor should provide a document formatted with Question #, RFP SOW Section, and Response Area followed by the vendor’s narrative response to that question. The response must address all items detailed below and provide the information and documentation as required. The response must be structured to address each question listed below. A table of contents (see “4. Table of Contents”) must also be completed as listed in this Attachment. Where appropriate, supporting documentation may be referenced by a page and paragraph number. However, when this is done, the body of the Technical Proposal must contain a meaningful summary of the referenced material. The referenced document must be included as an appendix to the technical proposal with referenced sections clearly marked. If there are multiple references or multiple documents, these must be listed and organized for ease of use.

For Section 1 refer to the column below ‘RFP SoW Section’ for detailed scope information for each question.

Section 1: Scope of Work Questions

The Respondent must provide a description regarding how it will meet and address the requirements shared in Section 3.2 IREAD Existing Items, Item Development, and Item Ownership from the Scope of Work document for this RFP.

Section 2: Assessment Criteria and Evidence Questions Instructions: Vendor should include the Question # and Criteria followed by a narrative response that addresses the questions and includes the required Evidence.

Part A. Meet Overall Assessment Goals and Ensure Technical Quality[footnoteRef:2] [2: The term “technical quality” here refers to the qualities necessary to ensure that scoring and generalization inferences based on test scores are valid both within and across years. Also refer to other sources, primarily The Standards for Educational and Psychological Testing.]

Ensuring that assessments are valid for required and intended purposes: Assessments produce data, including student-level achievement data and student growth data required under Federal Indicator 7 of the U.S. Department of Education’s Office of Special Education Programs, that can be used to validly inform the following:

· Positive Social-Emotional skills.

· Acquisition and use of knowledge and skills.

· Use of appropriate behaviors to meet needs; and

· Other purposes defined by the state.

· Provide a well-articulated validity evaluation based on an interpretive argument (e.g., Kane, 2006) that includes, at a minimum:

· Evidence of the validity of using results from the assessments for the three primary purposes, as well as any additional purposes required by the state (specify sources of data).

· Evidence that scoring and reporting structures are consistent with structures of the state’s standards (specify sources of data).

· Evidence that total test and relevant sub-scores are related to external variables as expected (e.g., other measures of the construct). To the extent possible, include evidence that the items are “instructionally sensitive,” that is, that item performance is more related to the quality of instruction than to out-of-school factors such as demographic variables.

· Evidence that the assessments lead to the intended outcomes (i.e., meet the intended purposes) and minimize unintended negative consequences. Consequential evidence should flow from a well-articulated theory of action about how the assessments are intended to work and be integrated with the larger accountability system.

· The set of content standards against which the assessments are designed is provided. If these standards are the state’s standards, evidence is provided that the content of the assessments reflects the standards, including the cognitive demand of the standards. If they are not the State’s standards, evidence is provided of the extent of alignment with the State’s standards.

· Evidence is provided to ensure the content validity of test forms and the usefulness of score reports (e.g., test blueprints demonstrate the learning progressions reflected in the standards, and experts in the content and progression toward readiness are significantly involved in the development process).

Ensuring that assessments are reliable: Assessments minimize error that may distort interpretations of results, estimate the magnitude of error, and inform users of its magnitude.

· Provide evidence of the reliability of assessment scores, based on the State’s student population and reported subpopulations (specify sources of data).

· Provide evidence that the scores are reliable for the intended purposes for essentially all students, as indicated by the standard error of measurement across the score continuum (i.e., conditional standard error).

· Provide evidence of the precision of the assessments at cut scores, and consistency of student level classification (specify sources of data).

· Provide evidence of generalizability for all relevant sources, such as variability of groups, internal consistency of item responses, variability among schools, consistency from form to form of the test, and inter-rater consistency in scoring (specify sources of data).

A.3 Ensuring that assessments are designed and implemented to yield valid and consistent test score interpretations within and across years:

· Provide evidence of valid and reliable linking procedures to ensure that the scores derived from the assessments are comparable within year across various test “forms” and across time.

· Provide evidence that the linking design and results are valid for test scores across the achievement continuum.

· Provide evidence that the assessments are designed and scaled to ensure the primary interpretations of the assessment can be fulfilled. For example, if the assessments are used as data sources for growth or value-added models for accountability purposes, evidence should be provided that the scaling and design features would support such uses, such as ensuring appropriate amounts of measurement information throughout the scale, as appropriate.

· Provide evidence, where a vertical or other score scale is used, that the scaling design and procedures lead to valid and reliable score interpretations over the full length of the scale proposed; and evidence is provided that the scale is able to maintain these properties over time (or a description of the proposed procedures is provided).

Providing accessibility to all students, including Multilingual learners and students with disabilities:

Provide a description of how non embedded standalone assistive technology devices that students use on a regular basis can be used during state testing.

· Provide a description of access to translations and definitions, consistent with State policy.

· Provide a description of the construct validity of the available accessibility features with a plan that ensures that the scores of students who have accommodations that do not maintain the construct being assessed are not combined with those of the bulk of students when computing or reporting scores.

· Assessment items must be associated with meta-data that describe any changes that will be made to the content, display, or input method necessary to provide appropriate accommodations to the student.

· Provide the functionality to track/capture a student's use of tool and accessibility features by item.

Meeting all requirements for data privacy and ownership: All assessments must meet federal and State requirements for student privacy, and all data is owned exclusively by the State.

· Provide an assurance of student privacy protection, reflecting compliance with all applicable federal and State laws and requirements.

· Provide an assurance of State ownership of all data, reflecting knowledge of State laws and requirements.

· Provide an assurance that the State will receive all underlying data, in a timely and useable fashion, so it can do further analysis as desired, including, for example, achievement, verification, forensic, and security analyses.

· Provide a description for how data will be managed securely, including, for example, as data is transferred between vendors and the State.

Part B: Yield Valuable Reports on Student Progress and Performance

Focusing on student achievement and progress to readiness: Score reports illustrate a student’s progress on the continuum toward kindergarten readiness, age by age, and domain by domain. Reports stress the most important content, skills, and processes, and how the assessment focuses on them, to show whether or not students are on track to readiness.

· Provide a list of reports, and for each report, a sample that shows, at a minimum:

· Scores and sub scores that will be reported with emphasis on the most important content, skills, and processes for each domain.

· Explanations of results that are instructionally valuable and easily understood by essentially all audiences.

· Results expressed in terms of performance standards (i.e., proficiency “cut scores”), not just scale scores or percentiles; and

· Progress on the continuum toward kindergarten readiness, which can be expressed by whether a student has sufficiently mastered the current age or domain content and is therefore prepared for the next level.

(Note: Not all reporting information need be numerical; for example, actual student work on a released item could be presented, along with the rubric for the item and a discussion of common errors.)

· Provide evidence that the reporting structure can be supported by the assessment design, including data confirming that test blueprints include a sufficient number of items for each reporting category, so that scores and sub scores lead to the intended interpretations and minimize the possibility of misinterpretation.

Providing timely data that inform instruction: Reports are instructionally valuable, easy to understand by all audiences, and delivered in time to provide useful, actionable data to students, families, and teachers.

· Provide a timeline and other evidence to show when assessment results will be available for each report.

· Provide a description of the process and technology that will be used to issue reports in as timely a manner as possible.

· Provide evidence, including results of user testing, to demonstrate the utility of the reports for each intended audience.

Part C: Adhere to Best Practices in Test Administration

2.12

Maintaining necessary standardization and ensuring test security: In order to ensure the validity, fairness, and integrity of State test results, the assessment systems maintain the security of the items and tests as well as the answer documents and related ancillary materials that result from test administrations.

· Provide a comprehensive security plan with auditable policies and procedures for test development, administration, score reporting, data management, and detection of irregularities consistent with NCES and CCSSO recommendations for, at a minimum:

· Training for all personnel – both test developers and administrators.

· Secure management of assessments and assessment data, so that no individual gains access to unauthorized information.

· Test administration and environment; and

· Methods used to detect testing irregularities before, during, and after testing, and steps to address them.

· Provide a description of how security safeguards have been tested and validated for computer-based tests and for paper-and-pencil tests, as relevant.

Part D: Meet State-Specific Criteria

2.13

Ensuring item interoperability

· Provide evidence showing the interoperability of computer-administered items. Computer administered items must be consistent in all ways with the specifications laid out in the Assessment Interoperability Framework (2012) developed by the Common Education Data Standards (CEDS) project, so that tests and items can be easily ported from one technology platform to another.

Section 3: IT Related Questions Instructions: Vendor should provide the Question #, the Question, and a narrative response to that question for each of the following IT Related Questions.

· How will the Respondent handle significant increases in web traffic?

· Is this solution a manual or automatic adjustment?

· With either solution, what is the time period expected for implementing these adjustments?

Section 4: Table of Contents Instructions: After responding to all questions above, vendor should add page numbers for each question/response.

Section 1: Scope of Work Questions

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

1.9

1.10

1.11

1.12

1.13

1.14

1.15

1.16

1.17

1.18

1.19

1.20

1.21

1.22

1.23

1.24

1.25

1.26

1.27

1.28

1.29

1.30

1.31

1.32

1.33

1.34

1.35

1.36

1.37

1.38

Section 2: Assessment Criteria and Evidence Questions

Part A. Meet Overall Assessment Goals and Ensure Technical Quality

2.1

2.2

2.3

2.4

2.5

2.6

2.7

2.8

2.9

Part B: Yield Valuable Reports on Student Progress and Performance

2.10

2.11

Part C: Adhere to Best Practices in Test Administration

2.12

Part D: Meet State-Specific Criteria

2.13

Section 3: IT Related Questions

3.1

3.2

3.3

3.4

3.5

3.6

3.7

3.8

3.9

3.10

3.11