RFP-DAA-2017000223

STUDENT ASSESSMENTS

SECTION 4 – FORMAT FOR RESPONSES

The purpose of this section is to specify a format for responses, provide a guide to assure all required information is provided, and to facilitate review of proposals.

The Offeror must be registered on VENDOR SELF SERVICE (ColoradoVSS) in order to be considered responsive at the time of proposal submission.

Private contractors and not-for-profit firms and institutions, that meet the minimum requirements, with experience in developing, producing, administering, scoring and reporting large-scale assessments are eligible to respond to this

RFP.

A. Executive Summary. The Executive Summary must be factual and brief, summarize the Offeror’s overall capabilities, and cover the core aspects of the project, condensing and highlighting the Offeror’s approach and contents of the proposal. The summary should provide the reader with an overall understanding of the proposal and Offeror’s approach within six (6) pages.

B. Narrative. The Statement of Work details the deliverables and services required of the Successful Offeror. It is the Offeror’s responsibility to fully understand the Statement of Work and to project the scope through the potential life of the contract, including any extensions. The SOW is outlined by responsibility, requirement and specification. In their narratives, Offerors must specifically identify the requirement or specification (ex. A. 2.2.Time Allocation of Key Personnel and Services and A.3.1.a Preliminary Program Plan) and submit a complete response to, or where appropriate, agreement to, each requirement and specification. The narrative must follow the order presented in the RFP. Attachment 2: Statement of Work Checklist should be used to guide the writing of the narrative. In addition, it must be completed and submitted with the proposal to facilitate the review for responsiveness. Offerors are encouraged to provide detailed, yet concise, responses in order to allow the State to fully evaluate the Offeror’s capabilities.

It is the Offeror’s responsibility to complete each response section completely and in detail. No assumptions should be made that the State is aware of any Offeror’s capabilities, staffing, prior experience, past performance or any other required information. The evaluation will be based on the information submitted with the proposal according to the terms of the RFP.

Page Limit.

CDE discourages lengthy proposals. Proposals should provide a straightforward, concise description of the Offeror’s capabilities to satisfy the requirements and specifications of this RFP. Emphasis should be on completeness and clarity of content. Unnecessarily elaborate proposals are not desired. CDE strongly encourages the narrative response to this solicitation not to exceed 200 pages, excluding the Cost proposal, appendices, exhibits and attachments. Indexes, tables of contents, lists of figures/tables, and glossary of terms will not be counted toward the overall page count. Allowance will be made for tabular or graphical presentations and screen prints, whether incorporated in the text of the technical description or attached as separate exhibits. Textual explanations of screen prints or graphic materials, standard commercial brochures or descriptions, or other standard product documentation that are attached in appendices or exhibits will not be counted against page limitation. However, evaluators cannot be expected to comprehend all material in exhibits whose content and relevance to the proposal description are not clearly integrated into the technical discussion.

Font size for basic narrative descriptions must be no smaller than 11 characters per inch.

Offerors shall not reference dollar amounts within the narrative proposal. All cost and pricing information must be submitted separately per Section 5 of the RFP.

C. Cost Proposal. See Section 5

D. Additional Response requirements

1. Submit the RFP Cover Sheet (first page of RFP), signed in ink by the Offeror or an Officer of the Offeror who is legally authorized to bind the Offeror to the proposal.

2. Financial Resources. The Offeror must also have adequate financial resources for performance, or have the ability to obtain such resources as required during performance.

a. Provide a positive statement that your organization is willing to, and able to, provide the

CDE with the services described in this RFP.

b. Provide a statement that your organization has the financial strength to maintain the contract if awarded. The CDE may request true copies of your most recent audited annual financial statements. DO NOT submit financial statements with your proposal.

If requested this information may be used to assist CDE in making its determination of Offeror’s responsibility in accordance with C.R.S. §24-103-401. The statements must have been prepared by a Certified Public Accountant and meet the Generally Accepted Accounting Principle (GAAP) standards.

Note: the Offeror may request the financial information provided be granted confidential status in accordance with Section 1.14, Proprietary/Confidential Information.

4. Complete and submit EXHIBIT 2, Contractor Non-Conflict of Interest Statement and

EXHIBIT 3, Offeror Disclosure Statement.

E. Packaging Requirements. Proposals must be packaged and sealed in six parts:

1. NARRATIVE: One (1) original hard copy labeled NARRATIVE ORIGINAL, One (1) hardcopy of Narrative labeled COPY, and twenty-five (25) electronic copies on flash drives in searchable pdf format of the narrative non-confidential proposal labeled NARRATIVE 1-25.

One copy of the State of Colorado Request for Proposal – Cover Sheet form MUST be completed and signed in blue ink by a person who is legally authorized to bind the Offeror to the proposal.

The signed form is to be included with the proposal that is marked as ORIGINAL.

One copy of the Criteria for Responsiveness Certification Form found in Attachment 1 of this RFP must be included with the proposal marked ORIGINAL and included on each of the non-confidential flash drives.

One copy of the Statement of Work Checklist found in Attachment 2 of this RFP must be included with the proposal that is marked ORIGINAL and included on each of the non-confidential flash drives.

2. CONFIDENTIAL MATERIALS: If (additional) approved confidential materials are included, one

(1) original hardcopy labeled CONFIDENTIAL ORIGINAL, one (1) hardcopy copy of the labeled CONFIDENTIAL COPY and twenty-five (25) electronic copies on flash drives in searchable pdf format of the confidential and proprietary materials labeled CONFIDENTIAL 1-25. Please keep in mind that it is CDE’s preference for there to be no confidential materials other than sample items submitted as part of responses to this RFP. Refer to Section 1.N for more information related to confidential materials.

3. COST PROPOSAL: One (1) hardcopy original labeled COST ORIGINAL, one (1) hardcopy copy labeled COST COPY and twenty-five (25) fifteen (15) electronic copies on flash drives in searchable pdf format of the cost proposal labeled COST 1-15.

One (1) original of Exhibit 2 (Contractor Non-Conflict of Interest Statement) and Exhibit 3

(Offeror Disclosure Statement) and acknowledgements labeled CERTIFICATIONS

4. CONTRACT NEGOTIATION MATERIALS: Two (2) flash drives with the narrative and cost proposals in Microsoft Word format (tables, charts, timetables/schedules may be in Microsoft Excel) labeled CONTRACT NEGOTIATION MATERIALS.

Each package of the proposal(s) must be clearly labeled with the RFP Title, the Offeror’s name, the proposal due date and time, and the wording: “SEALED PROPOSAL DO NOT OPEN”.

Prices and terms of the proposal(s) as stated must be valid for the length of any resulting contract(s) and subsequent extensions and or amendments.

All costs incurred in responding to this RFP shall be borne by the Offeror.

Proposals that are determined to vary from these requirements may not be accepted.

All participating Offerors, by their signature, shall agree to comply with all of the conditions, requirements and instructions of this RFP as stated or implied herein. Should the State omit anything from this document which is necessary to provide a clear understanding of the work, or should it appear that various instructions are in conflict; the Offeror shall inquire and secure written instructions from the Purchasing Agent by the written inquiry deadline set forth in this RFP.

The State respects the environment and desires to do business with Offerors that share this concern. The Department desires and encourages that proposals be submitted on recycled paper, printed on both sides.

While the appearance of proposals and professional presentation is important, the use of non-recyclable or non-recycled glossy paper is discouraged.

Telegraphic or electronic Vendor Self Service (COLORADOVSS) (Fax, Western Union, Telex, e-mail, etc.) cannot be accepted directly in the Purchasing Office as a sealed bid for the original proposal. Proposals must be submitted and sealed in a package per packaging requirements (Section 4-E) with an appropriate label affixed.

The label must show the following information clearly on the outside of the sealed package:

• [Offeror’s Name]

• RFP2017000223 State Assessments

• Proposal Due: April 28, 2017 (noon MT)

• Offeror’s Email Address:

Send or delivery submissions to: Colorado Department of Education

Attn: Eva Reynolds 201 E. Colfax Ave., Room 409 Denver, CO 80203

End of Section 4

SECTION 5 - Pricing (Cost) Proposal

The evaluation process is designed to award this procurement to the Offeror whose proposal best meets the requirements of this RFP, and is most advantageous to the State, not necessarily to the Offeror with the lowest cost.

However, Offerors are encouraged to submit proposals that are consistent with state government efforts to conserve state resources.

A. Pricing Proposal Content. Offerors must specifically identify all costs including expenses to be charged for performing the services necessary to accomplish the objectives of the contract. Pricing must be inclusive of all Offeror staff costs, administrative support costs, materials, office supplies, cell phones, parking fees, meals, lodging, rents, mileage, travel expenses, training, after hours or weekend time, insurance, use of subcontractors, overhead, profit, and costs for all other items consumed, utilized, and/or required by Offeror’s staff or subcontractor’s staff. Unless otherwise specified, all hardware and software deemed necessary by the Offeror shall be included in the proposed costs. Items or costs required to provide the services and deliverables as proposed not identified in the Offeror’s pricing proposal will be the sole responsibility of the Offeror.

The budget should be sufficiently detailed to clearly delineate how costs are reasonable for each required service, activity and deliverable, and the budget should reflect cost efficiency.

Costs for subcontractors are to be broken out separately.

B. Cost Worksheets. CDE’s goal is to have clear proposals that allow the evaluators to view comparable information. In addition to the budget called for above, the Offeror shall include:

• Cost worksheets found in Attachment 5 must be completed.

• Additional cost portrayals in spread sheets or tables may also be submitted to fully document the Cost Proposal. Cost portrayals that detail costs by activity, service and deliverable by component and content area are especially desirable. (Additional breakouts may be required during contract negotiations.)

C. Payment Plan. Offerors must submit a proposed payment plan relating payments to the specific milestones and deliverables for the project. It is expected that the payment schedule for this contract will be four quarterly and one final payment for the services performed and deliverables provided during each period. The fiscal year for the State of Colorado runs from July 1 to June 30. The last invoice for each fiscal year must be received by June 15. The final invoice for each assessment cycle must be provided by September 15. The payment plan is subject to modification and approval by the State.

End of Section 5

SECTION 6 – PROPOSAL EVALUATION

A comprehensive, fair and impartial evaluation of each proposal received will be conducted, in accordance with the State Procurement Code (C.R.S. §24-103-203) and the related State Procurement Rules.

All responsive proposals received by the deadline will be evaluated on best value (as defined by CDE) by representatives of CDE and its qualified designees. The evaluation will be conducted in four stages:

Stage 1: Responsiveness to RFP (Pass/Fail) Stage 2: Narrative Evaluation Stage 3: Demonstrations & Interviews Stage 4: Cost Proposal Evaluation

Failure of the Offeror to provide any information requested in this RFP may result in disqualification of the proposal. The Offeror must be registered on VENDOR SELF SERVICE (COLORADOVSS) in order to be considered responsive at the time of proposal submission. Offerors not meeting the requirements identified in this RFP shall be ineligible for further consideration. The State may conduct discussions with Offerors in the acceptable range for the purpose of promoting understanding of the State’s requirements and the Offeror’s proposal, clarifying requirements, and making adjustments in services to be performed and in prices. Changes to proposals, if permitted, will be requested by the State in writing from Offerors.

If proposal revisions are permitted after demonstrations and interviews, a date will be established in writing by the State for submission of best and final offers. Offerors will not be provided an opportunity for comprehensive proposal revisions.

Stage 1. Responsiveness to RFP. Eight requirements for responsiveness exist for each Offeror’s proposal.

In any component, the Offeror may submit a proposal with additional alternatives to the requirements and specifications to those stated in this RFP. However, CDE reserves the right to deem any Offeror that does not – at a minimum – meet the eight requirements stated below at various points in the text of this document as “non-responsive”.

Criterion 1 for Responsiveness: Intent to Respond by Deadline. A proposal received by CDE must be prefaced by a letter required in Section 1 B.4 and meet the timelines set forth in Table 2 in Section 1 B.2 in the Significant Dates for Offerors. CDE reserves the right to deem any proposal not meeting this criterion as “nonresponsive” and to terminate further review of the proposal.

Criterion 2: Contract Agreement: The Offeror either agrees to the model contract language or submitted suggested revised language by the designated time. Any concerns regarding other provisions in the policy detailed in the sample contract must have been submitted to Eva Reynolds by April 7, 2017. CDE reserves the right to deem any proposal not meeting this criterion as “nonresponsive” and to terminate further review of the proposal.

Criterion 3 for Responsiveness: Organization and Packaging. Proposals must be formatted

(i.e., proposals include responses to all of the requirements and specifications for each relevant component) and packaged consistent with the requirements outlined in Section 4. CDE reserves the right to deem any proposal not meeting this criterion as “nonresponsive” and to terminate further review of the proposal.

Criterion 4 for Responsiveness: On Time. The proposal was received by CDE according to the timelines set forth in Table 2 in Section 1 B.2 in the Significant Dates for Offerors. CDE reserves the right to deem any proposal not meeting this criterion as “nonresponsive” and to terminate further review of the proposal.

Criterion 5 for Responsiveness: Online Test Engine. The Offeror has a working online test engine in place that meets CDE’s Statement of Work requirements at the time of proposal submission. Proposals not meeting this criterion shall be deemed “nonresponsive” and will not be evaluated further by the Reviewers.

Criterion 6 for Responsiveness: Automated Scoring System. The Offeror has a working Automated Scoring System in place that meets CDE’s Statement of Work requirements at the time of proposal submission. Proposals not meeting this criterion shall be deemed “nonresponsive” and will not be evaluated further by the Reviewers.

Criterion 7 for Responsiveness: Live Demonstration/Interview. If the Offeror advances to Stage 3, the Offeror agrees to be in Colorado to participate in live demonstrations and interviews on the requested date(s) at the location(s) specified by CDE. Proposals not meeting this criterion shall be deemed “nonresponsive” and will not be evaluated further by the Reviewers.

Criterion 8 for Responsiveness: Meets Mandatory Minimum Requirements. The Offeror meets the Mandatory Minimum Requirements as specified in the Introduction in Section 3: Scope of Work and completed all three pages of the Criteria for Responsiveness Certification Form found in Attachment 1.

Stage 2 Narrative Evaluation. The sole objective of the review committee will be to recommend the

Offeror(s) whose proposal is most responsive and advantageous to the State’s needs while within available monetary resources. The specifications within this RFP represent the minimum performance necessary for response. The State reserves the right to have the selected Offeror conduct a proof of concept to ensure that the Offeror has the capability to conduct the work outlined in this solicitation.

At a minimum narratives will be evaluated based on the requirements and specifications found in the following responsibilities:

General Requirements Corporate Capacity and Personnel, including References Program Management Assessment Development Assessment Administration Web-Based Online Test Delivery System Scanning and Scoring Reporting of Data Files and Assessment Results Psychometric, Research, and Technical Activities Training and Support Customer Service

Stage 3 Demonstrations and Interviews. Any Offeror that is a finalist at Stage 3 must be prepared to be on-site in the Denver, CO metropolitan area (CDE or a local school district) in mid-May 2017, or a later date as determined by CDE. At that time, the Offeror finalists shall provide a series of live demonstrations for evaluation as well as make available key staff for interviews.

During the evaluation process, the evaluation committee may, at its discretion, request any one or all Offerors to provide demonstrations or make oral presentations or answer questions about their proposals. Not all Offerors may be asked to make such oral presentations. Offerors should not assume that they will have an opportunity for demonstrations or oral presentations, so they should submit their most favorable proposals in their initial written proposal. If award is not made on receipt of initial proposals, Offerors in the competitive range (those most responsive to the requirements and reasonably likely to be selected for award) may be provided an opportunity to demonstrate and make an oral presentation. The competitive range determination will be based on the written proposals, so Offerors are cautioned to ensure that their proposals adequately convey the soundness of their approach and understanding of the requirements and specifications. Any presentations that are held may consist of an Offeror demonstration or briefing concerning its approach, as well as the Offeror shall address any clarifications and deficiency items that may have been identified in advance by the State. The participating Offeror’s team must include the executive team contact person identified in the proposal (i.e., the executive team member who will be responsible for actual delivery of the program) and the proposed program director or the program manager(s). Based on demonstrations requested and questions asked, additional Offeror personnel intended to be active, on-going participants in the project may be requested to attend.

Stage 4 Cost Proposal Evaluation. Evaluation of cost proposals are not completed until after Stage 3.

The State may conduct discussions with Offerors in the acceptable range for the purpose of promoting understanding of the State’s requirements and the Offeror’s proposal, to clarify requirements, and make adjustments in services to be performed, and in prices. Best and final offers may, in the State’s discretion, be requested in accordance with Section 1.K. of this RFP Changes to proposals, if permitted by the State, will be requested by the State in writing from Offerors.

While a numerical rating system will be used to assist the evaluation committee in selecting the competitive range and making the award decision, the award decision ultimately is a business judgment that will reflect an integrated assessment of the relative merits of the proposals using the factors and their relative importance as disclosed in the RFP. At the conclusion of the four stages, the proposal deemed most advantageous to the State will be formally recommended to the Department of Personnel & Administration, Division of Finance and Procurement, State Purchasing Office for award.

AWARD METHODOLOGY

As stated in Section 6, the Evaluation Committee will evaluate the proposals and demonstrations/interviews, then identify the proposal(s) most advantageous to the State. The evaluation will result in a recommendation for an award(s) of a contract under this RFP. The Evaluation Committee will submit their recommendation to the State Purchasing Office (SPO) for review and final acceptance. The SPO will issue “Notice of Intent to Make an Award” letter(s) to the selected Offeror.

The Offeror shall further agree to cooperate with CDE and expedite the contracting process upon notice of award.

Should the contract not be completed and agreed to by both parties within 30 calendar days following the issuance of a draft contract to the Successful Offeror for review, through no fault of the Department’s, the Department, at its sole discretion, may elect to cancel the existing award announcement and make an award to the next most advantageous Offeror.

The resulting contract is subject to and contingent upon the continuing availability of federal funds for the purpose hereof. The Offeror recognizes that it is to be paid, reimbursed or otherwise compensated with funds provided to CDE for the purpose of contracting for the services provided herein. The Offeror expressly understands and agrees that all its rights, demands and payment to compensation arising under the resulting contract are contingent upon receipt of such funds by CDE. In the event that CDE does not receive such funds or any part thereof, CDE may immediately terminate this contract without liability, including liability for termination cost.

End of Section 6

GLOSSARY

Many abbreviations, acronyms and terms are used throughout this document.

AAAS American Association for the Advancement of Science

ADE Automated Data Exchange

APIP Accessible Portable Item Profile

BAFO Best and Final Offer

CAP4K Colorado Achievement Plan for Kids

CAS Colorado Academic Standards

CCHE Colorado Commission on Higher Education

CCSS Common Core State Standards

CDE Colorado Department of Education

CMAS

CoAlt

Contractor

Colorado Measures of Academic Success

Colorado Alternate Assessment

The individual, company, corporation, or firm to whom the RFP is awarded. In the RFP, Subcontractors are included when Contractor is used.

COPPA Children’s Online Privacy Protection Act

CORE Colorado’s Operations Resource Engine which is a web sited designed to notify interested vendors of the State of Colorado’s intent to purchase goods or services competitively. These notifications are termed “solicitations”.

CSLA

DAC

Colorado Spanish Language Arts

District Assessment Coordinator

Department Colorado Department of Education

DIF Differential Item Functioning

DTC

ED

District Technology Coordinator

U. S. Department of Education

ELA English Language Arts

ESEA Elementary and Secondary Education Act

ESSA

FERPA

Every Student Succeeds Act

Family Educational Rights and Privacy Act

FTE Full Time Equivalent positions

FTP File Transfer Protocol

GLE Grade Level Expectation

HB 08-1168 House Bill 08-1168, Financial Literacy

ICAP Individual Career and Academic Plan

IDEA Individuals with Disabilities Act

IRC Item Review Committee

IRT Item Response Theory

LDS Longitudinal Data System, also known as SLDS

MT

NAEP

Mountain Time

National Assessment of Educational Programs

Offeror The individual, company, corporation or firm formally submitting a response to this Request for Proposals. Once the RFP is awarded, the terms “Offeror”, “Successful Offeror” and “Contractor” shall be understood to mean the individual, company, corporation, or firm to whom the contract is awarded, unless otherwise specifically defined otherwise (i.e., “Verification Contractor”).

OMR Optical Mark Recognition

PARCC

PARCC

Consortium

PARCC

content

PARCC

Governing Board

Partnership for Assessment of Readiness for College and Careers

A collective of states that have worked together to develop a common set of assessments in English Language Arts/Literacy (ELA) and Mathematics in response to the awarding of a Comprehensive Assessment Systems Grant funded in fiscal year 2010 under the auspices of the U.S. Department of Education‘s 2010 Race to the Top Assessment Program to develop assessment systems that provide valid, reliable, and fair performance results for individuals and groups of students against a common set of college and career readiness standards.

Content created for the PARCC assessment system, both existing content created under a current contract and future content that will be created under the new contract

The Governing Board of PARCC established the governance structure for the consortium and it is responsible for the major policy and operational decisions about the design, administration and scoring of the PARCC test. The current Governing Board is made up of state education leaders from the following states: Colorado, Illinois, Maryland, New Jersey, New Mexico, Rhode Island and the District of Columbia.

PPT Paper Pencil Test

Proposal A written offer to provide services in accordance with the requirements, terms and conditions of the Request for Proposals. Any exceptions to the Colorado Department of Education’s requirements, terms and conditions shall be clearly stated by the Offeror in its proposal

QA Quality assurance

QC Quality control

Quality Control Contractor

Third party with whom CDE intends to contract for quality control of assessments

RFP Request for Proposals

SAC School Assessment Coordinator

SASID State student ID

SB 09-212 Senate Bill 09-212, also known as the “Preschool to Postsecondary Education Alignment Act,”

“CAP4K”

SB 09-163 Senate Bill 09-163, also known as the “Education Accountability Act of 2009”

SB 10-191 Senate Bill 10-191, “Concerning Ensuring Quality Instruction through Educator Effectiveness”

SBAC Smarter Balanced Assessment Consortium

SBD Student Biographical Data

SBE Colorado State Board of Education, also known as the State Board

SchoolVIE W.org

Colorado Department Of Education website that provides users with tools to analyze academic data

SLDS Statewide Longitudinal Data System, also known as LDS

SPT

State

Supported Performance Task

State of Colorado

State Board Colorado State Board of Education, also known as SBE

Successful Offeror

The individual, company, corporation, or firm to whom the contract is awarded

TAC Technical Advisory Committee

APPENDICES

Appendix A: Table of Contents from the Current Procedures Manual

Note: The Procedures Manual for the Colorado Measures of Academic Success and Colorado Alternate Assessment Program can be found at the following link: http://www.cde.state.co.us/assessment/2017coproceduresmanual.

http://www.cde.state.co.us/assessment/2017coproceduresmanual

Appendix B: Sample Technical Report Table of Contents

SAMPLE TECHNICAL REPORT TABLE OF CONTENTS

List of Tables

List of Figures

List of Abbreviations

Executive Summary

Background

Administration

Student Performance

Evidence Supporting the Validity of Inferences from the Assessment Scores

Chapter 1: Introduction

1.1 History of the Student Assessments

1.2 Brief Description of the Student Assessments

1.3 Summary of the Student Assessments

1.4 The Colorado Academic Standards

1.5 Testing, Reporting, and Accountability

1.6 Purpose of the Technical Report

Chapter 2: Test Development

2.1 Introduction

2.2 Eligible Content

2.3 Claims

2.4 Design of the Student Assessments

2.5 Test Blueprints

2.6 Test Specifications

2.5 Development of Test Items

2.5a Item Specifications

2.5 b Prototypes and Cognitive labs for new item types

2.5c Passage and storyboard review

2.5d Item writers

2.5e Item writer training

2.5f Item review

2.5g Bias and Sensitivity Review

2.5h Accessibility Review

2.5i Alignment to standards methodology

2.6 Field testing (initial and embedded)

2.7 Item Bank

2.8 Test Form Assembly (Form development)

2.9 Online Forms Construction

2.10 Quality Control for Test Construction

2.11 Summary

Chapter 3: Achievement-Level Setting

3.1 Introduction

3.2 Goal of the Standard Setting

3.3 Staff and Participants

3.4 Development of Achievement- Level Descriptors

3.5 Overview of Standard-Setting Activities

3.6 Specific Standard-Setting Activities

3.7 Session Results by Panel and Round

3.8 Results of Participant Evaluations

Chapter 4: Item Analysis

4.1 Introduction

4.2 Analysis of Forms for Student Assessment

4.3 Item Statistics

4.4 Calibration of accommodated items

4.5 Summary

Chapter 5: Accommodations

5.1 Introduction

5.2 Accommodations eligibility

5.3 Accessibility features

5.4 Number of students using accommodations by accommodation and subgroup

5.6 Item performance for students using accommodations

Chapter 6: Test Administration

6.1 Introduction

6.2 Test administration training

6.3 Test Administration

6.4 Online Assessment Administration

6.5 Test Security

6.6 Materials Handling and Return

6.7 Summary

Chapter 7: Scoring, and Quality Control Procedures

7.1 Introduction

7.2 Quality Control Overview

7.3 Preparation and Materials Check-In

7.4 Materials Scanning

7.5 Quality Control in Report Production

7.6 Scoring Requiring Human Judgment

7.7 Processing Documents and Image Quality Control

7.8 Range-Finding and Development of Scoring Materials

7.9 Rubric validation

7.10 Anchor Review for new Items

7.11 Project Staffing

7.12 Training.

7.13 Qualification to Score.

7.14 Scoring Procedures

7.15 Monitoring for Quality Assurance

7.16 Irregularity Analysis

Chapter 8: Scaling and Equating

8.1 Introduction

8.2 Item Response Theory (IRT)

8.3 Selected IRT Models

8.4 Scaling and Equating

8.4a Model fit analyses

8.4b Item local independence

8.4c calibration evaluation

8.4d anchor item evaluation criteria and results

8.4e p-value comparisons

8.4f Item parameter comparisons

8.4g Scaling constants

8.4h flagged items

8.5 Item sampling for equating

8.6 Student sampling for equating

8.7 Operational item equating procedures

8.8 Placing field test items on operational scale

8.9 Item pool maintenance

8.10 Linking

8.11 Vertical scale design

Chapter 9: Reporting

9.1 Introduction

9.2 Individual Student Report

9.3 State and district summary reports

9.3a Description of scores

9.3b Raw Score

9.3c Scale Score

9.3 d. Performance Levels

9.4 Description of Reports

9.4a Overall proficiency at a glance

9.4b Summary reports

9.4c Subgroup reports

9.5 Appropriate Score uses

9.6 Interpreting Score means

9.7 Cautions for score use

9.7a Understanding measurement error

9.7b Using scores at extreme ends of the distribution

9.7c Interpreting score means

9.7d Using objective/strand-level information

9.7e Program evaluation implications

9.8 Summary

Chapter 10: Summary Statistics

10.1 Introduction

10.2 Descriptive Statistics for Total Raw Scores

10.3 Descriptive Statistics for Total Raw Score by Section

10.4 Descriptive Statistics by Demographic Group

10.5 Descriptive Statistics by accommodation

Chapter 11: Reliability

11.1 Introduction

11.2 Reliability and Measurement Error

11.3 Evidence of Raw-Score Internal Consistency

11.4 Conditional Standard Error Estimates for Scale Scores

11.5 Evidence Supporting Scorer Reliability

11.6 Reliability of Classifications

11.7 Alternate forms reliability estimation

11.8 Internal consistency reliability estimation

11.9 Scoring reliability for constructed response items and written compositions

11.9a Reader agreement

11.9b Score appeals

11.9c Auditing of test administrations and task ratings

Chapter 12: Validity

12.1 Introduction

12.2 Purpose and Intended Uses of Test Scores

12.3 Criterion validity

12.4 Content and Curricular-Related Evidence of Validity

12.5 Construct Validity

12.6 Divergent Validity

12.7 Predictive Validity

12.8 Classification Accuracy

12.9 Validity Evidence Based on the Internal Structure of the Student Assessments

12.10 Additional Validity Evidence for the Student Assessments

12.11 DIF Analysis

12.13 Scoring validity evidence

12.14 Generalization validity evidence

12.15 Extrapolation validity evidence

12.16 Implication validity evidence

12.17 Summary

Glossary of terms

References

Appendices

ATTACHMENTS

Attachment 1: Criteria for Responsiveness Certification Form

The criteria for responsiveness to this RFP are outlined in the table below. Place a checkmark in each box to confirm responsiveness to the applicable criterion. For the Minimum Qualifications table on the following two pages, please list relevant programs and the number of years the Offeror was associated with the program. In the last column, when appropriate, the Offeror should indicate where within the proposal the relevant experiences and skills are further discussed. A person with the authority to make commitments on behalf of the Offeror shall provide a signature for each assurance or commitment indicating the Offeror’s acknowledgement that the Offeror meets each of the minimum qualifications in that section.

Name of Offeror(s) _____________________________________________________________________

Criteria for Responsiveness Description of Criterion

Criterion 1:

Intent to Respond

The Intent to Respond letter was submitted and received by CDE by the designated time.

Criterion 2:

Contract Agreement

The Offeror either agrees to the model contract language or submitted suggested revised language by the designated time.

Criterion 3:

Organization and Packaging

The proposal includes responses to all of the requirements and specifications listed under each component (as detailed in Attachment 2: Statement of Work Checklist).

Criterion 4:

On Time

The proposal was received by CDE by the designated time.

Criterion 5:

Online Testing Engine

The Offeror has an online test engine in place that has been used for at least 3 years for large scale, high stakes statewide assessments and meets CDE’s Statement of Work requirements at the time of proposal submission.

Criterion 6:

Automated Scoring System

The Offeror has a working Automated Scoring System in place that has been used for at least 2 years for large scale, high stakes statewide assessments and meets CDE’s Statement of Work requirements at the time of proposal submission.

Criterion 7:

Live Demonstration/Interview

If the Offeror advances to Stage 3, the Offeror agrees to be in Denver, CO to participate in live demonstrations and interviews on the dates and at the location specified by CDE. The Offeror agrees to demonstrate any of the relevant tools and services it is proposing, including, but not limited to, its item authoring and banking systems, and its test management and test administration systems. The Offeror will be required to demonstrate both its own sample items and the CDE-provided sample items on its system. Necessary files will be sent to those who submit an Intent to Bid.

Criterion 8:

Meets Mandatory Minimum Requirements

The Offeror meets the Mandatory Minimum Requirements as outlined in the following pages.

I hereby certify that I am authorized to act for this/these Offeror(s) in matters pertaining to the criteria for responsiveness for this RFP and that the above statements are true and correct to the best of my knowledge.

Printed Name ___________________________ Title:________________________________________

Signature___________________________________________ Date:__________________________

(Page 1 0f 3)

Criteria for Responsiveness Certification Form (Page 2 of 3)

Minimum Qualifications Experience: Program(s) Number of Years Proposal References A minimum of ten (10) years of experience in developing, producing, administering, scoring and reporting of large-scale, high-stakes, statewide general and alternate assessments.

o Experience with development of short and extended constructed response items o Experience developing accommodated materials, including but not limited to transadapted Spanish forms, braille (with at least one year of UEB braille), large print, English and Spanish oral scripts o Experience with assessment of complex performances o Experience with scoring rubrics and performance scoring procedures

A minimum of five (5) years of experience with developing, producing administering, scoring and reporting of large-scale, high-stakes computer-based assessments.

o Experience with developing, administering, scoring and reporting of innovative, interactive technology-enhanced items.

o Online test administration system must allow for caching, as well as provide for downloadable apps.

o Experience with establishing security protocols for online assessments.

A minimum of three (3) years of experience with developing, producing and administering simulation-based science assessments for large-scale, high-stakes, state-wide computer-based assessments.

Demonstrated experience developing authentic Spanish language arts assessments aligned to CCSS in large scale, high stakes, state-wide assessment.

with practical application of automated scoring for constructed responses in large scale, high stakes state-wide assessment.

with use of pattern scoring in large scale, high stakes state-wide assessment.

Demonstrated experience applying psychometric procedures to ensure comparability across different modes of administration in high stakes, state-wide assessment.

Minimum Qualification: Assurances and Commitments Authorized Signature Offeror shall not have been the Contractor for programs that have cancelled testing in the last two years.

Offeror shall not have been the (scoring and/or reporting) Contractor for a program that has had publically-released scores retracted in the last two years.

Many Colorado districts have made significant investments in Chromebooks in the last few years.

Offeror’s systems must be fully functional and secure on Chromebooks. Responses not returned to the Successful Offeror’s system must be backed up on the local device upon student log out or shut down and (easily) retrievable.

A single instance of any application must be sufficient to run all state tests provided by the Successful Offeror in Colorado, whether those be through this contract or other contracts.

Item types developed for this project must adhere to the Question and Test Interoperability (QTI) and Accessible Portable Item Protocol (APIP) standards. If the Offeror develops item types for use in Colorado that extend beyond the current QTI and APIP standards, the Successful Offeror shall share the Colorado-specific item type development standards with the IMS Global Learning Consortium.

Willingness to submit a set of sample items created by Offeror. While story boards are acceptable for simulations and TEIs submitted with the proposal, if invited, the Offeror agrees to demonstrate the QTI-compliant items programmed in XML during the demonstration/interview phase.

Commitment to Colorado data privacy/security requirements. Offerors must commit to meeting FERPA, COPPA and Colorado’s HB 16-1423: Student Data Transparency and Security Act.

Attachment 2: Statement of Work Checklist

A.1. General Requirements

A.1.1. Professional Standards/Best Practices

A.1.2. State and Federal Requirements A.1.3. Communication

A.1.4. Cooperation with Quality

Control Contractor and CDE Staff

A.1.5. Interaction with Verification Contractor

A.1.6. Test Security A.1.7. Travel A.1.8. Educator Meetings A.1.9. Subcontractor Requirements A.1.10. CDE Sign Off on Procedures

A.2. Corporate Capacity and Personnel

A.2.1. Organizational Structure

A.2.2. Time Allocation of Key Personnel and Services

A.2.3. Staff Qualifications and Experiences

A.2.4. Corporate Capabilities and Capacities

A.2.5. Relevant Experience

A.2.6. Risk Management and Quality Assurance

A.3. Program Management

A.3.1. Key Activities and Transfer Dates

A.3.1.a. Preliminary Program Plan A.3.1.b. Project Schedule

A.3.2. Program Management Communication and Reports

A.3.2.a. On-going Communication A.3.2.b. Timeliness of Communication A.3.2.c. Weekly Meetings A.3.2.d. Program Meetings A.3.2.e. Monthly Reports

A.3.2.f. District Assessment Coordinator (DAC) Academy

A.3.2.g. District Assessment

Coordinator (DAC) Management Meeting

A.3.2.h. Program Improvement Plans

A.3.2.i. Quality Control and Sign-Offs

A.3.2.j. Invoices A.3.3. Transition

A.3.4. Home School and Nonpublic School Students

A.4. Assessment Development

A.4.1. Work Plan A.4.2. Content Test Design

A.4.2.a. Background Documents

A.4.2.b. CMAS Item Specifications, Task Models and Style Guides

A.4.2.c. CoAlt Item Types

A.4.2.d. Design and Test Blueprints/ Specifications

A.4.2.e. Samples for RFP A.2. A.4.3. 2021 Test Design

A.4.3.a. Determining Eligible Content (2021 Test Design)

A.4.3.b. Claims and Reporting Categories (2021 Test Design)

A.4.3.c. Frameworks (2021 Test Design)

A.4.3.d. Performance Level Descriptors Policy Level and Grade Level (2021 Test Design)

A.4.3.e. Item Specifications, Task

Models and Style Guides (2021 Test Design)

A.4.3.f. Passage/Source Specifications (2021 Test Design)

A.4.3.g. Prototypes and Cognitive Labs (2021 Test Design)

A.4.3.h. Design and Test Blueprints/

Specifications (2021 Test Design)

A.4.3.i. Samples for RFP (2021 Test Design)

A.4.4. Item Development and Banking System

A.4.4.a. Item Development System A.4.4.b. Item Bank

A.4.4.c. Ingesting Current Items into the Bank

A.3 A.4.5. Item Development Process and Cycle

A.4.5.a. Item Bank Gap Analysis A.4.5.b. Universal Design

A.4.5.c. Number of Items to be

Developed

. A.4.5.d. Passage/Source Selection and

Storyboard Creation

A.4.5.e. Passage and Storyboard Review

A.4.5.f. Item Writer Training and Assignments

A.4.5.g. Item Development Meetings (Educator Meeting)

A.4.5.h. Content and Bias/Sensitivity Review (Educator Meeting)

A.4.5.i. Cousin Items A.4.5.j. Field Testing

A.4.5.k. Data Review (Educator Meeting)

A.4.6. Form Development A.4.6.a. Core Form A.4.6.b. Embedded Field Test Forms A.4.6.c. Breach Form

A.4.7. Accommodated Materials and Administration Procedures

A.4.7.a. Accommodated Procedures A.4.7.b. Large Print

A.4.7.c. Braille A.4.7.d. Spanish A.4.7.e. English Oral Scripts A.4.7.f. Spanish Oral Scripts A.4.7.g. Braille Oral Scripts

A.4.7.h. Certification for District Translators

A.4.7.i. English and Spanish Language Arts Comparability

A.4.8. Document Development A.4.8.a. Procedures Manual

A.4.8.b. CMAS Test Administrator Manuals

A.4.8.c. Certification Forms

A.4.8.d. CMAS Test Booklets and Answer Documents

A.4.8.e. CoAlt Combined Test

Administrator/Student Test Booklets

A.4.8.f. Ancillary Materials

A.5. Assessment Administration A.5.1. Work Plan A.5.2. Test Window

A.5.3. Registration and Materials Ordering

A.5.3.a. Registration A.5.3.b. Personal Needs Profile File A.5.3.c. Pre-identification Labels A.5.3.d. Additional Materials Ordering

A.5.4. Online Test Form Production

A.5.5. Document Production A.5.5.a. Test Administrator Manuals

A.5.5.b. Test Booklets/Answer Documents

A.5.5.c. Breach Forms A.5.5.d. Ancillaries

A.5.5.e. Accommodated/ Accessibility Materials

A.5.6. Secure Distribution, Collection, Storage and Destruction

A.5.6.a. Packaging of Paper-Based Testing Materials

A.5.6.b. Distribution and Collection

A.5.6.c. Materials Storage and Destruction

A.5.6.d. Disaster Recovery A.5.6.e. Verification

A.5.7. Test Monitoring

A.5.7.a. Fidelity to Test

Administration and Security Procedures

A.5.7.b. Reported Violations of

Administration and

A.6. Online Test Management and Test Administration System A.6.1. Work Plan

A.6.2. Transition to Successful

Offeror’s Online Administration Platform

A.6.3. Online Test Administration System

A.6.3.a. Online Test Administration System Characteristics

A.6.3.b. Accommodations and Accessibility Tools

A.6.3.c. Online Tutorials A.6.3.d. Demonstration of Online Test

Management and Test Administration Systems

A.6.4.e. System Security

A.6.4.f. Minimum Hardware

Specifications and Technical Standards

A.6.4. Student and Assessment Data Management System

A.6.5. Test Management System A.6.6. Site Readiness Tools

A.6.7. Testing Support and Application Testing

A.6.8. Data Integration and Collection

A.6.8.a. Data Integration A.6.8.b. System Requirements

A.6.8.c. Data Collection Protection Features

A.6.8.d. Access to Data Collection System

A.6.8.e. Student Verification

A.6.9. System Reliability and Mitigation Experience

A.6.9.a. System Reliability

A.6.9.b. Online Assessment Challenges and Remedies

A.7. Scanning and Scoring

A.7.1. Work Plan

A.7.2. Scanning of Assessment Documents and Scoring of Selected Responses CMAS, including CSLA

A.7.2.a. Scanning of Student Demographics and Responses

A.7.2.b. Capturing of Online Responses

A.7.2.c. Scoring of Selected Response and Technology-Enhanced Items

A.7.3. Constructed Response Scoring Materials

A.7.3.a. Historical Scoring Materials

A.7.3.b. Range Finding and Anchor

Validation Meetings (Educator Meeting)

A.7.3.c. Scoring Decision Guide

A.7.4. Scoring Center Procedures and Scoring Processes for CMAS, including CSLA

A.7.4.a. Scoring Center A.7.4.b. Scoring Staff A.7.4.c. Scorer Training and Qualifying

A.7.4.d. Scoring Confidentiality and

A.7.5. Automated Scoring

A.7.6. Additional Scoring

Considerations for CMAS, including CSLA

A.7.6.a. Reliability and Validity of Scoring Procedures

A.7.6.b. Final Scores A.7.6.c. Documentation of Item Issues

A.7.6.d. Procedure for Responses in Languages Other than English

A.7.6.e. Procedure for Unusual Responses

A.7.6.f. Rescoring

A.7.7. Scoring Processes for

Component C) CoAlt:

Science and Social Studies

A.7.7.a. CoAlt Teacher Scoring A.7.7.b. CoAlt Scoring Guidance A.7.7.c. CoAlt Scoring Monitors

A.7.7.d. Online Score Submission System

A.7.8. Quality Assurance Procedures

A.8. Reporting of Data Files and Assessment Results

A.8.1. Work Plan A.8.2. Data A.8.2.a. Data Files A.8.2.b. Data Documentation A.8.2.c. Data Ownership

A.8.2.d. Student Biographical Data

Review (SBD) and Online Record Corrections

A.8.3. Reporting A.8.3.a. Quality Assurance A.8.3.b. Formatting of Reports A.8.3.c. Individual Student Reports A.8.3.d. State/District/School Reports A.8.3.e. Report Approval A.8.3.f. Reporting Options A.8.3.g. Report Dissemination A.8.3.h. Reporting Errors A.8.3.i. Reprinting A.8.3.j. Student Responses A.8.3.k. Assessment Interpretive Guide

A.8.4. Released/Practice Items, including Scripts and Scoring Guides

A.9. Psychometric and Technical Activities A.9.1. Work Plan

A.9.2. Piloting and Field Testing A.9.2. a. Piloting/Cognitive Labs

A.9.2.b. Psychometric Analysis of

Piloting/Cognitive Labs for New Item Types

A.9.2.c. Field Test – Sampling Design

A.9.2.d. Psychometric Analysis of

Field Test to Support Operational Assessment

A.9.3. Operational Analysis of Items

A.9.4. Initial Scaling and Standard Setting

A.9.4.a. Scaling

A.9.4.b. Performance Standards Setting/

Validation Meeting (Educator Meeting)

A.9.5. Ongoing Scaling and Equating

A.9.6. Linking

A.9.7. Establishing Technical Adequacy

A.9.7.a. Peer Review Requirements A.9.7.b. Alignment Studies

A.9.7.c. Establishing Comparability

A.9.7.d. Technical Advisory Committee

A.9.7.e. Technical Report

A.9.8. Irregularity and Data Forensic Analysis

A.10. Training and Support

A.10.1. Work Plan A.10.2. Face-to-Face Training

A.10.2.a. District Technology Coordinator Training

A.10.2.b. Assessment Administration Training

A.10.2.c. Understanding the Assessment Results Training

A.10.3. Webinars A.10.4. Online Training Support

A.11. Customer Service A.11.1. Work Plan

A.11.2. District Assessment Coordinator Routine Communications/ Notifications

A.11.3. Training of Customer Service Representatives

A.11.4. Telephone, Chat and E-mail Support

A.11.5. Tiered Support for District Technology Coordinators

A.11.6. Issue Resolution System

A.11.7. Records of Interaction with Customers

A.11.8. Customer Service Initiated Calls and Communications

A.11.9. Customer Feedback Survey

A.11.10. DAC Website

A.11.11. Customer Support Access to Data

Attachment 3: Critical Key Activities for the Colorado Assessment Program Create a Separate Table for Initial Contract and Each Renewal

Key Activities Timeline Reference in Statement of

Work

Deliverable Date

Executed Contract or Renewal

June

General Security Procedures Program Management

Annual Project Schedule

1st two weeks of contract.

June 1 all other years

CMAS: Sci/SS CMAS: Math/ELA

CSLA

CoAlt: Sci/SS

Critical Dates and Activities for the Field

CMAS: Sci/SS CMAS: Math/ELA

CSLA

CoAlt: Sci/SS

Orientation Meeting 1st two weeks of contract.

Start-Up Program Meetings Within one month of orientation meeting

CMAS: Sci/SS

CoAlt: Sci/SS

CMAS: Math/ELA

CSLA

Planning Meeting 1 & Notes July Planning Meeting 2 & Notes August Planning Meeting 3 & Notes September Planning Meeting 4 & Notes October Planning Meeting 5 & Notes November Planning Meeting 6 & Notes December Planning Meeting 7 & Notes January Planning Meeting 8 & Notes February Planning Meeting 9 & Notes March Planning Meeting 10 & Notes April Planning Meeting 11 & Notes May Planning Meeting 12 & Notes June Upper Management Meeting

Monthly Reports 3rd business day of the month

CMAS: Sci/SS CMAS: Math/ELA

CSLA

CoAlt: Sci/SS

DAC Academy August DAC Management 1 Fall DAC Management 2 Spring

Program Improvement Plan – Development

Within 1 month of phase completion

CMAS: Sci/SS CMAS: Math/ELA

CSLA

CoAlt: Sci/SS

Program Improvement Plan - Production

Within 1 month of phase completion

CMAS: Sci/SS CMAS: Math/ELA

CSLA

Program Improvement Plan – Shipping and Receipt

Within 1 month of phase completion

CMAS: Sci/SS CMAS: Math/ELA

CSLA

CoAlt: Sci/SS

Program Improvement Plan – Administration

Within 1 month of phase completion

CMAS: Sci/SS CMAS: Math/ELA

CSLA

CoAlt: Sci/SS

Program Improvement Plan – Scanning

Within 1 month of phase…