RFP-DAA-2017000223 Inquiries and Answers.pdf

OFFEROR’S INQUIRIES and

COLORADO DEPARTMENT OF EDUCATION RESPONSES

RFP-DAA-2017000223

Student Assessments

- 1 -

Note: All CDE answers are printed in BOLD

Vendor inquires have been reproduced as received.

Page

Component, Responsibility, Requirement Code

(Title)

Questions & Answers

A.4.6.a.

A.4.6.b.

RFP Modification 1 page 78 now includes:

Note: Items for the high school social studies core forms have been selected.

RFP Modification 1 page 79 now includes:

Note: Items for the high school social studies embedded field test forms have been selected.

52 A.4.2.a.

RFP Modification 1 page 52 lines 7-9 now reads:

Listed under “Extended Evidence outcomes (Adopted August 3, 2011)”, the standards for each content area with the EEOs:

Introduction Page 3:

“The Successful Offeror shall be responsible for the development and administration activities necessary for a successful operational administration of the science and social studies, including high school, assessments starting in school year 2017-2018.”

Page 5: “*Grades 4 and 7 social studies assessments are administered to approximately 1/3 of students per year. High school is currently scheduled to be administered as a census test in 2018 and then to approximately a third of

Please confirm if the Colorado Department of Education (CDE) planning to reinstate the High School Social Studies test in 2018-2019? If so, will a standard setting or standards validation be required to set the performance levels needed for reporting?

Answer: Offerors shall plan on the high school social studies test to be administered to all 11th grade students in 2017-2018 and 1/3 of schools/students each year thereafter. Offerors shall include a plan for a standards validation based on the recommended cut scores from 2015.

- 2 -

Colorado students and schools each year thereafter.”

A.9.7.d. Honorariums

Please confirm how much honorarium will be paid to TAC members, $1750 per day (described in the narrative or $1500 per day (indicated in Table 22)?

Answer: $1,750 per day

See RFP Modification 1 page 137.

Cost Table 2 Science and Social Studies, the request of “Estimated cost for adding or subtracting a grade”

Is it the intent of CDE to provide this for each grade level tested separately in order to accurately indicate costs for the given test development cost and administration/scoring costs associated with the student volume for the given grade level?

Answer: Yes. Offerors should assume relatively similar student volumes across grades within a content area with the exception of high school social studies in 2018, which is scheduled to be administered census-wide.

Introduction, B.

Colorado Measures of Academic Success (CMAS): Mathematics and English Language Arts, including Colorado Spanish Language Arts Assessment (CSLA).

How many anchor items by grade, content area, and item type will be included in the 2019 assessment?

Answer: Offerors shall propose what they deem is a technically sound number of anchor items to be included by grade, content area and item type for CMAS: Math and English Language Arts, including

CSLA.

Introduction, B.

Colorado Measures of Academic Success (CMAS): Mathematics and English Language Arts, including Colorado Spanish Language Arts Assessment (CSLA).

The RFP indicates that the price for 2019 anchor items will be determined at a later date. Please clarify if offerors should include costs for licensing.

Answer: Yes, Offerors shall include costs for licensing.

A.9.5. Ongoing Scaling and Equating

The text states, “the Successful Offeror shall work with CDE to implement a scaling procedure that will result in scale scores that will remain comparable to prior years’ assessments and preserve trend data going forward.”

Will the test be reported on the PARCC scale going forward?

Answer: The current plan is to report on the same scale.

135 A.9.6. Linking In reference to the requirement to establish a linking between the Grade 8 CMAS: Mathematics and ELA

- 3 -assessments and the Grade 9 assessment, what is the name of the grade 9 assessments, and which contract are they included in?

Answer: Current grade 9 assessments are the CMAS:

PARCC ELA and Math assessments. The grade 9 assessments to be given in 2018 and beyond have not yet been determined. Proposed legislation requires that they be aligned to both the CAS and to the 10th grade assessment, currently PSAT 10.

Section 3 – Statement of Work with Responsibilities by Component Offeror Qualifications

– Mandatory Minimum Qualifications: Experi-ence with developing, administering, scoring and reporting of innovative, interactive technology-enhanced items.

Which specific TEIs is Colorado currently using? What specific types of TEIs would CO like to see as a part of this work?

Answer: Items are currently coded to the following item types:

· Choice Interaction - Inline Choice

· Choice Interaction - Multiple Cardinality

· choiceInteraction

· Complex Slider

· CR2

· CR3

· customInteraction

· drawingInteraction

· extendedTextInteraction

· Gap Match Gap Table

· gapMatchInteraction

· graphicGapMatchInteraction

· Hot Text

· Hotspot Interaction - Multiple Cardinality

· Hotspot Interaction - Single Cardinality

· hotspotInteraction

· hottextInteraction

· Inline Choice - Cloze

· Inline choice table

· inlineChoiceInteraction

· Match Interaction - Gap Match

· Match Interaction - Graphic Gap Match

· Match Interaction - Matching

· Match Table Grid

· Matching

· matchInteraction

· Multiple Response

· Order Interaction - Horizontal Ordering

· Order Interaction - Image Ordering

· Order Interaction - Text Ordering

· Order Interaction - Vertical Ordering

· orderInteraction

· Point Graph - Line Graph

· SR

· Text Interaction - Extended Text Entry

- 4 -

· Text Interaction - Text Entry

· textEntryInteraction

· Vertical Bar Chart

Offerors should identify TEIs they have experience developing, administering and scoring and are suggesting for Colorado use.

Section 3 – Statement of Work with Responsibilities by Component Offeror Qualifications

– Mandatory Minimum Qualifications:

Online test administration system must allow for caching, as well as provide for downloadable apps.

With regards to the caching requirement, is CO looking for a low-bandwidth or completely disconnected testing solution?

Answer: Low-bandwidth. (Offeror systems should have recovery capabilities should there be an internet disruption.)

Section 3 – Statement of Work with Responsibilities by Component Offeror Qualifications

– Mandatory Minimum Qualifications:

Willingness to submit a set of sample items created by Offeror.

While story boards are acceptable for simulations and TEIs submitted with the proposal, if invited, the Offeror agrees to demonstrate the QTI‐ compliant items programmed in XML during the demonstration/interview phase.

Which QTI-compliant item types is CO looking for with regard to TEIs

Answer: Colorado is providing the interviews/demonstrations as an opportunity for Offerors to showcase the TEI items types that they have experience developing, administering and scoring and are suggesting for Colorado use.

N.

Proprietary/Confidential Information

Do vendors need to obtain permission to submit sample items?

Answer: Offerors may choose to submit open source, commissioned or permissioned passages/materials as part of their responses. Offerors should follow copyright, trademark and intellectual property law

- 5 -and consult with their legal counsel in making determinations about what they can submit.

N.

Proprietary/Confidential Information

If the only confidential items that vendors are submitting are the requested sample items, do we still need to submit a redacted version of our proposal?

Answer: No. But Offerors must follow instructions for approval to submit “confidential item” and if approved must be submitted in your response as confidential approved item.

Section 3, Offeror Qualifications

The RFP states that vendors are to provide three references.

Are these references the same as, or in addition to, the required submission of three Company Experience and Demonstrated Fact Sheets (Attachment 6)?

Answer: They are the same.

3–5, 59, and 68

Introduction, A.4.3, 2021 Test Design, and A.4.5.c, Number of Items to be Developed

In several places in the RFP, assessments in Algebra I, Geometry, and Integrated Math I and II are mentioned (pages 3, 4, 5, 57, and 68). The statements imply that these are separate assessments available to grade 8 students. Are they also available to all high school students?

Then on page 59, the RFP mentions the following: “While Colorado intends to continue to allow middle school advanced mathematics students to take the currently available high school end-of-course assessments through 2020…”

Please provide clarification regarding whether there are existing, separate end-of-course math assessments for Algebra I, Geometry, and Int. Math I and II, and whether the Contractor will be expected to continue to develop and administer these separate assessments, in addition to the grade 8 math assessment. Also confirm whether these assessments are administered to all high school students, or just advanced grade 8 mathematics students.

Answer: This RFP does not include high school students for mathematics. As indicated in the Introduction, currently middle school advanced mathematics students have the opportunity to take Algebra I, Integrated I, Geometry or Integrated II end-of-course assessments in place of the grade-based assessments. This practice is expected to continue through 2020. The Offerors will be proposing assessment designs for post revised standards adoption in 2018 with new assessment implementation in 2021.

147 E. Packaging Requirements; 3. Cost

For the Cost Proposal, the RFP states that vendors are supposed to submit “twenty-five (25) electronic

- 6 -

Proposal copies…labeled 1–15.”

Are vendors to submit 25 copies or 15 copies of the Cost Proposal?

Answer: 15 copies The RFP should read “fifteen (15) electronic copies…labeled 1-15.”

See RFP Modification 1 page 148.

Attachment 1: Criteria for Responsiveness Certification Form

The language under Minimum Qualification: Assurances and Commitments on page 172 (page 3 of Attachment 1) does not match the language in the same category on page 33 of the RFP. In particular, on page 33, the 4th sub-bullet under Assurances and Commitments is: “None of Colorado’s 10 largest districts have refused to utilize the proposed systems in the last two years.” The 4th item listed under Assurances and Commitments on page 172 is “A single instance of any application must be sufficient to run all state tests provided by the Successful Offeror in Colorado, whether those be through this contract or other contracts.” Please confirm which statement the Offeror is committing to.

Answer: The Assurances and Commitments on page 172 should be considered the accurate set. “None of Colorado’s 10 largest districts have refused to utilize the proposed systems in the last two years” should have been removed from page 33 and should not be included in the official set of Assurances and Commitments.

See strikethrough on page 34 highlighted in red in RFP Modification 1.

Section 4, Part E, Item 4 (Packaging Requirements. Proposals must be packaged and sealed in six parts:)

The first paragraph states that the Offeror should provide 25 electronic copies on flash drives. The RFP indicates that these should be labeled: COST 1-15.

Question: Should this be COST 1-25?

Answer: No, only 15 copies of the cost proposal are required.

The RFP should read “fifteen (15) electronic copies…labeled 1-15.”

See RFP Modification 1 page 148.

Cover Sheet General

The proposal opening time is listed as 2:00 p.m. MT on April 28, 2017. The RFP also indicates that the proposal is due on April 28 at 12:00 p.m.

- 7 -

Please verify that the proposals are due at 12:00 p.m.

(noon) MT and not at 2:00 p.m. on April 28.

Answer: Proposals are due at 12:00 p.m. (noon) MT on April 28, 2017.

See RFP Modification 1, Cover Page

Sections 4-8

(FORMAT FOR

RESPONSES)

We understand and appreciate the benefit of page restrictions, but given how comprehensive, we believe that we will require more pages to be compliant with all requirements, and are thus hoping to gauge the degree to which exceeding the limit, despite CDE’s strong encouragement, will count negatively in the assessment of proposals (assuming that the proposal content reads as all relevant and beneficial).

Answer: Assuming that the proposal text is relevant and beneficial, evaluators will continue to read and rate accordingly.

Section 2, Part A (Colorado System

Improvement Efforts)

What is the expected date by which the 9th grade ELA and mathematics assessments will be administered?

Answer: Spring 2018.

A.4.2.d.

(Design and Test Blueprints/

Specifications)

Given the required 20 percent reduction in operational items for CMAS ELA and Math in 2019, what flexibility (if any) will the CDE permit in reducing the number of performance level categories being reported on, in light of simulated or empirical evidence that the shortened forms may not be able to support 5 performance categories?

Answer: CDE will look for guidance and recommendations regarding the number of performance levels abbreviated assessments may be able to support from a technical perspective from the Successful Offeror’s technical experts and Colorado’s Technical Advisory Committee (TAC) to be weighed against broader policy implications.

56;

A.4.2.d. (Design and Test Blueprints/ Specifications) A.4.3.c. (Frameworks - 2021 Test Design)

In either the 2019 or later test administrations, does CDE expect the successful Offeror to directly assess "Oral Expression and Listening" standards for CMAS ELA?

How about "21st Century Skills and Readiness Competencies" across the assessment program?

Answer: Oral Expression and Listening is not currently part of the state assessment expectations.

The 21st Century Skills and Readiness Competencies portion of the Colorado Academic Standards should be used to inform item development. The Scientific Investigations and the Nature of Science subscore directly reports on aspects of the 21st Century Skills and Readiness Competencies.

- 8 -

A.7.

(SCANNING AND

SCORING)

In Table 20, CDE indicates that CMAS ELA is already being scored using automated scoring. Do we know who is doing this scoring?

Answer: NCS Pearson, Inc.

A.9.2.c.

(Field Test - Sampling Design)

This section indicates that the CMAS: ELA (excluding CSLA) is field tested in a semi-stand-alone administration

- with perhaps an additional testing session of approximately two hours. Is this administered during an additional day/session of testing contiguous with the operational test session(s) for the sampled schools and grades?

Answer: Districts create schedules that work for their schools and students. The session can be administered on the same day or on a different day during the established testing window.

A.9.7.d.

(Technical Advisory Committee)

The text of this section states that, "TAC members shall be given a $1,750 honorarium per day," but Table 22 - TAC Member Expenditures indicates that, "[e]ach TAC member shall be given a $1,500 honorarium per day." Please indicate which is accurate.

Answer: $1,750

See RFP Modification 1 page 137.

166 Appendix B

The sample table of contents refers to "Vertical scale design" (Section 8.11), but the RFP does not make mention of requirements to construct and maintain a formal vertical scale. If a formal vertical scale is required, please provide any additional information affecting the following sections as they pertains to vertical scaling:

- A.9.2. Piloting and Field Testing

- A.9.3. Operational Analysis of Items

- A.9.4. Initial Scaling and Standard Setting

- A.9.5. Ongoing Scaling and Equating

- A.9.6. Linking

- A.9.7. Establishing Technical Adequacy

- A.9.8. Irregularity and Data Forensic Analysis Answer: The current assessments do not use a vertical scale, so Vertical Scale Design will not be included in the first few years of the contract. Offerors are to provide their suggested test design for the post 2018 revised standards assessments to be implemented in 2021. If they choose to include a vertical scale, Offerors should include information throughout the proposal as they deem relevant.

- 9 -

141 A.11 (Customer Service)

The RFP states, “The lead supervisor and other Successful Offeror trained staff shall be available to answer Colorado calls from 7:00 a.m. to 6:00 p.m. Mountain Time each day, excluding federal and Colorado state holidays.” Our understanding is that each day refers to each day during the testing periods. Please clarify CDE’s desired dates and times for customer service staff.

Answer: Customer service representatives should be available to answer Colorado calls year-round. A higher call volume is expected directly before and during the testing periods (January through May).

141 A.11 (Customer Service)

What is the typical volume of calls to customer support during test administration windows? At other times of the year?

Answer: Call volume is directly dependent upon the ease of the Offeror’s systems and how well the online test administration and test management systems function. Offerors should make budgeting and staffing decisions based on their experience with similar programs of size and scope. During administration windows, Offerors should assume higher call volumes if the field is having a difficult experience. As students are often engaged in actual testing, immediate responses are expected during the administration window.

Section 6

(PROPOSAL

EVALUATION)

The RFP states, “All responsive proposals received by the deadline will be evaluated on best value (as defined by CDE) by representatives of CDE and its qualified designees.” Please provide additional details on the State’s planned proposal evaluation team: size of team, background, types of positions held, knowledge of assessments represented by participants.

Answer: This information is confidential at this time.

General;

A.6.3.d (Demonstration of Online Test Management and Test Administration Systems)

Can the state please provide sample items of each interaction type prior to the submission of a proposal response to this RFP? This will allow vendors to accurately estimate import efforts.

Answer:

Items are currently coded to the following item types:

· Choice Interaction - Inline Choice

· Choice Interaction - Multiple Cardinality

· choiceInteraction

· Complex Slider

· CR2

· CR3

· customInteraction

· drawingInteraction

· extendedTextInteraction

- 10 -

· Gap Match Gap Table

· gapMatchInteraction

· graphicGapMatchInteraction

· Hot Text

· Hotspot Interaction - Multiple Cardinality

· Hotspot Interaction - Single Cardinality

· hotspotInteraction

· hottextInteraction

· Inline Choice - Cloze

· Inline choice table

· inlineChoiceInteraction

· Match Interaction - Gap Match

· Match Interaction - Graphic Gap Match

· Match Interaction - Matching

· Match Table Grid

· Matching

· matchInteraction

· Multiple Response

· Order Interaction - Horizontal Ordering

· Order Interaction - Image Ordering

· Order Interaction - Text Ordering

· Order Interaction - Vertical Ordering

· orderInteraction

· Point Graph - Line Graph

· SR

· Text Interaction - Extended Text Entry

· Text Interaction - Text Entry

· textEntryInteraction

· Vertical Bar Chart

For examples, Colorado practice environments may be reviewed:

http://download.pearsonaccessnext.com/co/co-practicetest.html?links=1

General

Can the state please provide an analysis of the coverage of the current item banks by standard so that vendors can make accurate item development estimates and schedules prior to the submittal of a proposal response to this RFP?

Answer: Below are approximate item counts for grades 4, 7 and high school social studies and grades 5, 8 and high school for science.

Grade 4 Social Studies

Standard SR CR 1 64 35 2 67 38 3 44 25 4 38 27 http://download.pearsonaccessnext.com/co/co-practicetest.html?links=1 http://download.pearsonaccessnext.com/co/co-practicetest.html?links=1

- 11 -

Grand Total 213 125

Grade 7 Social Studies

Standard SR CR 1 69 51 2 73 51 3 84 24 4 63 23 Grand Total 289 149

High School Social Studies

Standard SR CR 3 point

ECR 7

point

1 88 29 9 2 69 39 3 40 15 4 41 22 Grand Total 238 105 9

Grade 5 Science

Standard SR CR 2 point CR 3 point 1 39 20 10 2 81 48 20 3 83 41 17 Grand Total 203 109 47

Grade 8 Science

1 100 45 20 2 79 40 16 3 101 49 21 Grand Total 280 134 57

High School Science

1 109 43 13 2 116 40 16 3 96 42 16 Grand Total 321 125 45

- 12 -

See A.4.5.c. for information on mathematics and ELA Colorado-owned item development.

A.3.2.i.

(Quality Control and Sign-Offs)

When the RFP states, “Successful Offeror shall document,” is this a request to documenting in the proposal or documenting in the future when under contract and work on deliverables is under way?

Answer: These are services and deliverables for the Successful Offeror to provide when under contract.

97 A.6.3. Online Test Administration System

We note the RFP specifically mentions "caching" and "local backup of student responses.” These are methods used by some vendors to address some possible technical issues. Will Colorado evaluate on an equal basis alternative approaches (other than caching and local backup) that have been proven to work in virtually 100% of the schools in more than a dozen states?

Answer: Based on the information provided, CDE is not inclined to remove caching (local storage) from the minimum qualifications.

146 E. Packaging Requirements

The RFP states the Narrative and Cost Proposal must be provided on flash drives in searchable PDF format. There is an additional requirement for Contract Negotiation Materials of the Narrative and Cost Proposal to be provided on flash drives in Microsoft Word format. Please clarify if CDE would like the Narrative and Cost Proposal to be provided in both Microsoft Word and PDF format.

Answer: Yes. Offerors should provide both formats.

97 A.6.3. Online Test Administration System

The RFP asks about the adaptive testing capability of the vendor’s system. To what extent will Colorado use, or may the vendor propose, adaptive testing?

Answer: Current assessments are not adaptive.

Offerors are to propose new test designs that could include adaptive functionality to be used post the 2018 adoption of revised standards with new assessment implementation in 2021.

26 A.5. Assessment Given the low quantities of paper testing and the potential for adaptive online testing, is Colorado open to local, on-demand printing of paper for some assessments?

Answer: No.

37 A.1.9. Subcontractor Requirements

What is Colorado’s definition of “subcontractor” for this RFP, as opposed to a vendor? We note, for example, that the RFP refers to the Braille vendor. Many states do not include typical vendors (e.g., Braille, FedEx/UPS, communications, cloud providers, printers, temporary agencies) within the definition of subcontractor.

Answer: For purposes of this section, “Subcontractor”

- 13 -means third-parties, if any, engaged by Contractor to aid in performance of the Work. The entities included in the questioners parenthetical would be subject to A.1.9 as applicable.

117 A.7.5. Automated Scoring

By grade, subject, and item type, please provide information on Colorado’s current use of automated scoring of constructed response items.

Answer: Grades 3-8 ELA extended constructed response items are scored using automated scoring.

69 A.4.5.d. Passage/Source Selection and Storyboard Creation

We note the RFP line “Arranging for transferable permissions is strongly preferred, to ease transition to next contractor.” Are any of the current permissions help by Pearson transferable to the new vendor? If yes, what percentage (by grade/subject)? Are any of these transferable permissions paid up past the current contract period?

Answer: Offerors should assume that current permissions are not transferable.

79 A.4.7.c. Braille We understand Colorado does not currently know all UEB/EBAE/Nemeth Braille requirements going forward, and vendors must be prepared to be flexible. Will Colorado provide specific requirements by year for pricing purposes so that all vendors are bidding the same assumptions?

Answer: Offerors should assume both UEB with Nemeth and EBAE in 2017-2018 and a full transition to UEB with Nemeth by 2018-2019.

42 A.3.1.a Preliminary Program Plan

We understand that Liquidated Damages Attachment 4 will be subject to contract negotiations. May vendors propose changes to the Liquidated Damages Attachment 4 in advance of contract negotiations?

Answer: Yes, but not required. Offerors may submit proposed changes to the Liquidated Damages as part of the proposal; however, Offerors should not assume any agreement on the part of CDE to such proposed changes.

33 Offeror Qualifications – Mandatory Minimum Qualifications, Assurances and Commitments

Virtually every major provider of statewide assessments fails to meet the RFP requirement regarding retraction of publicly released scores. Not all incidences of score reporting retractions appear in the public domain however, we assume that retractions have taken place. Did Colorado intend to make this a disqualifying condition, which will significantly reduce competition?

Answer: “Offeror shall not have been the (scoring and/or reporting) Contractor for a program that has had publically-released scores retracted in the last two years” is a mandatory minimum qualification.

- 14 -

See RFP Modification 1 pages 34 and 173rr.