201-26-045 Exhibit A Requirements.xlsx
XLSX spreadsheet 124 KB Posted
- Solicitation number
- RFP-124939
- Issued by
- Virginia
About this file
Summary of RFP 201-26-045 Exhibit A Requirements
This document is an Exhibit A requirements specification for Virginia Department of Education (VDOE) detailing prerequisites and comprehensive technical, security, functional, and operational requirements for RFP 201-26-045: Virginia Statewide K-12 Student Assessment Program (SOL). The procurement seeks a supplier-hosted cloud solution capable of developing, delivering, scoring, administering, and reporting Virginia's modernized K-12 summative assessment program in dual-mode format (online and paper). Prerequisites require the supplier to have successfully implemented substantially similar statewide K-12 summative assessment systems with at least two state clients, with at least one client having a minimum K-12 student enrollment of 400,000 students. The solution must comply with all current Commonwealth of Virginia Information Technologies Agency (VITA) policies and standards, including information security, data standards, artificial intelligence policies, and technology roadmaps. The supplier must support Multi-Factor Authentication (MFA) using application-based authenticators, Okta Single Sign-On (SSO) compatibility, FERPA and HIPAA compliance, FedRAMP authorization for cloud services, and encryption of data in transit and at rest. The supplier is responsible for comprehensive test item development and management including annual item development plans, field testing, psychometric analysis, standard setting facilitation, equating and scaling, technical reporting, test administration material production, scoring and reporting across multiple formats, and support for the Virginia Alternate Assessment Program (VAAP) through September 2029.
The contract requires the supplier to manage all aspects of annual assessment administration across three testing windows (fall, spring, summer), produce and distribute secure and non-secure test materials with specified timelines, support paper and online testing formats, manage retake and term graduate testing workflows, and meet legislatively mandated reporting deadlines requiring individual score reports and statewide data release within 45 days of the assessment window conclusion as of July 1, 2026. The supplier must implement a 100-point scale reporting requirement in addition to existing 0-600 point scales beginning July 1, 2026, and maintain item security through non-disclosure agreements and documented procedures. All vendor staff, including subcontractors, must be physically located in the United States. The supplier must sign the Virginia Standard Student Data Privacy Agreement (Appendix J) and coordinate with VDOE's Technical Advisory Committee for guidance on psychometric matters. The supplier is required to archive all committee proceedings and documentation according to Library of Virginia retention schedules and provide annual technical reports meeting United States Department of Education peer review standards within 180 days of the final assessment window. No specific contract term, pricing information, or award date is identified in this requirements document.
View the file
Other files
Show all 15
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
Instructions RFP 201-26-045, Virginia Statewide K-12 Student Assessment Program (SOL)
EXHIBIT A REQUIREMENTS
Instructions
| This Excel workbook has multiple tabs that set forth requirements to which Supplier is required to provide a response indicating its capability to fulfill each specific requirement. Supplier’s response will be reviewed in accordance with the stated requirements within each section in order to determine the best solution for Agency. |
| The Prerequisites tab, which will be evaluated as Pass/Fail, will determine whether the Supplier moves on to the next phase of evaluation. |
| For each requirement, Supplier is asked to enter in the space provided in Column C a response (Meets, Does Not Meet) that best corresponds to its intended response for the requirement listed. In cases where VITA has pre-filled Column C for certain requirements with "N/A," Supplier must provide sufficient information in its response in Column D to allow VITA to properly evaluate its proposal. |
| The acceptable responses are as follows: |
| "Meets" = Supplier can fully meet the requirement as documented with its current application or proposed solution. Supplier shall provide an explanation in Column D of how it will fulfill the requirement. This may include use of alliances with other suppliers. Supplier's detailed response should NOT be an affirmative reiteration of the requirement, but should explain how the requirement will be met. |
| "Does Not Meet" - Supplier cannot meet the requirement as written. Supplier should provide an explanation of why it cannot be met OR whether it has an alternative approach (with description). |
| Supplier should review all responses for completeness before submitting its proposal. Ensure that all applicable sections of the questionnaire are answered, as incomplete answers may impact the evaluation of your proposal. A blank response in Column C will be interpreted as a "Does Not Meet" response. |
| Supplier must respond to all tabs in this workbook. Do not reformat or revise the tabs as this may cause Supplier to be removed from being considered for award of this contract. |
Prerequisites RFP 201-26-045, Virginia Statewide K-12 Student Assessment Program (SOL)
EXHIBIT A
Prerequisites
| **(Supplier Name Here)** | ||
| Meets/Does Not Meet | ||
| The Supplier must have successfully completed the implementation of products/solutions substantially similar to what is proposed here—i.e., a statewide K-12 summative assessment system delivered online and on paper--with at least two state clients, at least one of which has a total K-12 student enrollment of at least 400,000 students, as of the date of this RFP posting. | ||
| As proof of meeting this prerequisite, Supplier must list the customer(s) Supplier has provided with a successful implementation in Appendix C: Supplier Profile under the Supplier References section. |
VITA's Requirements RFP 201-26-045, Virginia Statewide K-12 Student Assessment Program (SOL)
EXHIBIT A
VITA Requirements
Section A General Security Requirements Section B Artificial Intelligence Requirements Section C Cloud Requirements Section D COV Technology Roadmaps Section E Other Requirements
| **(Add Supplier Name Here)** | |||
| Section A | General Security Requirements | Meets/Does Not Meet | Additional Detail |
| 1A | Supplier's proposed solution shall comply with all current COV ITRM Policies and Standards, including those pertaining to: |
1. Information Security
2. Data Standards
3. Prohibited Hardware and Software Services
4. Artificial Intelligence
Policies are located at: https://www.vita.virginia.gov/policy--governance/policies-standards--guidelines/
If proposed solution does not, please provide details that specify the Standard/Policy and how Supplier's solution does not comply.
2A Supplier’s proposed solution conforms with the accessibility requirements of Virginia Code § 2.2-3502, as the term “accessibility” is defined in Virginia Code § 2.2-3501.
-If yes, Supplier shall certify by providing a current Supplier-paid and completed Accessibility Conformance Report indicating the level of the proposed solution’s conformance with the accessibility requirements.
-If no, Supplier shall document any areas of nonconformance with a Supplier-paid and completed Vendor Accessibility Roadmap highlighting areas requiring improved accessibility, including a timeline for each nonconforming area’s completion.
3A Supplier's proposed equipment shall meet the current U.S. Environmental Protection Agency's and Department of Energy's Energy Star guidelines.
If not applicable to this procurement, mark “N/A” in Column D and provide an explanation.
4A Supplier shall ensure all appropriate staff have read the Commonwealth’s security policies, standards and guidelines, applicable to the proposed solution, located at the following URL. https://www.vita.virginia.gov/policy--governance/policies-standards--guidelines/
Please state yes or no and explain the top 5 concerns you identify, if any.
| Section B | Artificial Intelligence | Meets/Does Not Meet | Additional Detail |
| 1B | In compliance with 2024 Executive Order 30, regarding the Implementation of Standards for the Safe Use of Artificial Intelligence Across the Commonwealth (EO 30), Supplier shall indicate and describe in detail whether any aspect of the proposed Solution and/or Service contain Artificial Intelligence (AI), as defined in EO 30, the Utilization of Artificial Intelligence by COV Policy Standard and VITA's Artificial Intelligence Standards. |
Additional information regarding VITA's Artificial Intelligence Standards is available at the following URL:
https://www.vita.virginia.gov/artificial-intelligence/ 2B If any aspect of Supplier's proposed Solution or performance of the proposed Services contain AI as defined in EO 30 and VITA’s Artificial Intelligence Standards, then describe the process that Supplier will follow to ensure that Customers, as applicable, have registered the Solution/Services with VITA before a purchase order is fulfilled.
Additional information regarding VITA's Artificial Intelligence Standards is available at the following URL:
https://www.vita.virginia.gov/artificial-intelligence/
| Section C | Cloud Requirements | Meets/Does Not Meet | Additional Detail |
| 1C | Numerous solutions are offered as cloud platforms/services or have cloud options. Executive branch agency procurement of solutions with cloud elements require that the agency seek VITA cloud approval before procurement, including COVRamp assessment, agreed Cloud Terms and ongoing oversight. |
Describe the process for ensuring that VITA cloud approval has been obtained before a purchase order has been placed.
| 2C | Describe whether the proposed cloud solution is a Software as a Service (SaaS) delivery model or a Platform as a Service (PaaS) delivery model. |
| 3C | Describe whether the proposed solution is a public, private, government, or hybrid cloud model. Describe all available models and ensure pricing response in Exhibit B includes offered options. |
| 4C | Describe whether the proposed solution allows for agency hosting. Elaborate on the advantages and disadvantages of offsite and onsite hosting that the solution offers. |
| 5C | Describe whether the proposed cloud solution is FedRamp authorized. If so, describe the FedRamp level of certification (low, moderate, or high). |
| 6C | The proposed Solution shall comply with all Customers’ individual Information Security Policies and applicable Federal standards (e.g., FedRAMP, CJIS, FISMA,PCI, ISO27001, FERPA, FTI (IRS PUB-1075), SSA, HIPAA-HITECH). |
If proposed solution does not comply, please provide details that specify the Standard/Policy and how Supplier's solution does not comply.
| 7C | Describe whether the proposed cloud solution relies on third-party partners or subcontractors and elaborate on the third-party partners/subcontractors required as applicable. | ||
| 8C | Describe whether the proposed cloud solution allows a customer to solely manage its own encryption keys or if the function must remain with the solution provider. | ||
| 9C | Describe whether the proposed solution has a documented encryption key management process. | ||
| 10C | Describe whether the proposed solution uses standard data encryption techniques. If so, define in detail at all levels. Include, if applicable, what additional options may be added. | ||
| 11C | Describe how the proposed solution addresses security. Provide an attachment to detail the levels of security inherent in your application and what options can be added to support unique customer requirements to ensure confidentiality. | ||
| 12C | Given the desire for electronic payment, with approval from DOA, please describe in detail how this process will work and what safeguards and security measures will be employed to ensure audit compliance. | ||
| Section D | COV Technology Roadmaps | Meets/Does Not Meet | Additional Detail |
| 1D | The proposed solution should only use Approved versions of the technologies covered within the COV Technology Roadmaps. |
https://www.vita.virginia.gov/policy--governance/architecture/cov-technology-roadmaps/
If proposed solution does not only use Approved versions of COV technologies, please provide details on the technologies utilized.
2D For IT solutions and technologies that are not covered within COV Technology Roadmaps, describe if the proposed solution only uses versions or models of IT solutions and technologies that are current (N and N-1) and have vendor or equivalent support.
Example: an IT management utility that only works with a EOL database or N-2 operating system
Note 3: many N and N-1 versions are defined in the COV Technology Roadmaps 3D Describe if the proposed Solution ensures IT solutions and services as well as everything supporting supplier provided or managed technologies, IT solutions, and services shall comply with VITA Policies, Standards, and Guidelines for all supplier provided and managed technologies.
Note: This includes processes, IT solutions, technologies, devices, personnel, etc.
4D Describe if the proposed solution produces a documented lifecycle including information on currency, general availability, and support (end of service life and end of support) for all supplier provided or managed IT solutions and technologies. Note: See COV Technology Roadmaps for more information.
Supplier shall document all versions of software that their services consume or provide. This includes all software that the service providers will be using to host, develop, support, test, and deploy their IT solutions (operating systems, languages, databases, editors, etc.).
Supplier shall produce and maintain defined and VITA-approved baselines for all supported technologies for all IT solution and technologies provided or managed by Supplier.
Note: Examples of baselines are Center for Internet Security (CIS) and Department of Defense (DoD) Security Technical Implementation Guides (STIGs).
| Section E | Other VITA Requirements | Meets/Does Not Meet | Additional Detail |
| 1E | Describe whether the solution incorporates security and privacy recommendations and best practices from the National Institute Standards and Technology (“NIST”). | ||
| 2E | Supplier, upon contract award, shall ensure its line item contract pricing is uploaded in eVA: |
https://eva.virginia.gov/eva-catalog.html 3E Supplier's solution shall have the ability to produce custom financial information and reporting as required to ensure proper accounting and financial management by VITA and Authorized Users, in compliance with the pronouncements of the Governmental Accounting Standards Board (GASB), with particular emphasis on Statements No. 87 "Leases" and No. 96 "Subscription-Based Information Technology Arrangements." Examples may include but are not limited to, asset/inventory management for leased equipment, discount/interest rates for leases, inventory management for software licenses including coverage terms, etc. For more information see:
https://www.gasb.org/standards-and-guidance/pronouncements
VDOE's Security Requirements RFP 201-26-045, Virginia Statewide K-12 Student Assessment Program (SOL)
EXHIBIT A
VDOE's Security Requirements
| **(Add Supplier Name Here)** | |||
| VDOE Security | Meets/Does Not Meet | Additional Detail | |
| S-1 | The solution shall support Multi-Factor Authentication (MFA) for both internal and external users in accordance with current Virginia Information Technologies Agency (VITA) security standards. MFA must utilize a secure method such as a push notification via an application-based authenticator (e.g., Microsoft Authenticator, Google Authenticator). Use of SMS or email-based authentication methods is not permitted. The vendor must describe how MFA is implemented and managed within the solution. |
https://www.vita.virginia.gov/media/vitavirginiagov/it-governance/psgs/pdf/SEC530_Information_Security_Standard.pdf
| S-2 | The solution shall be compatible with Okta, the Commonwealth of Virginia’s (COV) enterprise identity and access management platform, to support Single Sign-On (SSO) functionality for internal users. |
| S-3 | The solution vendor shall identify and provide a detailed explanation of any open-source software components utilized within the application or solution, including their purpose and associated licensing terms. The response must include a copy or link to licensing terms of any open-source software. |
| S-4 | The solution vendor shall identify and provide a detailed explanation of any third-party partners or subcontractors involved in the development, implementation, or maintenance of the solution. |
| S-5 | The solution shall support encryption of data both in transit and at rest, and must include detailed documentation of encryption key management practices, including key generation, storage, and access control procedures. |
| S-6 | Vendor staff, including any subcontractors, must be physically located in the United States, and must follow all COV and VDOE policies and procedures for data privacy and security at https://www.vita.virginia.gov/policy--governance/policies-standards--guidelines/. |
| S-7 | Vendor must be prepared to sign Appendix J: Virginia Standard Student Data Privacy Agreement. |
| S-8 | Vendor must maintain confidentiality of student information, and comply with the Family Educational Rights and Privacy Act (FERPA) Health Insurance Portability and Accountability Act (HIPAA). |
BusinessFunctional Requirements RFP 201-26-045, Virginia Statewide K-12 Student Assessment Program
EXHIBIT A
Business/Functional Requirements
| **(Add Supplier Name Here)** | |||
| Requirements | Meets/Does Not Meet | Additional Detail | |
| 1 Test Development | |||
| 1.1 Item Development | |||
| 1.1.1 | The Offeror shall be responsible for all activities related to the development of items, stimuli and passages, and scoring materials for each test during each year of the contract to ensure reliable, valid assessments aligned to the Standards of Learning, for all assessments specified in the Solicitation Table 2, in all formats as specified, including all accommodated forms, at the volumes needed for testing all students in the required grades and/or subjects. These activities include, but are not limited to the writing and field testing of new items and passages for the Virginia Assessment Program, including the maintenance of item specifications, annual review of the item bank and preparation of an item development plan, item writing, and planning and facilitation of item and data review committees. The item banks used for the tests must be supplemented annually to maintain the currency, comprehensiveness, security, and integrity of the test content, in addition to allowing for the regular release of items to support instruction and fulfill any legislative or policy requirements. | ||
| 1.1.2 | As noted in the solicitation, any items developed for use in Virginia shall be the exclusive property of VDOE in perpetuity. However, Virginia is open to the possibility of leasing a certain number of existing items owned either by a vendor or another state, particularly in the initial years of the contract, in order to increase the depth of the item banks or more expeditiously allow for new item types. |
If the Offeror intends to propose leasing items as part of their solution, the Offeror must include proposed contract terms as part of its response including terms for work product ownership and licensing permissions.
1.1.3 If the Offeror intends to propose leasing items as part of their solution, the Offeror must provide a plan for how these items will be aligned to Virginia standards and test blueprints, and adhere to all other item specifications in Exhibit A: Requirements section 1.2. The plan must include VDOE review and approval of selected items, including VDOE's ability to reject or require replacement of items that do not meet the state’s expectations.
To support this review, VDOE must have access to the technical documentation needed to support use of the items, including information about item development, review processes, alignment evidence, field-test or operational statistics, item parameters, DIF or other fairness analyses where available, scoring rubrics, exposure history, and any other evidence needed to support validity, reliability, fairness, scoring, reporting, and peer review.
| 1.1.4 | If the Offeror intends to propose leasing items, the Offeror must field test the existing items with Virginia students prior to operational use. | |
| 1.1.5 | If the Offeror intends to propose leasing items, the terms of any such lease or use agreement will need to ensure continued availability of the items, stimuli, and any other associated content, for the life of any forms in which they are used, and must consider any item release plans or mandates, as noted in Exhibit A: Requirements section. | |
| 1.1.6 | If the Offeror intends to propose leasing items, the Offeror must provide information about any use of the items in other contexts outside the Virginia Assessment Program (including in other states, districts, interim assessments, practice systems, or commercial products). The Offeror must provide the VDOE with information about the potential exposure risk, and the procedures in place for monitoring for any item security concerns, and retiring or replacing items if security concerns arise. | |
| 1.1.7 | All elements of test development, including, but not limited to, item writing and forms construction shall be accomplished with the principles of Universal Design to make the tests as accessible as possible by all students, including those with disabilities and those for whom English is not their primary language. Details regarding the training materials used for item writers on the principles of Universal Design and the process to ensure the accessibility of items shall be subject to approval by the VDOE. | |
| 1.1.8 | The Offeror must provide and carry out annually an item development plan for each test that describes the manner in which each required task is to be accomplished and the timeline for accomplishing each task. The plan must address: | |
| • | An annual comprehensive review of the item banks for each test and a targeted item development plan designed to address deficiencies in each existing bank and maintain item bank health. Current item counts by type and test are provided in Appendix H: Item and Form Counts for reference. | |
| • | Anticipated item release - see Exhibit A: Requirements 1.1.9 for more details. | |
| • | The number of potential field test items developed annually for each test will be determined collaboratively by VDOE and the Offeror and must be based on the targeted item development plan. The plan should address developing more items than will be needed for field testing (to allow for possible attrition during internal and external reviews), as well as consideration of the size of the specific test-taking population, given that varies significantly by assessment. More specifics about the types of items that must be developed are provided in Exhibit A: Requirements section 1.2. | |
| • | Strategies and changes to the item development process to increase the efficiency of the process and minimize the number of items rejected during item review. | |
| 1.1.9 | The Offeror must be prepared to release a substantial number of items on a regular schedule (ideally annually) to support instruction and fulfill any relevant legislative or policy requirements, including any changes due to new legislation or policy initiatives, with the review and approval of the VDOE. Item releases must include all forms, including online, paper, large-print, and braille. Any item release will be subject to determinations of item bank health and ongoing successful administration of the assessment program, as collaboratively determined by the Offeror and the VDOE with the advice of the Technical Advisory Committee. | |
| 1.2 Item Specifications | ||
| 1.2.1 | The Offeror must develop new items for the computer adaptive SOL tests for grades 3-8 reading and mathematics and for the linear forms used for the remaining SOL tests. The Offeror must develop and carry out procedures to ensure that each item meets the specifications detailed below. | |
| 1.2.2 | Item development shall be accomplished in accordance with the principles of Universal Design to make the tests as accessible as possible for all students, including students with disabilities and English language learners. Details regarding the training materials used for item writers on the principles of Universal Design and the process to ensure the accessibility of items shall be subject to approval by the VDOE. | |
| 1.2.3 | The Offeror must provide and implement a comprehensive plan that addresses the requirements related to item specifications listed in Exhibit A: Requirements 1.2.4-6 below, and collaborate with VDOE staff annually to review test item specifications and update or revise them as needed. This review must occur prior to the item development plan detailed in Exhibit A: Requirements 1.1.8. | |
| 1.2.4 | Each item developed must: | |
| • | be aligned to a content standard from the Virginia Standards of Learning. Items may be aligned and coded to multiple sets of standards or multiple versions of those standards. | |
| • | be aligned with a selected cognitive complexity level, using Webb's Depth of Knowledge framework | |
| • | be developmentally appropriate for the grade level being assessed | |
| • | be designed using universal design principles to improve accessibility for all students | |
| • | be clearly, precisely, and unambiguously worded, including taking into consideration wording that may be confusing or difficult for English learners | |
| • | be free of bias, stereotyping, and insensitive and/or provocative language and content | |
| • | contain/represent content that is unequivocally correct with factual information supported by primary source reference material. | |
| • | provide correct answer(s) and distractors that are plausible and defensibly incorrect; for constructed response questions, must include a rubric and scoring notes/guide. | |
| • | adhere to established VDOE style, which will be provided to the successful Offeror. | |
| 1.2.5 | Each item must be identified with a unique code. The item code schema currently in use identifies the SOL measured and the year the item was field tested and is expandable as tests or content standards are added or revised to the Virginia Assessment Program. The VDOE prefers to continue with its existing test item numbering scheme. Should the Offeror wish to propose a different item numbering scheme, such scheme must provide a unique code for each test item and must be associated with the required metadata in Exhibit A: Requirements 1.2.6 below. | |
| 1.2.6 | Each item must be have a comprehensive set of metadata that includes the following requirements: | |
| • | identify the content standards measured to the sub-bullet level, including the grade level and subject area, and may include alignment to multiple sub-bullets, sets of standards, or multiple versions of those standards. | |
| • | identify the type of item (i.e., multiple-choice, technology-enhanced, or constructed response); | |
| • | have sufficient flexibility to allow for re-coding an item when the SOL are revised or when a decision is made that the item better measures another content standard; and | |
| • | allow the historical tracking of an item that has been re-coded, used on a linear test form, been revised by a committee for use on special test forms (and has a "sister" item), and the time of release/do not use (DNU). | |
| • | at certain grade levels specified by the VDOE, mathematics items are to be designated as calculator active or "non-calculator", with a calculator yes/no designation to be maintained in the metadata. | |
| • | for items that have been field tested and gone through data review, identify the item descriptor aligned to the content measured by the item - see Exhibit A: Requirements 1.2.17 below for more detail. | |
| • | include item difficulty (high/medium/low) or estimated item difficulty for new items prior to field testing. |
Each new item developed must also inlude the following additional metadata:
| • | Cognitive complexity level, using Webb's Depth of Knowledge framework | |
| • | Performance Level Descriptors - the expectation for the performance level applicable to the standard | |
| 1.2.7 | The Offeror shall have the capacity to support the VDOE in expanding the development and use of constructed response items in the future, including additional potential item types, constructs, and use on additional assessments (e.g., constructed response items on math assessments that ask students to explain their thinking). This must include having scoring mechanisms in place that support the inclusion of these items in CAT. | |
| 1.2.8 | Offeror must develop or obtain reading passages and other stimulus material as necessary per test blueprints. Offeror must: | |
| • | Ensure that any passages used as item stimuli are checked for reading difficulty, as well as subjected to external validation review, including review by the appropriate item review committee, to determine that they are at the appropriate level of difficulty for each grade. Reading difficulty shall be determined by one or more methods approved by the VDOE. | |
| • | Ensure that stimuli for constructed response items or cluster items and passages used for reading are reviewed by item writers across grade levels to ensure that the content is unique and not represented in another grade level. | |
| • | Maintain established word count limits as specified in test blueprints. | |
| 1.2.9 | The Offeror must ensure that sufficient items that can be presented in a paper format are developed to support the necessary paper tests required for students who are unable to access an online testing format. This includes the presentation of scenario-based or “cluster” items and constructed response items. Images developed for items should be appropriate for use on paper test forms to the extent possible. | |
| 1.2.10 | The Offeror must develop and carry out procedures to ensure that all items are aligned to the Standards of Learning content standards (https://www.doe.virginia.gov/teaching-learning-assessment/instruction) and adhere to the item specifications detailed in these requirements. Matching the item to the content standard and the specifications shall be done by the original item writer, the internal reviewers, and VDOE staff. The match to the content standard shall be confirmed by the item review committees (see Exhibit A: Requirements section 1.3). Records to demonstrate that all items match the item specifications and content standards must be kept for all items throughout the contract period. Items must be internally reviewed by the Offeror prior to submission to the VDOE and the item review committees. | |
| 1.2.11 | The VDOE prefers to not use copyrighted permissioned documents and/or art. However, offeror must submit a plan to obtain, pay fees for, and maintain documentation of all copyright permissions in the event it is necessary to use them. If such materials are needed, copyright permissions shall be obtained for the use of the material for a minimum of ten (10) years in both written test form, secure online testing form, and for possible dissemination to the public in released items and interpretative materials. Such materials are given to the schools and school divisions with the express permission that they can be reproduced for educational purposes. As copyright permissions are negotiated, this must be made clear to the original copyright holders. All assessment materials that are distributed by the VDOE to schools and school divisions shall bear a VDOE copyright, with documentation of other copyrights within the documents. | |
| 1.2.12 | Offeror must: | |
| • | Ensure all art and images used in item development meet the VDOE’s accessibility requirements, including those for color and color contrast. Virginia online assessments are developed using a color palette that has been approved by the American Printing House for the Blind. Additionally, item writing must include a process to develop items that are not color dependent and test items must receive an additional check using a color contrast analyzer for color deficiency. Upon contract award, the VDOE will provide to the Offeror copies of the current versions of the color palette used by Virginia. | |
| • | Ensure that items include high quality graphics that will display equally well in both print and online presentations on all approved devices. | |
| • | When color images are grayscaled for use in print, appropriate contrast for accessibility must be maintained. If edits are required to address contrast, the contractor must note this in a metadata field visible to VDOE and confirm no errors have been introduced during the editing process. | |
| 1.2.13 | Offeror must provide documentation for each test item at each step of the test item development process. For all items, the documentation must include: stimulus material (if applicable), the item stem, estimated item difficulty level (high/medium/low), the cognitive complexity level of the item using Webb's depth of knowledge framework, and the anticipated Performance Level Descriptor. | |
| 1.2.14 | For all multiple-choice items and technology-enhanced items, th item documentation must include the answer choices, the correct answer(s). All multiple-choice items must also have descriptive rationales for each distractor; distractors must be plausible, defensibly incorrect, and represent common misconceptions or errors anticipated. For two-part items, the documentation must include the description of responses for 0 points/1 point/2 points. For constructed response items, the documentation must also include the scoring rubrics. | |
| 1.2.15 | The accuracy of test items is an issue of paramount concern to the VDOE. Therefore, all items developed for the Virginia Assessment Program shall be presented to the VDOE accompanied by documentation of the source upon which the item is based. The source documentation material shall demonstrate that the keyed answer(s) to each question are correct and defensible, and must be attached to each item (such as a PDF or Word document, not only a website) with relevant information pertaining to item highlighted. | |
| 1.2.16 | Item documentation must include the source of each reading passage or other item stimulus material, along with description of copyright restrictions. The VDOE will not approve items for use as field test items unless all documentation has been provided, including necessary copyright information. Offerors should propose a template to be used with accompanying documentation. | |
| 1.2.17 | Item Descriptions. The Offeror must prepare a description of each operational test item to be used for item analysis reports, provided to school divisions and parents and families. These descriptions must provide information about what the item measures without revealing the specific content of the item. Item descriptions must be identified by a unique code so that they can successfully be matched with items for the report and should be included as part of an item’s metadata. All item descriptions are subject to approval by the VDOE. Upon contract award, the VDOE will provide to the Offeror copies of the current versions of existing item descriptions. | |
| 1.3 Item Development Writers, Reviewers, Committees. | ||
| 1.3.1 | The test development process for the Virginia Assessment Program includes the extensive involvement of committees of Virginia educators selected by VDOE-- 10-15 Virginia educators for each SOL test. These committees lend instructional expertise and practical experience to ensure high-quality items and measure the SOL as intended. The item review committees review newly developed items and passages/stimuli before they are field tested and again after field testing before the items become eligible for inclusion in a computer adaptive test (CAT) item bank or on an operational linear form. ln addition, the committees review and provide recommendations on test blueprints and ancillary testing materials such as formula sheets, performance level descriptors, and scoring criteria. |
The Offeror must train, staff, and facilitate item review committee meetings with highly qualified individuals. Each facilitator must have expertise with the item and grade-level content for the committee and experience in working with and facilitating meetings with groups of educators. Facilitators must also be knowledgeable in and comfortable with explaining technical information (e.g., p-value, point biserial, Rasch difficulty, group difference flags, frequency distribution) to committee members. Each committee must also have a note-taker who can provide detailed and accurate notes from the proceedings.
There have typically been 27 total committees, with 9-10 committees meeting each week over the course of three weeks.
| 1.3.2 | The Offeror must provide training to prepare facilitators and note-takers prior to committee meetings. The training shall include a review of the Virginia Assessment Program test item specifications and test blueprints, characteristics of high quality test items and performance tasks, and how to identify bias in test questions. All training materials must receive prior approval from the VDOE. | |
| 1.3.3 | The Offeror must train item review committees in the item review process and to facilitate their review of items. The trainings must prepare committee members to review newly developed items before they are field tested (new item review), as well as items that were field tested the previous year with their field test data (data review). The training must include the following areas: | |
| • | Criteria to be considered in reviewing items before field-testing (new item review) and after field testing (data review) | |
| • | Practice for committee members in each phase of the review | |
| • | Clear guidelines for item review committees to use when examining newly developed test items, prompts, and stimuli/reading passages to ensure that each item is at an appropriate difficulty level for the subject and grade assessed, that there is an appropriate match between each item and the SOL assessed, and that items, prompts, and stimuli/passages are free of bias/sensitivity for any student group. | |
| • | Outcomes of the review. Newly developed items may be accepted as is for field testing or the committee may recommend revisions to the item prior to field testing. Similar review criteria are used for data review but the field test data associated with the items is also considered. Items that have been field tested may be accepted into the item bank, recommended for re-field testing, or deleted. | |
| • | Training for data review must include information about all the data collected and determined by field testing, with descriptions of each piece of data. Facilitators must be comfortable explaining technical information (p-value, point biserial, Rasch difficulty, group difference flags, frequency distribution, etc.) to committee members. | |
| 1.3.4 | The Offeror must deliver to the VDOE a record of decisions made by the committee for each stimulus, passage, prompt, and item. Information provided shall include a record of the committee’s votes regarding the disposition of each item, passage and stimulus as well as notes about comments made by the committee. This record must include a summary of the committee’s deliberations and recommendations for edits and changes. Decisions regarding the status of each item shall be subject to approval by the VDOE. All committee proceedings (notes, item review documents) must be archived in accordance with the Library of Virginia's retention schedules (see https://www.lva.virginia.gov/services/rm/schedules); Offeror shall work with VDOE staff to determine the specific retention categories for committee documentation. | |
| 1.3.5 | The Offeror is responsible for hiring and training item writers used to develop or identify passages and stimuli, test items, prompts, and for the Virginia Assessment Program. The VDOE reserves the right to reject the Offeror’s selection of item writers and reviewers at any time during the development process, in which case the Offeror shall provide acceptable replacements in a timely fashion so as to not jeopardize the overall test development timelines. |
The awarded Offeror shall provide VDOE with resumes for staff assigned to/responsible for items and forms for each SOL test. When the awarded Offeror has staffing changes, VDOE shall be notified and resumes for new staff shall be provided to VDOE.
| 1.3.6 | The Offeror must train item writers and reviewers to prepare them for the Virginia Assessment Program. Training for item writers and reviewers shall include a review of the associated Standards of Learning and Instructional Guides; Virginia Assessment Program test item specifications, test blueprints, and performance level descriptors; the determination of appropriate cognitive complexity levels for items using Webb's Depth of Knowledge framework; and the characteristics of high quality test items, use of universal design in item development, and how to avoid bias in test questions. All training materials must receive prior approval from the VDOE. | |
| 1.3.7 | The Offeror must take all necessary measures to ensure and maintain the security of test items, including but not limited to the use of non-disclosure agreements for item writers, reviewers, and all involved in the development process. | |
| 1.3.9 | The VDOE plans to add a benchmarking process as a key part of expanded constructed response item development/field testing in future years. The Offeror shall describe experience and methods for conducting benchmarking of student constructed responses, including identifying anchor sets of responses at each performance level and ways to identify atypical responses that may be more challenging to score. | |
| 1.4 Field Testing and Analysis | ||
| 1.4.1 | All new test items must be field tested with Virginia students prior to operational use. The purpose of the field test of new items is to gather statistical information that will be used to evaluate the quality of individual test items and assist in the creation of the final operational tests. | |
| 1.4.2 | The Offeror must develop a schedule and methodology to accomplish the required field testing for tests administered in a CAT format as well as for linear forms, subject to the approval of VDOE. | |
| • | The plan must ensure that an adequate number of items are field-tested to replenish item pools and sustain test forms that match their intended blueprints in subsequent administrations. | |
| • | The Offeror must ensure that field test items administered as part of the CAT are administered to a representative sample of students at all ability levels to establish appropriate item parameters. | |
| • | SOL tests primarily depend upon embedding field test items within operational SOL tests. However, the Offeror may propose the inclusion of stand-alone field tests where use of embedded field tests is not possible. If stand-alone field tests are used, the plan must address the provision of all necessary materials for field testing. | |
| • | Items must be field tested in the order of priority to achieve balance in the item pool. | |
| 1.4.3 | The plan must address strategies for field testing reading passages and writing prompts used for the integrated reading and writing assessment in grades 5 and 8, and which may be used in additional assessments going forward, if new passages and prompts are needed. Additionally new writing prompts for the end-of-course writing tests may need to be field tested. If stand-alone prompt field tests are proposed, plans must address grade levels targeted for the field testing and potential incentives for school divisions to choose to participate if voluntary. |
Readers scoring the field test responses must be experienced with the rubric used to evaluate the responses to the writing prompts, and must have had training in scoring. Readers scoring the responses to the field test prompts shall also complete a questionnaire about the prompt. Scorer training materials must be reviewed and approved by VDOE. The scoring process must include validity approval by VDOE assessment staff.
Field tests for writing prompts should be designed to produce 500 student papers per prompt.
| 1.4.4 | The plan must address strategies for field testing additional constructed response items as the decisions to include these new item types are made, including in Computer Adaptive Testing. | |
| 1.4.5 | The Offeror shall conduct appropriate statistical analyses on all field test items. The field test data analyses shall be completed in a timely manner, i.e., so that decisions about the use of items are made in time for the building of test forms in the next test administration cycles. At a minimum, the statistical analyses of field test items will include: | |
| • | Item p-values | |
| • | Point biserial and/or biserial correlations | |
| • | DIF bias analyses | |
| • | Item calibration values (as previously mentioned, existing items in have been calibrated using the Rasch model). | |
| • | Frequency of scores at the domain and total level for writing prompts including a graphic representation of the scores at the total level. | |
| • | Frequency distribution spreadsheets which indicate TEI answer option(s) entered or selected, the frequency and the frequency percent of the answer. | |
| • | Other data as may be specified by the VDOE, such as item and person fit statistics |
All of these data are to be maintained with the items and included in data review for consideration by educator committees.
| 1.4.6 | The Offeror must monitor TEI/open-ended/constructed response items during field testing for unexpected potentially correct reponses and consult with VDOE on potential scoring changes through an adjudication process, and must re-run item statistics if keys are added before items become operational. |
| 1.4.7 | The Offeror must provide the VDOE an annual report on the results of new item review and data review to include an analysis of item attrition and recommendations as to needed changes in the item development process and plan for the following year. |
| 1.4.8 | Cognitive Labs. As new item types are considered during the course of this contract, the Offeror shall provide a plan for the purpose and use of cognitive labs with small groups of 8-10 students per relevant grade level or small pilots to try out new item types. Items should be formatted into sets that are easily administered; the pilot tests or cognitive lab sets are not to be considered as full-length models of the final assessment instruments. Students should be interviewed as a part of the cognitive lab. A report on the cognitive labs/pilots shall be produced; the report should document the cognitive strategies students used to interact with the test materials, and if they had any difficulties with the vocabulary, the items, the stimulus materials, or the response formats. It should also indicate whether the items elicited the intended cognitive thinking and skills. The proposed solution should include plans to complete all arrangements for the pilot test or cognitive lab including preparation of item sets, selection of the sample, contacting the school authorities, preparing administration materials, distributing materials, administering the items, interviewing the students, scoring the results, and reporting the findings to the VDOE. |
The VDOE reserves the right to approve or modify the plan for the pilot test or cognitive lab procedures.
| 1.5 Test Design | |
| 1.5.1 | The Test Blueprints describe the structure of each test, show the total number of test items, the number and percentage of items, and the number of score points for each reporting category measured by the test. Test Blueprints for the SOL test are developed for each grade and content area with the adoption of new standards, typically every seven years. The current SOL test blueprints may be found on the VDOE website on the Test Blueprints page at https://www.doe.virginia.gov/teaching-learning-assessment/student-assessment/virginia-sol-assessment-program/test-blueprints. |
The Offeror shall use the current test blueprints for each SOL test unless and until the VDOE notifies the Offeror blueprint revisions or new blueprints are required.
1.5.2 When new content standards are adopted (typically every seven years) and at other times as needed, upon notification by the VDOE, the Offeror shall lead the development of the new or revised test blueprints. All blueprints must be approved by the VDOE prior to their adoption and use in the test development process. Virginia educators must be involved in the development and review of the blueprints, at minimum through the item review committees.
Each blueprint shall:
• clearly define the standards to be assessed;
• provide the reporting structure, including the grouping of standards within reporting categories/claims;
• delineate the number of items and points for each reporting category/claim as well as the percentage of the total test represented by each category;
• specify the number of operational and field test questions
• identify the types of items to be included on each assessment;
• provide the basis for selecting items for test forms;
• define the test length and overall characteristics;
• delineate the intended cognitive levels addressed, using Webb's Depth of Knowledge framework unless otherwise specified; and
• include a variety of item types, and specify the percentage of questions that may be technology-enhanced items, and on certain tests the number or percentage of questions that will be "non-calculator" items.
| 1.5.3 | When new content standards are adopted (typically every seven years) and at other times as needed, the Offeror shall lead the development of ancillary materials (such as mathematics formula sheets) to be used on test forms and posted publicly to share with educators. Ancillary materials must meet the same accessibility standards as items. | |
| 1.5.4 | The Offeror shall conduct formal reviews of test blueprints and ancillary documents using the item review committees after any adopted revisions to content standards. Review at other times may be conducted at the discretion of the VDOE and may or may not result in changes. | |
| 1.5.5 | Each item must be have a comprehensive set of metadata that includes the following requirements: | |
| • | identify the content standards measured to the sub-bullet level, including the grade level and subject area, and may include alignment to multiple sub-bullets, sets of standards, or multiple versions of those standards. | |
| • | identify the type of item (i.e., multiple-choice, technology-enhanced, or constructed response); | |
| • | have sufficient flexibility to allow for re-coding an item when the SOL are revised or when a decision is made that the item better measures another content standard; and | |
| • | allow the historical tracking of an item that has been re-coded, used on a linear test form, been revised by a committee for use on special test forms (and has a "sister" item), and the time of release/do not use (DNU). | |
| • | at certain grade levels specified by the VDOE, mathematics items are to be designated as calculator active or "non-calculator", with a calculator yes/no designation to be maintained in the metadata. | |
| • | for items that have been field tested and gone through data review, identify the item descriptor aligned to the content measured by the item - see Exhibit A: Requirements 1.2.17 below for more detail. | |
| • | include item difficulty (high/medium/low) or estimated item difficulty for new items prior to field testing. |
Each new item developed must also inlude the following additional metadata:
| • | Cognitive complexity level, using Webb's Depth of Knowledge framework | |
| • | Performance Level Descriptors - the expectation for the performance level applicable to the standard | |
| 1.5.6 | SOL Grades 3-8 Mathematics CAT Algorithm. The Offeror shall develop and implement a plan to: | |
| • | develop item level computer adaptive testing (CAT) algorithms for each of the SOL Grades 3 – 8 Mathematics tests that delivers a fixed length individualized online test to each student | |
| • | use the test blueprint and the SOL content constraint files (provided by VDOE) to ensure the algorithms for Grades 3 – 8 Mathematics tests comply with parameters established by VDOE such as: | |
| o | match of delivered tests to test blueprints | |
| o | total number of items delivered; | |
| o | number of technology-enhanced items; | |
| o | spread of content coverage based on the test blueprint and the constraint files; | |
| o | number of items with/without the use of a calculator; | |
| o | limits to overall item exposure; | |
| o | blocking of items between student test attempts as determined by the VDOE. | |
| 1.5.7 | SOL Grades 3-8 Reading CAT Algorithm. The Offeror shall develop and implement a plan to: | |
| • | develop passage-based computer adaptive testing (CAT) algorithms for each of the Grades 3 – 8 Reading SOL tests that deliver a fixed length individualized online test to each student | |
| • | use the test blueprint and the SOL content constraint files (provided by VDOE) to ensure the algorithms for Grades 3 – 8 Reading tests comply with parameters established by VDOE such as: | |
| o | match of delivered tests to test blueprints | |
| o | total number of items delivered; | |
| o | total number of technology-enhanced items | |
| o | spread of content coverage in compliance with the test blueprint and the constraint files; | |
| o | limits to overall item and passage exposure; |
This is the start of the file's text. The full file is on GovTribe.
File details come from the government source that posted it. Updated .