91990024R0008 All Questions with Responses 8.7.24.xlsx
XLSX spreadsheet 37 KB Posted
- Solicitation number
- 91990024R0008
About this file
This document contains a set of questions and responses related to a federal solicitation for the National Assessment of Education Progress (NAEP) Assessment Content Team. The solicitation, numbered 91990024R0008, is for three contract tasks under the NAEP Coalition of Teams: Task 4 - Assessment Content Development, Task 5 - Scoring, and Task 6 - NAEP Platform Development.
The key details include: the solicitation is for a one-year base period with four one-year option periods; the anticipated award date is November 2024; the government is seeking solutions to accomplish the requirements outlined in the Statement of Objectives (SOO); pricing approaches such as firm fixed price and time and materials are acceptable; the government owns the source code for the existing eNAEP platform and is open to alternative solutions; and there will be a transition period between the incumbent and new vendors. The questions cover topics such as pricing, platform requirements, content development, research studies, accessibility, and performance data.
View the file
Other files
Show all 30
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
Sheet1
| Number | Questions | Technical/Business | Reference to RFP | Page # | Responses |
| 001 | On page 65 of the solicitation, it states that Task 6 is to be FFP. However, some tasks in the SOO (eg.., Task 6.1.1) state that it should be FFP, while other tasks (e.g., Task 6.1.3, 6.1.4) do not indicate that. Is the Government's intention that all of Task 6 should be FFP? | Business | Task 6 | 65 | The references to Firm Fixed Price (FFP) in Task 6 subtasks were recommendations from the program office. It's up to the offeror to propose the pricing approach they deem most appropriate for each task/subtask. The references to FFP have been removed in the amended Attachment A- NAEP Assessment Content Team (SOO) to avoid confusion. |
| 002 | On Page 79 Section M, Evaluation criteria, we understand and acknowledge the numerical evaluation factors, but are there any additional weighting factors beyond what is listed in descending order? | Business | Section M | 79 | It is important to note that these are not numerical evaluation factors, this is purely a numbered list and the numbers are of no weight. There will be no other evaluated factors than what has been listed in the evaluation criteria section. |
| 003 | In several places throughout the SOO, there are references to "a copy of the Scoring Memo is available on the Bidder's Page." Please provide the website address to the Bidder's page and/or please provide a copy of the most recent Scoring Memo. | Technical | Attachment A Task 5 SOO | 73 | Copies of recent Scoring Memo's will be posted on sam.gov under the solicitation as a separate Attachment. |
| 004 | Can offerors provide the schedule of task activities and delivery of products as an appendix to the technical volume that is excluded from the page limits? | Business | Instruction to Offerors | 69 | Yes, schedule of task activities and delivery of products can be included as an appendix and do not count to page limits. |
| 005 | Task 6.3.4 on page 130 of the Task 6 SOO, Task 8.1.1 on page 49 of the Task 8 SOO, and Task 7.3.2 on page 7 of the Task 7 SOO all speak to help desk responsiblities differerntly. To help vendors define the appropriate technical solution and pricing, can the government please provide specific details about the delineation of help desk team responsibilities for Task 8 and Task 6? | Technical | Attachment A (Task 6, 7, and 8) | (130,49,8) | These Tasks each represent different contracts and the SOOs cover different requirements that need to be met based on that Task's needs. It is also important to note that only Task 6 is part of this solicitation. Task 6 includes technical help desk support for the platform and supporting systems--including support for schools and districts completing the pre-assessment activities and technical troubleshooting on the assessment day as specified in the SOO. How Task 6 might work with the other NAEP contractors with help desk responsibilities should be considered in offerors' proposals. As an example, the existing NAEP Help Desk utilizes a tiered approach, and the Platform contractor is accountable for Tiers 2 and 3 technical support, as well as supporting Tier 1 with prepared responses and Knowledge Based Articles (KBAs) for easily responding to common technical issues and frequently asked questions. Additional resources on past help desk ticket volumes will be posted for potential bidders on sam.gov under the solicitation. |
| 006 | Please provide a recent version of the Task 6 Usability Study Plan. | Technical | Attachment A Task 6 | 136 | Usability Study Plan requirements are outlined extensively within Subtask 6.4.2 of the SOO. |
| 007 | Please confirm that the direct costs for computing and data storage hosting for the DBA Platform (and related systems) are to be budgeted by the Task 10 (WDOMS) contractor and not the Task 6 (NPD) Contractor. | Business | Attachment A Task 6 | Gen | Task 10 (WDOMS) hosts all NAEP system and data storage production environments including those within Task 6. The Task 6 contractor is responsible for all direct costs associated with systems development including solution components and associated services. |
| 008 | Could the government please share an example of a record of development for a released cognitive item as well as for a survey questionnaire item? | Technical | Attachment A | 9 and 25 | Task 4.2.3 and task 4.3.3 of the Statement of Objectives provides details of what should be included in the cognitive and survey questionnaire record of development. |
| 009 | Could the government share an example of a survey questionnaire issues paper? | Technical | Attachment A | 19 | An example of a survey questionnaire issues paper will be posted on sam.gov under the solicitation as a separate Attachment. |
| 010 | Could the government share an example of a survey questionnaire development plan? | Technical | Attachment A | 19 | Task 4.3.2 of the Statement of Objectives provides details of what should be included in a survey questionnaire development plan. |
| 011 | The stated NAICS code for the applicable Task Content (TC) found on the first page of the solicitation in section 10 is 541720 - Research and Development in the Social Sciences and Humanities. |
1. Can offerors who are classified under the 541720 NAICS code propose a small business concern as a subcontractor(s) whose primary code is not 541720?
| Business | Page 1 #10 | The prime must be registered under the applicable NAICSs code listed in the RFP. Refer to page 74 of the RFP for additional guidance. | |
| 012 | The stated NAICS code for the applicable Task Content (TC) found on the first page of the solicitation in section 10 is 541720 - Research and Development in the Social Sciences and Humanities. |
| 2. Can offerors who are classified under the 541720 NAICS code propose subcontractors not classified as a small business concern whose primay code is not 541720? | Business | Page 1 #10 | The prime must be registered under the applicable NAICSs code listed in the RFP. Refer to page 74 of the RFP for additional guidance. | |
| 013 | Since this acquisition is unrestricted and there is no set-aside, do the limitations on subcontracting contained in FAR 52.219-14 apply for a services contract (e) (1)? |
| Do the limitations apply only to subcontractors who have the 541720 NAICS code or do they also apply to subcontractors who do not carry the 541720 NAICS code? | Business | Page 1 #10 | Yes the limitations to subcontrating in FAR 52.219-14 (e)(1) are applicable to all subcontractors. | |||
| 014 | NAICS Code 541519 Other Computer Related Services appears in the RFP. Can offerors who are not classified as a small business concern propose a small business subcontractor(s) whose primary code is not 541519 ? | Business | Page 58 L4 | NAICS 541519 only appears in RFP under clause 52.212-1 as a situational event and the NAICS 541519 is not applicable to the acquisition of services. | ||
| 015 | Are the two annual scoring debriefing and planning meetings intended to be onsite or virtual? | Technical | Attachment A | Page 90 | The government prefers what provides most flexibitly to ensure and maximize participation by the contractor and the govenment. This meeting should incorporate a virtual option if possible. | |
| 016 | Was Section # 2.40 intentionally omitted from the table? | Business | Attachment A | Page 104 | Yes, the current table that is included is correct. | |
| 017 | Will we be receiving a list of Contract Options to cost? | Business | Gen | Options are identified in the Statement of Objectives, potentional bidders should cost those task identified as such. There will not be a list of options. | ||
| 018 | Section IV. Volume I - Technical Proposal Overview on page 67 states that no additional appendices, outside of a resume appendix, letters of commitment appendix, and sample research reports appendix, will be accepted. Please confirm this is accurate and bidders may not include supplementary appendix material such as company quality plans or information security plans. | Technical | RFP | Page 67 | Those items identified on page 67 are what may be included as appendicies. | |
| 019 | We acknowledge the section: Achieving a Goal of Diversity at All Levels of Staffing. Are there any other specific diversity hiring requirements outside of the standard EEOC inclusion criteria? | Business | Attachment A | Page 77 | Please refer to page 30 clause- X (34) (i) 52.222-26, Equal Opportunity (SEP 2016) (E.O. 11246). (ii) Alternate I (FEB 1999) of 52.222-26. | |
| 020 | In reference to iii and iv, are subcontractors responsible for paying the fringe rate or can they have an exemption? | Business | Attachment A | Page 52 | No exemptions will be issued. | |
| 021 | Which FedRAMP impact level is required under the current version of the Department Information Security and Privacy Requirements (e.g., High, Moderate, Li-Saas)? | Technical | Attachment A | Page 21 | The FedRAMP impact level required will most likely be moderate for cloud-based software, dependent on which system will store the information. | |
| 022 | FAR Clause 52.227-17 Rights in Data-Special Works states in paragraph C(2) that the Contractor shall not incorporate in data delivered under this contract any data not first produced in the performance of this contract, unless the Contractor is granted written permission by the Contracting Officer. |
If there is data delivered under this contract that is not first produced under this contract, and such data is not incorporated with data first produced under this contract, will the Contractor retain exclusive ownership rights over such data? Business RFP Page 24 If data is delivered under this contract that is not first produced under this contract or not incorporated with data from this contract then the Contractor retains ownership rights over such data.
023 FAR Clause 52.227-17 Rights in Data-Special Works states in paragraph C(2) that the Contractor shall not incorporate in data delivered under this contract any data not first produced in the performance of this contract, unless the Contractor is granted written permission by the Contracting Officer.
| In circumstances described above where data is delivered that is not first performed under the contract, but is not incorporated with data first performed under the contract, is permission necessary to deliver the data under these circumstances? | Technical | RFP | Page 24 | Yes, permission from the Contracting Officer would still be required. |
| 024 | FAR Clause 52.227-17 Rights in Data-Special Works states in paragraph C(2) that the Contractor shall not incorporate in data delivered under this contract any data not first produced in the performance of this contract, unless the Contractor is granted written permission by the Contracting Officer. |
| If the Contractor wishes to assert ownership rights in its pre-existing intellectual property or data, what method or steps are necessary to assert them in order for the Contractor's rights in the intellectual property or data to be properly recognized by the Government? | Technical | RFP | Page 24 | In accordance with FAR 52.227-17(c)(2), a contractor may not incorporate any pre-existing intellectual property, including any data not first produced in the performance of this contract, in data delivered under this contract unless the contractor first (1) identifies such intellectual property to the Department in writing; (2) enters into a written license agreement with the Department that grants the Department, and others acting on its behalf, a paid-up, nonexclusive, irrevocable, worldwide license/sublicense to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, such intellectual property by or on behalf of the Department (3) receives the Contracting Officer’s prior written permission to incorporate such intellectual property. The license agreement between the contractor and the Department must include a warranty by the contractor that the contractor has all rights in the intellectual property that are necessary to grant to the Department the license identified above. | ||
| 025 | The following Volumes that are in blue font appear to be broken hyperlinks in the RFP. If these are hyperlinks, can you please provide access to these documents and any other hyperlinked documents? Team Volume- Team proposal, Volume I. Technical Proposal, Volume II. Past Performance, Volume III. Business Proposal | Business | RFP | Page 64-65 | The blue font was not intended to be a hyperlink. | |
| 026 | Please clarify if the offeror should include large and small business subcontractor, consultant, and/or contractor pricing in its proposal submission? | Business | RFP | Page (23,70,73,74) | The offeror should include all cost relevant to their proposal. | |
| 027 | Is the offeror required to complete the Time Commitments table for all named subcontractor staff or solely consultants? | Business | RFP | Page 70 | Offerors should only complete this table for Key Personnel Staff. | |
| 028 | Since there is not a set aside small business goal and if the offeror does not need to utilize large or small business subconctractors not consultants/contractors to fulfill any of the requirements of one of the tasks, and plans to do so with its own resources and staff, is the offeror still required to utilize small business? If yes, what would be considered "substantic participation" to receive and "excellent" rating based on Exhibit M.1 (page 84), since no large subcontractors will be utilized? | Business | RFP | Page (1, 84) | Refer to page 74 of the RFP for the small business participation requirements | |
| 029 | Task 6 seems to indicate that the successful vendor will have no role in supporting the existing NAEP systems portfolio, but rather only the DBA and supporting systems that the vendor implements under Task 6. Would the government please confirm whether this interpretation is correct? | Technical/Business | Att. A - SOO, Task 6 | 110 | As stated in the SOO, the successful vendor for Task 6 will be responsible for implementing, supporting, maintaining, and enhancing NAEP's digitally based assessment (DBA) Platform and supporting technology systems delineated in the Task 6 SOO. This may include the existing NAEP systems portfolio (fully transferrable to the vendor) or alternative solutions proposed and implemented by the vendor as long as they are comparable to the existing NAEP systems portfolio. | |
| 030 | In the absence of detailed requirements and specifications, it would only be possible to provide the government with a firm fixed price (FFP) based on a complex and largely arbitrarty set of assumptions and dependencies that could vary wildly between bidders, thus rendering a meaningful comparison extremely difficult if not impossible. Would the government consider releasing this as a Time and Materials (T&M) acquisition? | Business | Att. A - SOO, Task 6 | N/A | Although firm fixed pricing is preferred, it's up to the offeror to propose the pricing approach they deem most appropriate for each task/subtask. This would include alternate pricing structures such as T&M. | |
| 031 | Task 6 and the referenced attachments provide at best high-level functional descriptions of the applications and their interrelationships. Significantly more detail as to functional requirements, process flows, anticipated user and data volumes, and current levels of support tickets (categorized by type and severity) would be necessary to model the application in sufficient detail to enable pricing a solution with the associated level of effort. Will the government please provide that level of detail, either in the request for proposal or in a reading room available to bidders? | Technical | Att. A - SOO, Task 6 | N/A | The government is looking for bidders to provide recommendations based on the context provided in the SOO, NAEP's goals, and emerging trends, technologies and innovations. Task 6 of the SOO, Attachment K – eNAEP Overview, and Attachment L – AMS Overview provide details on the requirements and expectations for the DBA Platform and supporting systems to enable bidders to develop and cost a best fit solution for the future needs of the program. Additional information on NAEP’s digitally based assessments and the eNAEP Platform can be found at: https://nces.ed.gov/nationsreportcard/dba/ including the eNAEP Student Tutorials and Online Style Guide which include information and/or demonstrations of many of eNAEP’s features, tools, and UI elements. Information on DBA Platform user volumes (i.e., students taking the assessment in each assessment cycle) is made available to the public at: https://nces.ed.gov/nationsreportcard/. Approximate user volume for the current AMS system is 40,000 users. Language or links to other resources related to help desk ticket volume will be posted for potential bidders on sam.gov under the solicitation. | |
| 032 | Does the government expect bidders to include the hosting costs for the solution components, and the associated services for managing the hosting environment independent of the DBA platform and associated systems? | Technical | Attachment F - Pricing Template, Task 6 | 1 | The Government recommends offerors respond with all applicable prices associated with their proposals. Task 10 (WDOMS) hosts all NAEP systems production environments including those within Task 6. | |
| 033 | The language of the SOO implies that a comprehensive, detailed requirements document for the DBA exists. Would the government please provide vendors with access to this document to ensure that vendors have the necessary information to generate accurate and reasonable bids? | Business | Att. A - SOO, Task 6 | 110 | The goals and priorities of the NAEP program are outlined in the Statement of Objectives (SOO); bidders should propose solutions to accomplish the outlined requirements. Task 6 of the SOO and Attachment K – eNAEP Overview provide details on the requirements and expectations for the DBA Platform and supporting systems to enable bidders to develop and cost a best fit solution for the future needs of the program. Additional information on NAEP’s digitally based assessments and the eNAEP Platform can be found at: https://nces.ed.gov/nationsreportcard/dba/ including the eNAEP Student Tutorials and Online Style Guide which include information and/or demonstrations of many of eNAEP’s features, tools, and UI elements. | |
| 034 | How many students take assessments in a year? | Business | Att. A - SOO, Exhibit D | XV | This information is made available to the public at: https://nces.ed.gov/nationsreportcard/ | |
| 035 | How many assessments does each student take in an assessment cycle? | Business | Att. A - SOO, Exhibit D | XV | This information is made available to the public at: https://nces.ed.gov/nationsreportcard/ | |
| 036 | Could you provide more details on the specific goals and priorities within NAEP’s Strategic Plan that our research agenda should prioritize, especially regarding innovations in assessment development? | Business | Att. A - SOO, Section 4.6 | 34 | The goals and priorities of the NAEP program are outlined in the introduction section to the Statement of Objectives (SOO). The planning of the reseach agenda should examine current industry practices and innovations. | |
| 037 | What criteria does NAEP use to evaluate the alignment of potential innovations with its strategic plan and operational capabilities, particularly concerning the assessment schedule and impact on student trends? | Business | Att. A - SOO, Section 4.6 | 34 | Evaluation factors are defined in Section M of the RFP. | |
| 038 | How flexible is NAEP in adjusting the research agenda to accommodate emerging trends or findings that may significantly impact assessment development strategies? | Business | Att. A - SOO, Section 4.6 | 34 | The intent of task 4.6.2 is to evaluate and incorporate emerging trends in NAEP assessment content development. Bidders should propose a process for conducting landscape reviews on the latest trends and approaches in large-scale assessment design and development and incorporating recommendations from landscape reviews into research and development. | |
| 039 | Are there particular areas within the latest trends, technologies, and innovations in assessment design and development that NAEP is most interested in exploring through the landscape reviews? | Business | Att. A - SOO, Section 4.6.2 | 34 | The government is looking for bidders to provide recommendations based on the context provided in the overview of the SOO, NAEP's goals, and emerging trends, technologies and innovations. | |
| 040 | Regarding the pursuit of research to exploit technology for improved measurement and increased efficiencies, are there specific technological advancements or approaches NAEP is particularly interested in? | Business | Att. A - SOO, Section 4.6 | 34 | Task 4.6.3 of the Statement of Objectives provides potential topics to study. | |
| 041 | How does NCES prioritize potential topics for research studies, and how much flexibility do contractors have in suggesting and focusing on new areas of research? | Business | Att. A - SOO, Section 4.6 | 34 | It is up to the bidder to provide a solution to accomplishing the requirements outlined in the SOO, including recommendations for a robust research agenda that is also flexible enough to address emerging needs. | |
| 042 | For investigations involving data collection, could you provide more details on the expectations for participant recruitment, data privacy, and methodologies? | Business | Att. A - SOO, Section 4.6 | 34 | The type of study will dictate expectations for recruitment and methodolody. It is up to the bidder to propose the methodologies used for research studies that require data collection. | |
| 043 | Could you clarify the process and typical timelines for obtaining Office of Management and Budget (OMB) clearance for research studies that require it? | Business | Att. A - SOO, Section 4.6 | 34 | The process and timelines for obtaining OMB clearance will vary depending on the type of data collection. Some OMB requests under the ACD contract will fall under the "Generic or Developmental Clearance" category, which is used for cognitive, field testing, or user experience studies. The typical timeline for this clearance is 6 weeks and includes preparing the materials and reviews by NCES and OMB and does not require a public comment period. Other OMB rquests under the ACD contract will fall under the "Normal Clearance" review, which is used for operational data collections. The typical timeline for this request is between 3-months and 6-months and includes preparing the materials, reviews by NCES and OMB, and includes either a 30-day or 60-day public comment period. Any changes to the OMB materials will require additional time for review by NCES, OMB, and potentially a public comment period depending on the substance of the change. | |
| 044 | How does NCES envision the findings and recommendations from technical memos and investigation reports being utilized to inform NAEP’s assessment development and strategic planning? | Business | Att. A - SOO, Section 4.6 | 34 | It is up to the bidder to provide a solution to accomplishing the requirements outlined in the SOO. | |
| 045 | Could you share any challenges or obstacles encountered in research and development support during the previous contract period that NAEP aims to address or improve upon in this upcoming cycle? | Business | Att. A - SOO, Section 4.6 | 34 | The goals and priorities of the NAEP program are outlined in the Statement of Objectives (SOO); bidders should propose solutions to accomplish the outline requirements. | |
| 046 | What are the key elements or considerations that NCES expects to be included in the design and plan for each study? | Business | Att. A - SOO, Section 4.6 | 34 | Task 4.6.3 of the Statement of Objectives (SOO) provides details on what should be included in a study design plan. It is up to the bidder to provide a solution for accommplishing the requirements outlined in the SOO. | |
| 047 | Could you elaborate on the specific types of bias that NCES is most concerned about in automated scoring, and any preferred methods or approaches for monitoring and mitigating these biases? | Business | Att. A - SOO, Section 5.2 | 76 | The Goverment is committed to mitigating bias for all subgroups for which scores are provided on the Nation's Report Card. The Goverment is seeking bidders' recommendations for methods to detect, monitor and mitigate bias, based on their expertise, experience, and demonstrated success. | |
| 048 | How does NCES envision the findings from these studies being shared with or contributing to the broader large-scale assessment community? | Business | Att. A - SOO, Section 4.6 | 34 | NCES welcomes bidders to propose any and all ideas for this in their proposals. | |
| 049 | Could you provide more examples of the types of issues or questions that have arisen during work on NAEP’s platform and supporting systems that might be suitable for these studies? | Business | Att. A - SOO, Section 4.6 | 34 | The research studies included in task 4.6 should focus on the design and development of assessment content, although there will likely be a need for close collaboration with NAEP's assessment and questionnaire platforms. The ACD bidder should work closely with the NPD bidder to collaborate on any suitable studies related to assesssment content that requires the NAEP platform and supporting systems. | |
| 050 | What are the key elements or considerations that NCES expects to be included in the design and plan for each usability study? | Business | Att. A - SOO, Section 6.3.5 | 131 | Usability study requirements and expectations are outlined extensively within Subtask 6.4.2 of the SOO. | |
| 051 | What specific types of platform performance data is NCES most interested in analyzing, and how does NCES envision these data informing design and implementation decisions? | Business | Att. A - SOO, Section 6.3.5 | 131 | Expectations for platform performance data capture and analysis are outlined extensively withing Subtasks 6.3 and 6.4 of the SOO. | |
| 052 | How frequently does NCES anticipate usability studies will need to be conducted, and how flexible is this schedule based on findings and platform updates? | Business | Att. A - SOO, Section 6.4 | 135 | NAEP currently conducts a minimum of 4-5 usability studies a year depending on program areas of need and evolving systems capabilities. Findings are formally documented and reported out at least quarterly with summary reports provided to key stakeholders immediately after completion of each study. To ensure that all systems are providing users with sufficient access and ease of use, the contractor shall conduct frequent and comprehensive usability studies around the systems and any planned changes or enhancements prior to implementation. Bidders are encouraged to address their proposed frequency of study to best acheive these goals in their proposal; and proposed frequency of study is a requirement of the Usability Study Plan to be developed by the contractor. | |
| 053 | How does NCES envision the findings from these studies being shared with or contributing to the broader large-scale assessment community? | Business | Att. A - SOO, Section 6.4 | 135 | NCES welcomes bidders to propose any and all ideas for this in their proposals. | |
| 054 | Is the assessment content created new (from scratch) for each testing year, or is it based on previous tests but revised? | Business | Att. A - SOO, Section 4.2 | 4 | A portion of each assessment is carried over from from one administration to the next to support measuring trends; the other portion is newly developed. |
| 055 | How many team members are typically involved in the development of one test? (e.g., grade 4 reading) | Business | Att. A - SOO, Section 4.2 | 4 | It is up to bidders to determine a staffing model based on requirements outlined in the SOO. |
| 056 | Are best practices or common issues available to the new vendor based on previous content development? | Business | Att. A - SOO, Section 4.2 | 4 | The bidder should leverage current industry best practices and reflect appropriately in approach to the work. There will be a transition period to hand off previously developed content. |
| 057 | Universal Design is listed as a challenge. At this time are there any learners excluded from the test due to accommodations not being available? If so, please describe. | Business | Att. A - SOO, Section 4.2 | 4 | NAEP universal design features and accommodations are available to the public at https://nces.ed.gov/nationsreportcard |
| 058 | Does the platform have adaptivity functionality currently? Or would that need to be solutioned and supported with custom development? | Business | Att. A - SOO, Section 4.2 | 4 | The eNAEP platform will include adaptive functionality, although it will not have been tested and validated prior to the end of the current contract cycle. The ACD contractor will need to coordinate with the NPD contractor regarding implementation of any adaptive testing taking into the capabilities of the platform that will be used. |
| 059 | Presently, is data beyond performance data collected? If so, please describe. | Business | Att. A - SOO, Section 4.2 | 4 | As stated explicitly in the Task 6 SOO, both student response data and a wide-range of observable events (or “process data”) related to student interaction with the platform, items, and tools are collected for scoring and analysis. |
| 060 | Do you envision the questions being the same for PBA and DBAs or is variation appropriate to take advantage of the medium provided it is assessing the same content, skill, or level? | Business | Att. A - SOO, Section 4.2 | 4 | NAEP assessments are currently either in digital format or paper-based format (for those that have not yet transitioned to digital). |
| 061 | While Puerto Rico is explicitly mentioned, Attachment A suggests additional transitions for Spanish dialects are needed. Does Latin American Spanish include Puerto Rico? Or are there other dialects that will be required? | Business | Att. A - SOO, Section 4.2.9 | 14 | As described in section 4.2.9, translations into Spanish are needed for both Puerto Rico (for the mathematics assessment in Puerto Rico) and English-Spanish bi-lingual material for use in mainland U.S. |
| 062 | Related to but not exclusive to Spanish translations, what considerations are made for localization? As it relates to diversity, equity, and inclusion, students who do not have the background knowledge will be at a disadvantage if the assessment stimuli, prompts, or distractors do not consider aspects such as localization and vernacular. | Business | Att. A - SOO, Section 4.2.9 | 14 | Items are currenlty not adapted for localization, apart from adaptation as part of translation to Spanish, as described in the SOO. |
| 063 | Attachment A Page 26 Ancillary Materials - Should the DBA platform provide ancillary materials in a digital first manner? | Business | Att. A - SOO, Section 4.4 | 26 | It is up to bidders to propose a solution to accomplishing the requirements outlined in the SOO. |
| 064 | Presently are distractors designed to identify common misconceptions or error patterns? | Business | Att. A - SOO, Section 4.2.3 | 9 | The NAEP Questions Tools, available at https://nces.ed.gov/nationsreportcard presents, released items from past NAEP assessments. |
| 065 | What amount of feedback can be provided to the learner during the assessment? Is this presently baked into the assessment design and development? Or is there interest in expanding support to students? | Business | Att. A - SOO, Section 4.2.3 | 9 | It is up to bidders to propose a solution to accomplishing the requirements outlined in the SOO. |
| 066 | What amount and type of in-product support is provided to students? | Business | Att. A - SOO, Section 4.2.3 | 9 | As described in the SOO for Task 6, the student assessment experience includes a tutorial and instructional design components. |
| 067 | What amount and type of professional learning is provided to educators and those implementing the assessment? | Business | Att. A - SOO, Section 4.2.3 | 9 | NAEP assessments are currently administered by trained field staff external to assessed schools. |
| 068 | Will there be a transition period between the existing and new vendors? | Business | Overarching | N/A | Yes, there will be a set hand-off period for the transition to a new vendor as applicable. |
| 069 | Can you provide a detailed overview of the current platform that is owned by NAEP, including its primary functionalities and features? | Technical | Att. K - eNAEP Overview | N/A | Task 6 of the SOO and Attachment K – eNAEP Overview provide details on the current requirements and expectations for the DBA Platform to enable bidders to develop and cost a best fit solution for the future needs of the program. Additional information on NAEP’s digitally based assessments and the eNAEP Platform can be found at: https://nces.ed.gov/nationsreportcard/dba/ including the eNAEP Student Tutorials and Online Style Guide which include information and/or demonstrations of many of eNAEP’s features, tools, and UI elements. |
| 070 | Are you seeking a complete redevelopment, enhancements, or targeted improvements to the current platform? | Technical | Att. K - eNAEP Overview | N/A | The goals and priorities of the NAEP program are outlined in the SOO; bidders should propose solutions to accomplish the outlined requirements. As stated in the SOO, NAEP owns the source code for the eNAEP platform, and it is fully transferrable to a new contractor in support of this work. However, alternative options will be considered provided the proposed solution is open source, comparable to eNAEP in core features and technical specifications, adds value for the program over the current system, and is highly customizable for NAEP operational and content needs. Task 6 encompasses implementation of the platform and supporting systems for use in each assessment cycle and continuous improvement and innovation in response to program needs. |
| 071 | What are the specific technologies, development stacks, high-level diagrams, or frameworks that are currently used within the current platform? | Technical | Att. K - eNAEP Overview | N/A | The government is looking for bidders to provide recommendations based on the context provided in the SOO, NAEP's goals, and emerging trends, technologies and innovations. Task 6 of the SOO and Attachment K – eNAEP Overview provide details on the current requirements and expectations for the DBA Platform to enable bidders to develop and cost a best fit solution for the future needs of the program. Additional information on NAEP’s digitally based assessments and the eNAEP Platform can be found at: https://nces.ed.gov/nationsreportcard/dba/ including the eNAEP Student Tutorials and Online Style Guide which include information and/or demonstrations of many of eNAEP’s features, tools, and UI elements. |
| 072 | What are the gaps between the current requirements and the existing platform? Are there any existing features that require significant enhancements or modifications to meet the requirements specified in Task 6? | Technical | Att. A - SOO, Task 6 | N/A | The goals and priorities of the NAEP program are outlined in the Statement of Objectives (SOO); bidders should propose solutions to accomplish the outlined requirements. |
| 073 | What are the incumbent vendor's responsibilities in maintaining the platform? Are there specific areas where current maintenance is lacking or needs improvement? | Technical | Att. A - SOO, Task 6 | N/A | The current vendor is responsible for all subtasks identified within Task 6 for the DBA Platform and supporting systems. |
| 074 | What is the desired timeline for this project? Are there any critical milestones or deadlines we should be aware of? | Technical | Att. A - SOO, Task 6 | N/A | The timeline for Task 6 aligns with NAEP's overall assessment cycle timeline. See Attachment A, Exhibit D on SAM.gov: https://sam.gov/opp/c7c88ddd417b4a25906cf7e03f781be8/view |
| 075 | How many staff members are currently required to maintain and enhance the current platform? | Technical | Att. A - SOO, Task 6 | N/A | The goals and priorities of the NAEP program are outlined in the SOO; bidders should propose solutions to accomplish the outlined requirements. Bidders should account for in their proposal the number of staff members they anticipate are required to successfully complete the work outlined in the SOO. |
| 076 | Will there be a transition period between the existing and new vendors? | Technical | Att. A - SOO, Task 6 | N/A | Yes, there will be a set hand-off period for the transition to a new vendor as applicable. |
| 077 | In Figure 1-1 of attachment K, Key NAEP users and their interactions with the Next-gen Platform are outlined. What devices do admins and students use to access the Next-gen Platform? | Technical | Att. A - SOO, Task 6 | N/A | Currently, the NAEP DBAs are administered to students on NAEP-provided Chromebook devices using the eNAEP DBA Platform. However, the use of school-based equipment for NAEP administration is being pursued through a proof-of-concept study in 2024 and field test in 2025 with operational capability to administer the NAEP assessment on school-based devices planned for 2026. eNAEP employs a web-based application design currently downloadable and supported on Microsoft Windows and Chrome OS. Field staff administering and monitoring the assessment currently do so using a combination of NCES-owned SurfacePro and Chromebook devices. |
| 078 | For the section entitled Develop the Framework Implementation Plan (4.2.1) is there interest in AI tools that can support the evaluation and classification of content to new standards/frameworks? | Technical | Att. A - SOO, Section 4.2.1 | 5 | It is up to bidders to propose a solution to accomplishing the requirements outlined in the SOO. |
| 079 | For the Develop New Content and Scoring Rubrics (4.2.3), are the use of Generative AI/Human Hybrid tools of interest? | Technical | Att. A - SOO, Section 4.2.3 | 8 | The goals and priorities of the NAEP program are outlined in the Statement of Objectives (SOO); bidders should propose solutions to accomplish the outline requirements. |
| 080 | For the Develop New Content and Scoring Rubrics (4.2.3), is there interest in building scoring rubrics that could incorporate both human and AI scoring? | Technical | Att. A - SOO, Section 4.2.3 | 8 | The goals and priorities of the NAEP program are outlined in the Statement of Objectives (SOO); bidders should propose solutions to accomplish the outline requirements. |
| 081 | For new content and scoring rubrics (4.2.5), are there characteristics of existing pretest methodologies or protocols that NAEP would like to ensure are perpetuated with a new vendor? | Business | Att. A - SOO, Section 4.2.5 | 11 | It is up to bidders to propose a solution to accomplishing the requirements outlined in the SOO. |
| 082 | For the section entitled Support Scoring of Pilot and Operational Items (4.2.7), what specific rater data from the Scoring contractor be shared with the ACD contractor? | Business | Att. A - SOO, Section 4.2.7 | 13 | The ACD contractor will need to coordinate with the Scoring contractor on the support of scoring pilot and operational items. |
| 083 | For the section entitled Conduct post-data collection cognitive assessment quality review (4.2.10), there is reference to the development and maintenance of an item tracking system (e.g., tracking table). Are there specific data points, format or presentation designs that would be preferable? | Business | Att. A - SOO, Section 4.2.10 | 15 | It is up to bidders to propose a solution to accomplishing the requirements outlined in the SOO. |
| 084 | For the section entitled Develop and Maintain Content for the Survey Questionnaires (4.3), it is noted that NAEP is interested in innovations that leverage digital and technological approaches – are there particular limitations to the current survey process that NAEP is hoping to address with the introduction of new approaches? | Business | Att. A - SOO, Section 4.2.3 | 8 | The goals and priorities of the NAEP program are outlined in the Statement of Objectives (SOO); bidders should propose solutions to accomplish the outline requirements. |
| 085 | For the section entitled Develop a Research Agenda Aligned with NAEP’s Strategic Plan (4.6.1), are there sample content outlines, reports, or more detail that can be provided about the breadth and depth that is preferred within this report? | Business | Att. A - SOO, Section 4.6.1 | 34 | It is up to bidders to propose a solution to accomplishing the requirements outlined in the SOO. |
| 086 | For the section entitled Landscape Reviews (4.6.2), are there sample content outlines, reports, or more detail that can be provided about the breadth and depth that is preferred within this report? | Business | Att. A - SOO, Section 4.6.2 | 34 | Task 4.6.2 of the ACD SOO defines the requirements of the Landscape Reviews. It is up to bidders to propose a solution to accomplishing the requirements outlined in the SOO. |
| 087 | For the section Research and Special Studies (4.11.2.1), how frequently and in what volume has NAEP requested multi-contractor research studies, research projects as additional tasks, and quick turnaround projects in the last five years? | Business | Att. A - SOO, Section 4.11.2.1 | 55 | The Department will provide a chart with historical estimates for each size special study in an amendment to the RFP SOO. |
| 088 | Please provide clarification on what “inclusion of more contextual information in reporting,” refers to as an environmental factor that will influence NAEP during this period? | Business | Att. A - SOO, Procurement Overview | IV | An example is the COVID-19 pandemic and how NCES administered "COVID-19 items" in the NAEP questionnaires. NCES has reported results from some of these items (and may continue to) to help contextulize NAEP achievement. |
| 089 | Can you provide more detailed examples of what constitutes "minimal revisions" for the deliverables? | Business | Att. A - SOO, Section 6.11.1.2 | 140 | Core requirements are met with a limited number of changes needed to approve the deliverables. |
| 090 | Can you provide examples of past special studies that have been successful, and what made them stand out? | Business | Att. A - SOO, Section 6.11.2.1 | 155 | Please see this wbsite for some recent special studies: https://nces.ed.gov/nationsreportcard/tdw/data_collection/2019/study_special.aspx |
| 091 | What are the key innovation goals for NAEP that we should focus on in our proposals for targeted special studies? | Business | Att. A - SOO, Section 6.11.2.1 | 155 | These studies are unknown at this time as they are often the result of ongoing work, research, results, and/or reports. |
| 092 | “The contractor must use a local version of an LLM” - Does the requirement for using a local version of an LLM also imply that hosting must be on-premises, or does it permit hosting on a private cloud? | Technical | Att. A - SOO, Section 5.2 | 76 | When working with AI and LLMs, the Department of Education and its contractors are required to follow the Executive Order on the Safe, Secure, & Trustworthy Development & Use of Artificial Intelligence. https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/?utm_source=link |
| 093 | Will NCES please clarify how SBE dollars are counted towards goal? |
Do Tier II and Tier III subcontractors, if they are small business, count towards the small business subcontracting goals? Business RFP Page 84 For a prime that is a large business, all subcontracted dollars to a small business count towards the small business goal. Tier II and Tier II small business subcontractors count towards the small business subcontracting goals. Refer to page 74 & 75 of "91990024R0008-NAEP Assessment Content_V2_8.2.24" for additional information on small business subcontracting.
File details come from the government source that posted it. Updated .