CPS-21131-25-0590 Questions and Answers.pdf

CPS-21131-25-0590 Questions and Answers

1. On page 5 of the SOO, under item 7.D. Artificial Intelligence and Machine Learning Support, it states “The Contractor could seek to propose services for building AI/ML models for identifying relevant EHR records that involve consumer products and/or coding NEISS records from EHRs (as an alternative to these being built entirely by CPSC staff).

These services could be used to accelerate the use of AI/ML being built by CPSC. Any such proposal must provide the agency means of evaluating model performance (i.e. the ability to compare EHRs to the resultant coded NEISS records). Among the options for consideration might be the contractor playing a supporting role for the agency in construction of its own AI/ML models and/or converting the EHR data from its original native form to one more suited to AI/ML modeling.”

We see no guidance in the evaluation criteria nor in the budgeting guidelines for how a contractor, in its response, should propose this work. Should it be proposed and budgeted as an optional activity?

Answer: Consistent with our stated objectives CPSC is looking for solutions that will ingest electronic health records with a consumer product injury nexus. As part of that effort, it may make sense for the solution provider to ingest health record data from NEISS records as well to accelerate model training. Any AI/ML services can be proposed as optional or it can be priced as part of a total solution, or it can be omitted entirely. We will consider any such proposals.

2. Could CPSC please provide the two PDFs embedded in the SOO?

Answer: Those two documents were the “Authorization-to-Operate” for Microsoft Azure and General Support System Cloud. Upon award of a contract, the vendor will be provided with these documents as necessary to complete their tasks.

3. If the solution is hosted in Azure or AWS, is it considered FedRAMP compliant?

Answer: Azure, and yes, it is FedRAMP compliant.

4. Does CPSC have a budget target or range for this RFQ?

Answer: CPSC will not disclose budget information for this requirement.

5. Section 7.D of the SOO contemplates the Contractor providing services in the area of AI/ML Support, which (per Volume 1: Contract & Pricing Documents), are to be built-up to a FFP using contractor-provided labor hours and labor categories. To allow consistent Best Value comparison between bidders, is there a common number of labor hours (e.g., a 40 hour work week) that bidders should propose a FFP for?

Answer: It is up to the contractor to propose a price level that meets their solution.

6. Ref. page 3: ““The objective of this SOO is to provide the Commission with timely (i.e., within 10 business days) consumer product injury data from hospital emergency departments…” Will CPSC accept proposals that include partnerships with data aggregators, health systems, or Health Information Exchanges (HIEs) to satisfy the requirement for access to emergency department EHRs?

Answer: Yes.

7. Ref. page 3: “The objective of this SOO is to provide the Commission with timely (i.e., within 10 business days) consumer product injury data from hospital emergency departments…” What is the expected format of data delivery (i.e., HL7, CDA, JSON, CSV, etc.)?

Answer: The agency is open to multiple potential delivery formats. We would prefer to be told in the proposal what format(s) is/are available.

8. Ref. page 3: “CPSC seeks to reach a similar or larger scale in terms of the records the agency obtains annually, however, it does not require contractors meet that number of records for their proof of concept project.” Is there a minimum number of data sets acceptable for the proof of concept? A minimum number of existing NEISS hospitals? A minimum from non-NEISS hospitals? A minimum from non-emergency department settings?

Answer: There are no minimums on hospitals, though more existing NEISS hospitals is desirable by the agency. We seek enough data to be able to train AI/ML, so any solution that makes use of fewer than 10,000 EHRs is likely insufficient for that purpose and substantially more than 10,000 EHRs is desirable by the agency.

9. The solicitation does not unequivocally state what Offerors must provide; instead the solicitation lists information and services that an Offeror “could” provide (i.e., “The Contractor could provide prospective EHR data from non-emergency department settings,” “The Contractor could propose a solution using AI/ML models…”). ...This could result in unequal interpretations by Offerors, potentially undermining the fairness and consistency of the evaluation process. Please clarify.

Answer: This is a Statement of Objectives (SOO)– offerors are encouraged to provide solutions which will allow CPSC to meet the objectives stated in the SOO. CPSC will not dictate the solution vendors must provide. Multiple solutions are likely acceptable and could meet the objectives. CPSC has included multiple possible component solutions, although other solutions outside of those may also meet the requirements. It is up to the vendor to demonstrate how their quoted solution will meet the requirements as stated in the SOO.

10. What level of access to Electronic Health Records (EHRs), especially clinical notes, will the contractor be granted? will this be via a standardized API (e.g., FHIR), secure file transfer, or on-prem integration?

Answer: CPSC is seeking, not providing, the EHRs. It is the offeror's responsibility to determine how they will either provide the data to CPSC or gain access to it in order to utilize AI/ML. If the proposed solution requires access to CPSC data systems, you can assume that the awarded vendor(s) will be provided with government-furnished equipment and network access. Should your solution require access to CPSC data systems, please pay particular attention to “LC 10 Background Security Checks performed by the Contractor” and sections (a) and (d) of “LC 37 Security and Privacy”.

11. Is the contractor expected to establish API-based integration with specific EHR platforms (e.g., Epic, Cerner)? If so, are any vendor-specific APIs (such as Epic’s App Orchard) already authorized or expected to be part of this pilot?

Answer: The contractor is expected to either provide the agency EHRs and deliver them to CPSC in a manner that meet legal requirements for delivery of EHRs or obtain the EHRs in a manner that would allow them to code the NEISS record using the clinical notes. An API is not essential from CPSC’s perspective, but is an acceptable method for delivering EHRs.

12. Ref. page 5: “These services could be used to accelerate the use of AI/ML being built by CPSC.” It is unclear how CPSC intends to have the Contractor involved with AI/ML aspects of this contract. Does the government expect the contractor to deliver a fully developed AI/ML model for identifying and coding injury data, or is a hybrid model where CPSC leads modeling with contractor support expected?

Answer: Consistent with our stated objectives CPSC is looking at solutions where an offeror might deliver fully developed AI/ML models for identifying and coding injury data as well as any number of hybrid approaches where CPSC would work with the offeror to develop the AI/ML models.

13. To ensure continuity with NEISS historical data, will CPSC provide labeled examples or guidance for aligning AI-coded records with legacy NEISS standards and variables?

Answer: Yes, NEISS data is published to the public and so all coded NEISS records are available. The NEISS coding manual additionally describes how coding is done by hospitals. This is also publicly available. Further, we have a case number that we share with NEISS hospitals that allows for traceability between a coded NEISS record and hospital records. Where allowable, we expect to be able to connect the individual NEISS record to the EHR and thus the record can be used as training data for AI coding.

14. Are there specific computing or hosting environments preferred or required for AI/ML processing (e.g., AWS GovCloud, on-prem, FedRAMP-compliant cloud)?

Answer: The proposed solution must be hosted in a FedRAMP-compliant cloud environment. The agency's primary cloud services provider, which supports all current AI/ML initiatives, is Microsoft Azure.

15. Will the contractor have direct access to the NEISS proprietary coding software or an API version for integration/testing of automated coding?

Answer: It can be made available if your proposed solution requires it.

16. Can the contractor propose and use proprietary or open-source NLP/ML tools (i.e., spaCy, BioBERT, ClinicalBERT) for identifying consumer product-related injuries in unstructured clinical notes?

Answer: Yes, provided their use would not violate the security and privacy requirements in the contract.

17. Ref pages 5-9: “Separate scoring would be done for AI/ML services as shown in Performance Objectives I-a and I-b.” The solicitation references “Performance Objectives I-a and I-b.” Please confirm that this reference should be Performance Objectives I-b and II-b.”

Answer: Confirmed. It should be I-b and II-b.

18. Ref. page 6: “After the initial proof of concept phase, the electronic health records are provided within 10 days of the emergency department visits.”

“The timeliness rate is the % of cases received within 30 days of treatment.” Performance Objective I states the Performance Threshold is EHR provided within 10 days of the ER visit.

The How Rated portion states the timeliness rate is % of cases received within 30 days of treatment. Please clarify.

Answer: In terms of evaluating the proposal, we are seeking a 10 day delivery, since that resembles the current timing. However, once awarded, we provide some grace for records that may take longer than desired by the agency. Records older than 30 days become a problem for the agency in closing NEISS at the end of a data collection year.

19. Two provisions included in the solicitation – 52.204-26 and 52.227-15 – require Offerors to complete them. In which volume should Offerors include these completed provisions?

Please confirm that they will not count against page limits.

Answer: These may be included in the Price or Past Performance volumes, which have no page limit.

20. The instructions state that Offerors should not “combine all three volumes as one attachment.” The instructions then further describe Volume I – Price and Volume II – Technical Quote. The instructions do not describe a third volume. Please clarify if Offerors should submit two or three volumes. If three, please clarify what should be included within this third volume.

Answer: CPSC inadvertently left off the Volume 3 Past Performance instructions. The amended SOO is attached and include this volume.

21. The instructions for the Technical Quote state that Offerors should include pricing information (“A resulting cost to CPSC per record…”). Please confirm that Offerors should include pricing information within their technical quote.

Answer: This section has been moved to Volume 1 – all pricing information must be included in Volume 1.

22. Are there any restrictions on how long ago an Offeror may reference a previous contract (i.e., within the last 3 years)?

Answer: Please see the amended SOO which includes the instructions for Volume 3, Past Performance.

23. We would like to know if this is a brand new contract OR if there is (was) an incumbent performing these services. If not brand new, could you please provide the current / previous contract number?

Answer: This is a new contract. There is no incumbent.

24. Page 48 states that Volume 2 has a 20-page limit. Does this include cover, cover letter, table of contents, appendices, and references?

Answer: The cover, cover letter, and table of contents is excluded from the page limit. Any appendices and references must fit within the 20-page limit.

25. Page 47 mentions three volumes, however on page 48 only Volumes 1 and 2 are mentioned. What is considered Volume 3?

Answer: CPSC inadvertently left Volume 3 Past Performance out of the instructions. See the amended SOO which now includes the instructions for that volume.

26. Page 51 discusses Past Performance as an Evaluation Factor. Which volume should contain the Past Performance, and will it count towards the page count?

Answer: Please see the amended SOO, which now includes Volume 3 Past Performance.

27. Please identify whether all proposed technical activities must occur within the CPSC cloud instance or whether it is acceptable to interface with other FedRamped solutions before the data ultimately resides in the CPSC cloud instance.

Answer: Proposed technical activities can interface with other FedRAMP-compliant solutions, provided their use would not violate the security and privacy requirements in the contract.

28. Is there any incumbent/recompete or this is a brand-new project? If yes, could you please provide us with their award number or even if the customer bought similar services in the past.

Answer: This is a new project.

29. Does CPSC anticipate scaling this program beyond pilot program? Long-term initiative across the full NEISS hospital network or additional healthcare settings?

Answer: CPSC anticipates this program may go through a full base year and four option periods, if the pilot is successful. These options have been included in the period of performance for this requirement.

30. Can it be made clear whether there are any formatting or structural requirements for the quote submission such as volume breakdowns, page limits, and/or required sections?

Answer: All volumes must have formatting that is legible. Page limits are included in the Instructions for Offerors, specifically for Volume 2.

31. Should technical capability responses be limited to a specific number of pages or sections (e.g., cover letter, methodology, past performance)?

Answer: Quotes should follow the format in the Instructions to Offerors section of the SOO. The technical portion of the quote is limited to 20 pages.

32. Understanding that this is a proof-of-concept effort, is there any available guidance or precedent on the approximate funding range CPSC anticipates for pilot projects of this nature to help ensure realistic and well-scoped responses?

Answer: CPSC will not provide budget information for this requirement.

33. Can CPSC clarify how responses will be evaluated beyond the technical performance metrics?

Answer: Responses will be evaluated according to the evaluation section of the Instructions to Offerors.

34. If CPSC awards multiple contracts, will awardees be expected to coordinate (e.g., on data standards, reporting formats), or will each operate independently with its own data and architecture?

Answer: Awardees would work independently from one another, though the agency would allow the CORs to permit information sharing where feasible to advance the proof of concept.

35. Does CPSC plan to provide training data or should offerors demonstrate model performance using synthetic datasets or simulated NEISS-style records to illustrate feasibility?

Answer: NEISS data is the training data, however the agency does not possess the corresponding EHRs, so the data would be constructed by merging the two (likely using the case number CPSC uses for identifying records to NEISS hospitals).

36. Should we assume CPSC is anticipating vendors to provide a solution that will bring relevant EHR data to facilitate the objectives of the pilot program?

Answer: Yes.

37. Should offerors propose complete solutions independently or is CPSC envisioning collaborative approaches involving multiple partners?

Answer: Collaboration with non-CPSC parties is not required. However, it is permissible, provided legal requirements for accessing EHRs are met.

38. Are there any specific interoperability standards that offerors should keep in mind when proposing a solution?

Answer: No single standard is mandated. However, during the market research phase, the agency identified the following EHR standards: Fast Healthcare Interoperability Resources (FHIR), Health Level Seven (HL7), Consolidated Clinical Document Architecture (C-CDA), Integrating the Healthcare Enterprise (IHE), and the 21st Century Cures Act.