Amendment 1_Continuation sheet_Answers.pdf

SF-30 CONTINUATION SHEET

AMENDMENT NUMBER One (1)

NATIONAL CENTER FOR ADVANCING TRANSLATIONAL SCIENCES (NCATS)

REQUEST FOR PROPOSALS (RFP) NUMBER: 75N95023Q00223

PROJECT TITLE: LitCoin Foundational Knowledge Graph Generation

AMENDMENT NUMBER: One (1)

DATE OF ISSUANCE: July , 2023

The above numbered RFP is amended as set forth below. The hour and the date specified for receipt of proposals remains unchanged. The hour and date specified for receipt of questions remains unchanged. Offerors must acknowledge receipt of the amendment prior to the hour and the date specified in the RFP or as amended, by one of the following methods:

I. By acknowledging receipt of this amendment on each copy of the proposal submitted. Please note that this is the preferred method.

II. By separate letter or email which includes a reference to the RFP and amendment number.

FAILURE OF YOUR ACKNOWLEDGMENT TO BE RECEIVED AT THE PLACE DESIGNATED FOR THE

RECEIPT OF PROPOSALS PRIOR TO THE HOUR AND DATE SPECIFIED MAY RESULT IN REJECTION OF YOUR PROPOSAL. If by virtue of this amendment you desire to change a proposal already submitted, such change may be made by letter, provided each letter makes reference to the RFP and this amendment, and is received prior to the opening hour and date specified.

The purpose of this amendment is to 1) provide responses to questions received concerning the above numbered RFP and 2) make changes to the above numbered RFP.

A. This amendment provides responses to questions received concerning the above numbered RFP as stated below.

INTERESTED OFFEROR QUESTIONS

GOVERNMENT REPSONSES TO

INTERESTED OFFEROR QUESTIONS

INTERESTED OFFEROR QUESTION #1:

Are there any specific requirements that need to be followed for this response similar to the prior RFI?

Specific requirements that we expect: 1) The natural language processing algorithm/system used to generate the knowledge assertions to populate the graph will utilize the software that NCATS acquired through the LitCoin NLP Challenge, and NCATS will provide this software. 2) The Biolink ontological framework will be used for the knowledge graph.

INTERESTED OFFEROR QUESTION #2

1. Are there any guidelines or limits on the hourly rate or components (wages, overhead, general and administrative expenses, and profit) of the hourly rates?

2. What are the requirements for keeping time sheet records and their submission?

1) No within scope of SOW

2) Software created for this project will be required to be open source and NCATS must be granted a broad license to use, modify, redistribute, etc in perpetuity.

3) No.

3. Suppose the commercial software is approved by Government (according to section Data Rights in the Statement of Objectives). Can we add the licensing cost of the software developed and sold by us as the direct cost of the project?

4. We have an old Certifications and

Representations that we used for another NIH submission; can we use that an update it?

5. Since this is a software product that will not have any user interface, how should companies address the 508 requirements, is a statement sufficient?

4) Is there a standard for page limits for this kind of submission? I have no real preference, though of course shorter is easier to manage.

5) There is a requirement for a simplistic user interface which HEAL researchers can use to verify or adjust knowledge assertions. It is likely that this user interface will need to conform to 508 accessibility requirmements.

INTERESTED OFFEROR QUESTION #3

Q

Question RFP

Location

1 Could NIH please provide the anticipated start date of the period of performance?

Statement of

Objective

2 We would like to inquire what factor(s) in the Integrity level of the data is considered

Moderate when per the requirement No PII collected and the

Confidentiality and

Availability levels are both Low.

Terms and

Conditions

3 Is the ORCID component of the web interface meant to pull in a list of publications for that author?

Statement of

Objective

4 An SF 1449 was not provided with the solicitation. Will an SF

1449 (or similar) be required to be signed

Addendum

FAR

52.212-4

1) We expect this project to take one year, with an optional second year based on the needs of the project.

2) I do not have this information.

3) The ORCID is meant to pull a list of publications associated with that author, correct. We would also need to have the ability to put in a PubMed ID in the case that one or more of their papers was not correctly associated with their ORCID.

4) At time of award

5) If available, yes.

INTERESTED OFFEROR QUESTIONS

and submitted with offers?

5 Should past performance information be submitted as a separate attachment? Will

CPARs be sufficient for past performance information?

Combined

Synopsis/

Solicitation

INTERESTED OFFEROR QUESTION #4

1. In the "Scope of Work" section of the SOO document, the provided scope pertains solely to the one-time development of the NLP System and Knowledge Graph. It does not encompass the ongoing maintenance requirements for the developed system, such as updating the Knowledge Graph with newly published research papers in the future, applying security patches, or addressing bug fixes. We kindly request confirmation that the ongoing maintenance of the system is considered out of scope for this project and will be handled separately by NCATS.

2. In the "DELIVERY OR DELIVERABLES" section of the SOO document, specifically referring to item #3 (Foundational knowledge graph), we seek clarification on whether the development of a new web interface that enables users to view the knowledge graphs is considered part of the required base period scope (within the first year). We kindly request confirmation on this matter for further clarification.

3. We would like to inquire about the evaluation of the model performance. It was mentioned that the first stage of the project is to generate a 70% accurate knowledge graph. Will the Contractor perform this evaluation, or will it be done by NIH?

What percentage of the data will be used to perform this evaluation?

4. Regarding the cloud computing infrastructure, will NCATS provide a cloud computing space for the deployment of the knowledge graph for production and for the development

1) That is correct, ongoing maintenance of the system is out of scope.

2) A web interface to view the full knowledge graph is not required, but the ability to view and either confirm or suggested changes to knowledge assertions from specific publications will be required during the base period.

This system should allow a researcher to input their ORCID and view all knowledge assertions from publications they have been associated with, or to input a specific PubMed ID to do the same for a single publication (in the case that a publication was not correctly connected to their ORCID).

3) The evaluation of data accuracy will be performed by the contractor, and NCATS reserves the right to independently evaluate this at any time to verify it. The amount of data to be used for these evaluations will be negotiated between NCATS and the contractor.

4) NCATS will provide Amazon Web Services infrastructure for the project, with a minimum of three environments in which to perform the work.

5) Correct, if your organization qualifies as a small business, you are exempt from the small business subcontracting requirement. This applies to organizations that do not qualify as a small business.

INTERESTED OFFEROR QUESTIONS

environments, including Dev, QA, and Staging environments, during the development phase of the project?

5. Regarding the Terms and Conditions document, we have come across a clause that states “the submission of a small business subcontracting plan by other than small business offeror(s) is a requirement as a part of the proposal submission process” (If the proposed contract exceeds a total estimated cost of $750,000). Our understanding is that if our organization qualifies as a small business, we are exempt from submitting this small business subcontracting plan. Could you kindly confirm if our interpretation is accurate?

6. Do we need a Quality Assurance Plan (QAP) for the proposal, or this is only needed after the award?

7. When could we expect to hear about the decision after we submit our proposal?

6) A QAP should be included with the proposal. Additions or modifications to this plan are bopth possible and likely after the period of performance begins, based on the needs of the project.

7) Expect about 30 days from close of solicitation.

INTERESTED OFFEREOR QUESTIONS #5

Name of Doc Section STS Question

75N95023Q00223-

RFQ

of 7

1.) How many past performance examples are we required to submit?

75N95023Q00223-

RFQ

of 7

2.) Do the Past

Performance examples we submit have to be within a certain period of time

(e.g., within the past 3-5 years)?

General

General 3.) Are there any layout/format requirements and should the answer be structured in any certain way (e.g., in accordance with the

1. If available at least, three otherwise what is available.

2. Ideally.

3. No preference for layout or format of the required documentation.

INTERESTED OFFEROR QUESTIONS

Performance

Objectives under the SOW section of the Statement of Objectives?)

INTERESTED OFFEROR QUESTIONS #6

1) Can non-American Citizens/Residents be part of the project or is this open to only American Citizens/Residents?

2) Can we use overseas resources or do all resources have to be in the United States? If the resources can be overseas, is there a preference for US versus overseas resources?

3) The required output is one big knowledge graph containing the knowledge from the 5000 approx. studies. Is this correct?

4) Could the required NLP refined algorithm come from a potentially multi-stage open NLP competition in a similar style as the first LitCoin challenge? The 8 base ones could be provided to the participants. Or are you looking for just resources on Time and Material (T&M) basis to work under your direction?

5) How is the “80% of the knowledge” measured? Will the generated knowledge graph be compared to existing 5000 knowledge graphs containing this knowledge?

6) After the 80% accuracy is reached, further modifications with the government’s team correspond to curations of the outputed knowledge graph or are this curations to the NLP algorithm?

7) Assuming the July 17th date submission date stays firm, when is a decision expected on the choice of vendor?

1. Non-US citizens/residents may be included as part of the performant team.

2. Any overseas resources will likely require approval by NCATS and possibly the US State Department.

Therefore, it is preferred that overseas resources be kept to a minimum, but they are not prohibited.

3. The required output is the knowledge graph as stated, as well as APIs to access and add additional knowledge to the graph. In addition, a simple user interface will be required that will allow researchers to view and either confirm or suggest changes to knowledge assertions from publications they’ve authored or otherwise contributed to.

4. The implementation of the NLP algorithm itself is outside the scope of this solicitation, as it will be worked on by NCATS, though this work will be in close collaboration with the contractor in order to assure compatibility, etc.

5. We expect that the accuracy of the knowledge assertions will be validated through a small amount of manual curation on subsets of the data. This validation will be done by the contractor, and NCATS reserves the right to perform similar validation studies at any time to verify the accuracy.

6. After reaching the needed accuracy, further curation will likely be performed at both of these stages, but the contractor’s portion of the work will be complete.

7. Anticipate at least 30 days.

(End of Answers to Questions to the RFP)

B. This amendment revises the above numbered RFP as stated below.

RFP Remains the unchanged from Amendment 1 aside from what has been answered above.

INTERESTED OFFEROR QUESTIONS

INTERESTED OFFEROR QUESTIONS #7

- Can the NLP system be a proprietary software license and not be subject to posting source code?

1. The implementation of the NLP system itself is outside the scope of work, as it will be generated by NCATS using the output of the LitCoin NLP Challenge, but in close collaboration with the contractor to ensure compatibility, etc.

It will be developed as open source code.

INTERESTED OFFEROR QUESTIONS #8

• A “Description of Objectives” document specifies: “… using one (1), or the combination of any of the eight (8) NLP system from the winners of the LitCoin NLP Challenge…”. Our question is: will those “NLP systems from the winners of the LitCoin NLP Challenge“ be provided to the Contractor that wins the contract?

• Will there be guidelines on how “graph accuracy” is calculated, or/and examples of “expected” graphs that should be extracted from a small number of documents?

• If Contractor uses some proprietary content to train or fine-tune machine learning models, is it allowed under this contract to deliver to deliver only trained models, without delivering proprietary content that was used to train/fine-tune them?

• How many users are expected to be able to use Web Interface and API “simultaneously”, for inst. in terms of “typical requests per second”?

• Are there restrictions/requirements on infrastructure where Web Interface and API will run – number of servers, memory, CPUs, using GPUs, etc.?

1. During the review of these questions, program realized that this objective was supposed to be changed and was not updated before publication. NCATS plans to generate the NLP system from the output of the LitCoin NLP challenge themselves, with collaboration and consultation from the Contractor during that process. We are working on rectifying this oversight now.

2. Accuracy should be determined by comparison of a subset of the knowledge assertions to those generated through manual curation.

3. Use of proprietary content during the period of performance could be allowable, but at the end of the period of performance, code will need to be fully open source.

4. We expect no more than a few hundred users at a time on the web interface, as this would be limited to the HEAL investigators and their research teams. For the API, it would be good to have a plan for expansion to many simultaneous users, but initially also a few hundred at most.

5. NCATS will be providing AWS cloud space for this project. The specifics of the hardware made available will need to be worked out between NCATS and the contractor during the period of performance.

(End of Summary of Changes to the RFP)