D1 Statement of Objective.pdf

STATEMENT OF OBJECTIVES (SERVICES)

(SOO)

GENERAL INFORMATION

Title of Project:

LitCoin Foundational Knowledge Graph Generation

Statement of Need and Purpose:

The Office of Drug Development Partnership Programs (ODDPP) at the National Center for Advancing Translational Sciences (NCATS) seeks to obtain professional services to develop a system that can generate knowledge graphs from the free text of PubMed abstracts, using Natural Language Processing systems, as part of the LitCoin program.

The LitCoin program’s overall goal is to create a new currency for scientific research, one which aligns with current technologies and the expectations of researchers generating research findings.

Biomedical researchers are constantly generating new data, but do not report a large portion of them in ways that make them accessible and reusable. For example, a study published in BMJ Open in 20181 found that 87% of survey respondents reported including supplementary material with their most recently-published article, yet such findings are rarely made available in a way that adheres to FAIR (Findability, Accessibility, Interoperability and Reuse) data principles. Part of the reason for this is the fit-for-purpose nature of most biomedical databases… generalized databases exist but are used far less often. Researchers lack incentives to share these types of datasets, and disincentives, such as fear of being “scooped” by other research groups, also exist.

LitCoin will attempt to solve these problems by creating a convenient process for authors to generate computationally-accessible knowledge from their published work, and by normalizing the “pre-curation” of natural language processing-generated knowledge by the authors themselves. The use of the term “knowledge” here is deliberate… we’re not just talking about data, but the conclusions that can be reached based on that data. Knowledge graphs contain statements of assertion, such as “Smoking causes cancer”, which are backed up by data that has led to the assertion. These knowledge assertions are the basis for building a complex, highly-connected knowledge graph upon which to build with LitCoin.

NCATS recognizes that such an undertaking may be best suited to a pilot within a specific field of biomedical research in order to demonstrate the feasibility of the project, and so have partnered with the HEAL (Helping to End Addiction Long-term®) Initiative2 to pilot the LitCoin system for research focused on pain, addiction, and related scientific areas.

The HEAL Initiative is investing in data science technologies and approaches to advance discovery in addiction and pain research. At this time, natural language processing of biomedical publication text is primarily performed post-hoc with a maximum accuracy of 60-70%; substantial effort is currently exerted in manual curation and review of the outcome data from this kind of process.

1 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6157527/ 2 https://heal.nih.gov/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6157527/ https://heal.nih.gov/

Background Information and Objective:

The National Institutes of Health (NIH) is the nation’s leading medical research agency and the primary Federal agency whose mission is to seek fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to enhance health, lengthen life, and reduce illness and disability, conducting, supporting and making medical discoveries that improve people’s health and save lives.

NCATS is one of 27 Institutes and Centers at the NIH and its mission is to catalyze the generation of innovative methods and technologies that will enhance the development, testing and implementation of diagnostics and therapeutics across a wide range of human diseases and conditions. ODDPP promotes innovations that improve the efficiency of drug development from target identification — including expanding the target landscape — through early-stage clinical trials.

With an ever-growing number of scientific studies in various subject domains, there is a vast landscape of biomedical information which is not easily accessible to the public, even though it is stored in open data repositories. Open scientific data repositories often lack the necessary metadata to make them sufficiently useful to most researchers, or simply are too vast to be explored fully. In order to make the most of these repositories, a consolidated linkage map that relates all scientific discoveries will be necessary. One potential way to build this map is to create highly-connected knowledge graphs that can transform open data repositories by increasing usability. These graphs can assist in identifying gaps in medical research and accelerating research for unexplored medical domains.

However, open medical data on its own is not enough to deliver its full potential for public health. Engaging technologists, members of the scientific and medical communities and the public in creating knowledge graphs from open data repositories, funders can exponentially increase data utility and value to help solve pressing national health issues. NCATS held a Stakeholder Feedback Workshop3 in June of 2021 to solicit feedback on this concept and its implications for researchers, publishers, and the broader scientific community, and used that feedback to build an overall plan for implementation of LitCoin. One important piece of feedback from that workshop suggested that implementation of such a potentially disruptive concept would necessitate a pilot program within a narrower research space than “all biomedical research”. This has led to a collaboration with the HEAL Initiative, with the hopes of capitalizing on the highly collaborative nature of research in pain, addiction, and related scientific areas to demonstrate LitCoin’s potential to impact the current biomedical data sharing landscape.

In preparation for the LitCoin Pilot Program, a highly-connected and highly-accurate knowledge graph representing the bulk of knowledge in the pain and addiction research space must be generated, which will form the foundation that will spur future research and new discoveries.

The generation of this graph will normalize having authors curate knowledge assertions generated from the text of their published research, empowering other researchers to amplify their findings and reuse their data in new ways.

3 https://sites.google.com/ncats.nih.gov/litcoin-stakeholder-workshop/home https://sites.google.com/ncats.nih.gov/litcoin-stakeholder-workshop/home

In 2021, NCATS performed the LitCoin Natural Language Processing (NLP) Challenge4 with the objective of (1) helping data scientists better deploy their data-driven technology solutions towards accelerating scientific research in medicine and (2) ensuring that data from biomedical publications can be maximally leveraged and reach a wide range of biomedical researchers;

together this will drive toward solutions for the critical problems these scientists aim to solve. As a result, eight (8) NLP systems that accurately capture the information denoted in free text and provide output of this information through knowledge graphs were obtained by NCATS.

As a result, the development of a system that can generate knowledge graphs from the free text of PubMed abstracts, using Natural Language Processing systems will allow HEAL researchers to expand the reach of their scientific discoveries, even those that normally would be difficult to publish through traditional means. This will normalize the pre-curation of research findings by the researchers who generated them, creating a highly-accurate knowledge graph which will contain the bulk of the knowledge in the pain and addiction research space.

Period of Performance:

The initial (base) period of performance for this task order is 12 months. There will also be one (1) option periods of 12 months, for a total possible period of performance of 24 months...

Contract Type and Ceiling Price:

The government intends to issue a time and materials (T&M) contract, and all proposals must be submitted on that basis. The not to exceed ceiling on this contract will be $1,123,300.00 for the base period of performance and up to an additional $1,123,300.00 million for one (1) option period. Offerors are expected to submit a proposal reflective of its solution to fulfill the government’s requirements

SCOPE OF WORK

General Requirements:

Independently and not as an agent of the Government, the Contractor shall furnish all the necessary services, qualified personnel, material, equipment, and facilities, not otherwise provided by the Government as needed to perform the Statement of Objectives below:

Performance Objectives:

The Contractor shall generate a 90% accuracy knowledge graph representing knowledge of approximately 5000 published papers in the HEAL research5 space to be deployed on NCATS cloud environment and be accessible to the scientific community by means of a web interface.

The following objectives will need to be fulfilled by the proposed work:

1. Natural language processing (NLP) system

4 https://ncats.nih.gov/funding/challenges/litcoin 5 https://heal.nih.gov/research https://ncats.nih.gov/funding/challenges/litcoin https://heal.nih.gov/research

a. By using one (1), or the combination of any of the eight (8) NLP system from the winners of the LitCoin NLP Challenge, or with its own NLP system, the Contractor shall deliver an NLP system which can recognize biomedical concepts in the free text of PubMed abstracts, create node-edge-node knowledge assertions between these concepts, and identify from those generated assertions the principal observation being reported in the abstract.

2. Knowledge Graph

a. The Contractor shall build a highly-connected knowledge graph from the generated and curated knowledge assertions with 90% accuracy of the knowledge at the completion of the services.

i. The Contractor shall develop a knowledge graph with at least 70% of accuracy before involving the Government on curation of the generated knowledge assertions.

ii. With at least 80% accuracy, the contractor shall provide the generated knowledge graph to NCATS and associated vendors who will perform curation on approximately 5% of the generated knowledge assertions in order to assure the expected accuracy rate of the knowledge.

iii. If the knowledge proves to be less than 75% accurate according to this curation effort, the Contractor shall to re-train the NLP system and re-generate the graph to improve its accuracy.

iv. After curation by NCATS at 80% accuracy of the knowledge assertions, the Contractor shall present the knowledge graph to the HEAL investigators for curation and review in order to improve the accuracy to the 90% goal.

3. Presentation to HEAL investigators

a. The Contractor shall convene at least two (2) virtual review sessions, during which HEAL investigators will be asked to review and suggest changes to the knowledge assertions generated from their abstracts. These virtual review sessions shall include a training/tutorial component to ensure participants are aware of the functionality and what kinds of curation they should and should not perform on the knowledge assertions displayed to them. In the unlikely event of low participation by HEAL investigators, the Contractor shall supplement the invitee list by inviting additional researchers from the pain and addiction research spaces to participate.

b. After each virtual review session, the Contractor shall review the suggested changes generated during the review session and make changes to the knowledge graph as needed. If needed, NCATS can provide additional subject matter expertise to help with cases where suggested changes are not clear.

c. After reviewing and adjusting the knowledge graph, the Contractor shall provide the updated knowledge graph to NCATS and associated vendors who will perform another round of curation on approximately 5% of the generated knowledge assertions to verify that the virtual review sessions improved the accuracy of the graph, with the goal of 90% accuracy of knowledge therein.

OPTIONS

Option Period One (1) (Contract Year 2): This option will extend the period of performance for an additional twelve months. During the option period, the Contractor shall perform the work provided in the following tasks:

4. Create web interface for subject matter expert curation

a. The Contractor shall deliver a user interface which will allow researchers to log in using their Open Researcher and Contributor ID (ORCID), or alternatively submit a PubMed ID, and will display to the researcher all the knowledge assertions generated from PubMed abstracts authored by that researcher. The interface shall allow the researcher to review each assertion and either confirm that it is accurate, or to make comments on how the assertion should be different, based on the text in their publication(s).

5. Application Programming Interface (API) for knowledge graph access

b. The Contractor shall build an API which will allow data scientists to access the knowledge in the generated knowledge graph. This knowledge graph will be deployed to NCATS’ cloud services and made available for public use.

6. Presentation to HEAL investigators

c. The Contractor shall convene at least two (2) virtual review sessions, during which HEAL investigators will be asked to use the web interface to review and suggest changes to the knowledge assertions generated from their abstracts.

These virtual review sessions shall include a training/tutorial component to ensure participants are aware of the functionality and what kinds of curation they should and should not perform on the knowledge assertions displayed to them. In the unlikely event of low participation by HEAL investigators, the Contractor shall supplement the invitee list by inviting additional researchers from the pain and addiction research spaces to participate.

d. After each virtual review session, the Contractor shall review the suggested changes generated during the review session and make changes to the knowledge graph as needed. If needed, NCATS can provide additional subject matter expertise to help with cases where suggested changes are not clear.

e. After reviewing and adjusting the knowledge graph, the Contractor shall provide the updated knowledge graph to NCATS and associated vendors who will perform another round of curation on approximately 5% of the generated knowledge assertions to verify that the virtual review sessions improved the accuracy of the graph, with the goal of 90% accuracy of knowledge therein.

GOVERNMENT RESPONSIBILITIES

The government will allow access to all systems needed to accomplish our goals. NCATS and HEAL staff will be available to the Contractor to provide transition-in support. NCATS will make previously-generated NLP systems available to the Contractor. NCATS shall furnish a public GitHub repository and cloud computing space for deployment of the knowledge graph. NCATS will work with vendors to do curation of the knowledge graph before and after virtual review sessions.

For information technology (IT) related areas, the NCATS Information Technology Resource Branch (ITRB) is responsible for the operating system(s), databases, applications, system architecture, server and the security infrastructure provided and/or hosted by NCATS, while the contractor will be expected to be knowledgeable about and support these tasks and/or responsibilities in collaboration with ITRB as specified in this SOO.

DELIVERY OR DELIVERABLES

The Contractor must deliver all developed code to a GitHub repository designated by NCATS on each day that the Contractor develops software code. The Contractor must also deposit documentation deliverables into a GitHub repository designated by NCATS.

Deliverables:

1. Proposed project plan

2. NLP System with source code

3. Foundational knowledge graph

4. Knowledge graph API with source code (Option Period 1)

5. Web interface with source code (Option Period 1)

6. Convene investigators (Base Period + Option Period 1)

Reporting:

1. Status Reports (Content and frequency to be determined)

REPORTING REQUIREMENTS

The project methodology will determine the recurring reporting requirements.

Constraints Generated knowledge graphs must comply with the BioLink ontological framework in order to ensure compatibility with other NCATS data science programs. All generated code and graphs will be deployed to NCATS’ cloud computing infrastructure, in accordance with NCATS security and operating requirements for deployment of software systems.

OTHER CONSIDERATIONS

Travel:

Travel will not be required of the Contractor during the period of performance.

Key Personnel:

The Contractor must designate both a Project Manager (PM) as Key Personnel for this project.

The PM will be a direct liaison to the COR and will be responsible for the supervision and management of all of the Contractor’s personnel. The PM must have a full understanding of the technical approach to be used by the Contractor’s development team and will be responsible for ensuring that the Contractor’s development team follows that approach.

Data Rights:

The Government will require unlimited rights in any data produced in the performance of this contract in accordance with FAR 52.227-14, Rights in Data – General (May 2014).

Consistent with FAR 52.227-14, which applies to this Statement of Objectives, the Government intends that all software and documentation delivered by the Contractor will be either public domain or subject to a world-wide, royalty-free, non-exclusive open-source license that allows it to be freely used, modified, and shared.

This software and documentation includes, but is not limited to, data, documents, graphics, code, plans, reports, schedules, schemas, metadata, architecture designs, and the like; all new open source software created by the Contractor and forks or branches of current open source software where the Contractor has made a modification; and all new tooling, scripting configuration management, infrastructure as code, or any other final changes or edits to successfully deploy or operate the software.

To the extent that the Contractor seeks to incorporate in the software delivered under this SOO any software that was not first produced in the performance of this SOO, the Contractor shall incorporate either software that is in the public domain, or free and open-source software that qualifies under the Open-Source Definition promulgated by the Open-Source Initiative. In any event, the Contractor must promptly disclose to the Government in writing, and list in the documentation, any software incorporated in the delivered software that is subject to a license.

If software delivered by the Contractor incorporates software that is subject to an open-source license that provides implementation guidance, then the Contractor must ensure compliance with that guidance. If software delivered by the Contractor incorporates software that is subject to an open-source license that does not provide implementation guidance, then the Contractor must attach or include the terms of the license within the work itself, such as in code comments at the beginning of a file, or in a license file within a software repository.

In addition, the Contractor must obtain written permission from the Government before incorporating into the delivered software any software that is subject to a license that does not qualify under the Open-Source Definition promulgated by the Open-Source Initiative. If the

Government grants such written permission, then the Contractor’s rights to use that software must be promptly assigned to the Government.

Section 508—Electronic and Information Technology Standards:

Section 508 compliance will be included.

Publications and Publicity:

Not applicable.

Confidentiality of Information:

No proprietary or personal data will be generated in the performance of this work. NCATS may provide information, such as natural language processing algorithms submitted as part of the LitCoin NLP Challenge, that may be subject to non-disclosure requirements during the contract period.