Attachment_7_TO__2.docx

SENTINEL INITIATIVE

ATTACHMENT 7

TASK ORDER #2: INNOVATION CENTER TASK ORDER

STATEMENT OF WORK

Table of Contents

Project Title: Sentinel Initiative – Innovation Center (IC) Task Order

1. Introduction

In the Sentinel System’s Five Year Strategy[footnoteRef:2], FDA sought to refine the operating model for the Sentinel System, by “attracting novel external development through coordinated activity at a Sentinel System Innovation center… and use the process associated with re-competition of the Sentinel System operating contract to improve the Sentinel System operating model.” [2: https://www.fda.gov/downloads/Safety/FDAsSentinelInitiative/UCM629243.pdf ]

FDA conceptualized a future state that consisted of both a lead Operations Center, who would be the primary curator of the distributed database and continually enhance the analytic tools and one or more Innovation Centers, who would develop innovative methods to further evolve the Sentinel System. FDA outlined four important potential benefits to this new model:

1. Increasing and diversifying the number pathways for external investigators to engage with the Sentinel System for methods development or to leverage the resources for their own purposes

2. Freeing finite resouces at the Operations Center to manage increasing query volumes, improving efficiency and production speed, enhancing analytic tools, and accelerating novel data source acquisition

3. Providing alternative mechanisms for broadening capacity enabling growth in the quantity and breadth of questions that can be addressed in the Sentinel System

4. Attracting new data partnerships to improve system sustainability through continued diversification of the data network

The Innovation Center is designed to help FDA selectively incorporate innovative technologies, with a focus on natural language processing, data interoperability, and advanced analytics. The Innovation Center will serve as a test bed for emerging technologies that will be integrated into the Sentinel System by the Operating Center, or potentially lead to the development of new Analytic Centers.

The Innovation Center’s charge includes the following: clearly define the use cases where adoption of advanced analytics increases value, prioritize a set of demonstration projects along key dimensions such as cost, time horizon, feasibility and impact, and convert the analytics into scalable solutions within a multi-site distributed database environment. The work of the Innovation Center shall include, but not be limited to, strategic planning, proof-of-concept investigations, and demonstration projects that will help the Sentinel System improve over the long term.

2. Objectives This Task Order is issued under the Sentinel Initiative IDIQ contract in pursuit of the establishment and maintenance of the Innovation Center (IC). The IC will be responsible for identifying methodological gaps and developing methods to improve the utility of Sentinel by developing new Sentinel analysis tools and create new approaches to automate key epidemiologic study operations, such as chart review. To this end, this task order includes the following program tasks:

· Task 1: Innovation Center

· Task 2: Master Plan for Integration of Innovative Technologies

· Task 3: Innovation Projects and IC Collaboration Activities

3. Innovation Center (IC)

3.1. Establishment and Maintenance of the Innovation Center

The Contractor shall establish an Leadership Team to oversee all aspects of operations for the Innovation Center (IC). The IC Leadership Team shall create the organizational structure, administrative processes and governance for the IC and its network of scientific collaborators. The IC Leadership Team shall interface routinely with the FDA Sentinel Core Team, FDA Senior Leadership Team, Operations Center (OC) and Community Building and Outreach Center (CBOC). In collaboration with FDA, the IC Leadership Team shall participate in strategic development planning. The IC Leadership Team shall meet programmatic goals and oversee contract deliverables, finances and program management activities.

The IC shall build and maintain a diverse team with expertise in epidemiology, clinical medicine, pharmacy, statistics, health informatics, data science (specifically, artificial intelligence (natural language processing, machine learning)), network operations, and training to achieve the goals of this requirement.

The IC shall be capable of managing an estimated 1-3 new projects, depending on scope and size of each project. Projects will be intiated using Work Orders. The IC Leadership Team shall oversee all methods projects by:

· Ensuring access to a broad, diverse and deep pool of investigators from multiple disciplines to form project teams

· Creating and implementing transparent administrative processes to solicit proposals, and identify the most promising, feasible, executable, and realistic proposals to meet FDA’s objectives

· Ensuring that each project has access to the requisite data for analysis. This may entail leveraging existing data use agreements or establishing new agreements. The IC shall ensure that project teams have access to the local staff with deep knowledge of the data, particularly its provenance, limitations and clinical meaning

· Partnering talent from scientific disciplines not normally engaged in generating studies that can inform regulatory decision making, with personnel who have prior experience conducting drug safety or effectiveness studies

· Coordinating with the Operations Center to ensure that the new tools can function within Sentinel’s distributed database environment, if necessary

· Ensuring transparency, by posting programming code and technical documentation to allow other investigators to use the tools

3.2. Communication with Stakeholders and Scientific Collaboration The IC shall develop and maintain open communications with stakeholders and to promote scientific collaborations. In collaboration with FDA, the OC and the CBOC, the IC shall provide content related to IC activities for inclusion in the Sentinel Initiative website and online public developer community website. Such information might include study descriptions and protocols, study reports, published manuscripts, computer software, and sample datasets, as appropriate.

4. Master Plan for Integration of Innovative Technologies The IC shall develop a master plan for the prioritization, development and incorporation of innovative technologies into the Sentinel System. The IC shall include a set of clearly defined use cases for the adoption of analytics, a horizon scan of promising technologies for the Sentinel System, identification of one or more realistic, translatable and achievable technologies, and carefully survey data requirements and project risks. When relevant, the IC shall account for the frequency and depth of iteration required by the analytics with the source data, clearly define the approach to validating the methods and results, and acquire the necessary data to complete project. The master plan shall take between 9 and 18 months to develop and be jointly authored with FDA. The master plan shall outline a sequence of discrete, realistic, feasible and executable projects to test and develop one or more promising technologies within a five-year time frame, unless otherwise directed by FDA. The master plan shall be posted on the Sentinel website or result in a publication in a peer-reviewed scientific journal, unless otherwise directed by FDA. The master plan shall include approaches to innovation in the following domains described in Section 4.1 to 4.7:

4.1. Natural Language Processing (NLP) of Unstructured Data, Including Best Practices for Regulatory Use This technology can capture valuable clinical and patient information presented as unstructured text in EHRs and enrich structured data, such as for the insurance claims now available for analysis. Incorporation of data derived from NLP will enable the Sentinel System to identify previously undetected complex health outcomes, defined by multiple data elements, and directly enhance its abilities to assess patient safety and efficacy. Applications of NLP should ultimately enhance productivity during data collection and analysis. Staff using the Sentinel System will need to monitor the performance of NLP algorithms, as small changes in source data may require adjustments to ensure accuracy.

4.2. Advanced Analytics in Post-Market Safety and Efficacy

Machine learning can advance key elements of safety evaluations, including Health Outcomes of Interest (HOI) validation, propensity score matching and patient phenotyping. The technology may also help investigators evaluate or refine new HOIs that traditionally take significant manual effort. To realize this potential, the Sentinel System will provide an optimal learning ground where machine learning technology can be tested and improved, ensuring that it is used in scientific and responsible ways to evaluate drug safety and efficiency. This will require developing dependent technologies and methods.

4.3. Novel Primary and Secondary Data Sources

New clinical data sources are being introduced to the healthcare ecosystem quickly with the diffusion of once costly technologies, such as genome sequencing, and novel collection media such as wearables that capture huge amounts of data outside traditional healthcare settings. These additional sources can help improve all dimensions of data quality, including coverage, granularity and duration. FDA will work to understand their potential application and utility in safety monitoring and RWE generation for effectiveness. For example, incorporating genomics data may provide additional HOIs as endpoints, but using such data would require modifications to the Sentinel System common data model. Additional considerations for new data source integration include validation, security, confidentiality, privacy and legal concerns.

4.4. Interoperability Between the SCDM and Other Major Common Data Models to Selectively Supplement Data Breadth and Granularity The SCDM provides a common structure to enable the analysis of data across a variety of sources. To continue to broaden its scale, FDA might modernize the Sentinel System by harmonizing its SCDM with other established CDMs such as the Observational Medicinal Outcomes Partnership, PCORnet, Informatics for Integrating Biology at the Bedside (i2b2), and/or industry standards such as the American Medical Association’s Integrated Health Model Initiative, and link them with the Sentinel System data infrastructure. Exploring opportunities to enhance SCDM interoperability may become a long-term priority as alternative CDMs gain utility and evolve. As a part of exploring innovation in the Sentinel System context, an initial assessment of the broader CDM landscape, such as disease-specific CDMs, will help FDA establish objectives and a timeframe for evolving the SCDM.

4.5. Novel Technologies and Their Potential Implications for Sentinel System Infrastructure and Analysis The pace of technology innovation requires FDA to continuously monitor promising new technologies for potential applications. Blockchain, for example, is a digital ledger technology that makes records using cryptography. This may allow patients to permit access to their personal health data while maintaining their privacy. Some novel technologies are disruptive and garner intense interest. FDA will carefully assess their potential benefits, including efficiency gains and new capabilities. The Sentinel System can serve as a laboratory for exploring promising technologies. Exploration of small-scale experiments will enable an FDA-dedicated Sentinel System Innovation Center to ideate and quickly prototype promising technologies.

4.6. Novel Statistical and Epidemiological Methods for Improved Causal Inference and Enhanced Active Signal Detection in the Post-Market Setting While propensity score methods have become a standard in pharmacoepidemiology, statistical methods continue to be developed for estimating causal risk and methods for assessing potential bias in studies continue to advance. A key issue for the Sentinel System is deploying methods in the distributed data setting. Privacy-preserving distributed methods to link patient records across datasets, including the evaluation of different linkage approaches such as encrypted patient identifiers and algorithm-matching techniques across deidentified datasets may also be important. While the Sentinel System has transformed the way safety signals are assessed in the post-market setting, FDA still relies on non-Sentinel System sources such as company-generated signals, clinical trials, the Adverse Event Reporting System, medical literature and lay media to identify the first potential safety signals. Supplemental signal detection capabilities will enable the Sentinel System to mature as an integrated post-market safety surveillance asset. Current signal detection sources will continue to play an important role in the assessment of medical product safety, but the Sentinel System is uniquely positioned to fill a gap in the systematic analysis of large data sets to yield novel safety signals that may otherwise go undetected.

4.7. Improved Data and Methods for Using Real World Data for Generating Real World Evidence for Efficacy Sentinel and FDA-Catalyst are credible platforms for demonstration projects to provide insights or learnings that may inform the development of relevant expertise in RWE application. Given its experience, FDA-Catalyst will continue to serve as a development hub for important infrastructure that supports the Sentinel System, but with a focus on interaction or intervention. Continuing to advance methods to conduct randomized and non-randomized studies in the real-world setting will be important for FDA.

5. Innovation Projects and IC Collaboration Activities

Innovation Projects FDA will issue Work Orders to complete the activities outlined in the Master Plan, following its completion. The IC innovation projects are estimated to take 1-3 years each, with their own project teams, deliverables, and timelines. Any inter-project dependencies shall by planned and accounted for by the IC. The IC shall oversee all projects to ensure progression in a timely and coordinated fashion. The IC shall procure any necessary healthcare data (e.g., commercially available healthcare data such as EHR and claims data and clinical data collected by wearable devices, and patient-generated data) to test and develop methods and/or generate simulated datasets to achieve the deliverables defined by FDA. Such healthcare data could also include data from the Sentinel System network, in collaboration with the OC. For each of its proposed data resources, the offeror shall provide a detailed written narrative description and a Data Characteristics Table (See Attachment 2 – Data Characteristics Table).

FDA is not obligated to fund any specific project within the Master Plan and reserves the right to issue Work Orders for projects not listed in the Master Plan.

IC Collaboration Activities The IC shall work collaboratively with FDA, OC and CBOC to integrate the developed tools into the Sentinel production system. The IC in collaboration with the CBOC shall convene stakeholder groups across the healthcare ecosystem to seek expert input and shape new approaches to explore high-priority innovation use cases, conduct demonstration projects, publish results and share insights in the domains described above. The IC shall develop and implement a plan to effectively collaborate with the FDA and other Sentinel Centers (OC, & CBOC) in the development of a training program to train and update the FDA, OC, CBOC and Sentinel user community on the IC’s innovation activities.

6. Sample Innovation Projects The contractor shall provide brief proposals (max 4 pages) for the following innovation projects, including cost estimates. FDA may choose to fund these projects solely at its discretion.

6.1. Innovation Project 1: Improving Causal Inference in the Sentinel System Improving causal inference from observational data for purposes of evaluating safety and efficacy of medical products is an important goal of the Sentinel Initiative. Current approaches implemented in the Sentinel System include methods based on propensity scores for control of confounding. Targeted Learning has been proposed as an alternative and potentially better method (Van der Laan and Rose 2011).[footnoteRef:3] The goal of this project is to evaluate the potential use of Targeted Learning methods in the Sentinel System compared to existing approaches. The proposal should address the assessment of scientific validity of Targeted Learning methods compared with current approaches for evaluating medical product safety and efficacy in the Sentinel System, as well as practical aspects of the feasibility of how such approaches might be implemented in a distributed environment. [3: Van der Laan MJ, Rose Sherri. Targeted Learning. Causal Inference for Observational and Experimental Data. Springer, 2011.]

6.2. Innovation Project 2: Signal identification in Electronic Health Records (EHR) in the Sentinel System The FDA has explored signal identification in the claims data contained in the Sentinel Common Data Model using methods known as TreeScan and Drugscan. FDA recently launched an effort to integrate safety signal identification in the Sentinel System into routine use. An FDA-sponsored workshop was held to discuss current approaches and challenges (https://healthpolicy.duke.edu/events/implementation-signal-detection-capabilities-sentinel-system). A future with more EHR data available in the Sentinel System raises the question of how the more detailed and less structured data might be used for signal identification and whether such data and approaches offer improvement over current proposed approaches. The proposal should address the assessment of available approaches for signal identification in EHRs, how such approaches compare to existing methods, as well as practical aspects regarding feasibility of implementation of such approaches in a distributed environment.

7. Period of Performance