Attachment_6_TO__1.docx

SENTINEL INITIATIVE

ATTACHMENT 6

TASK ORDER #1: OPERATIONS CENTER TASK ORDER

STATEMENT OF WORK

Table of Contents

Project Title: Sentinel Initiative – Operations Center (OC) Task Order

1. Introduction The Food and Drug Administration Amendments Act of 2007 (FDAAA) required Food and Drug Administration (FDA) to collaborate with public, academic, and private entities to develop methods for obtaining access to disparate data sources and to validate means of linking and analyzing safety data from multiple sources.

In 2008, the Secretary of Health and Human Services (HHS) and FDA’s Commissioner announced the Sentinel Initiative, a long-term effort to create a national electronic system for monitoring FDA-regulated medical product safety in alignment with FDA’s mission to protect the public health. The Sentinel Initiative began with the Mini-Sentinel pilot in 2009. By 2012, Mini-Sentinel developed reusable programming tools for routine queries and completed the Mini-Sentinel Pilot in 2014. In 2016, FDA officially announced the full-scale Sentinel System operations and formal embedding of Active Post-Market Risk Identification and Analysis (ARIA) system into the regulatory decision-making process.

ARIA fulfills FDAAA’s mandate to modernize safety surveillance by capitalizing on electronic healthcare data and advanced epidemiological methods and expands the FDA’s capabilities of post-market surveillance for approved medical products. Now an integrated part of FDA’s regulatory review process, ARIA can also provide complex, comparative, multi-site studies using analysis-ready, curated data and preprogrammed, parameterizable analysis tools that give investigators analytic flexibility.

The Sentinel System and FDA-Catalyst are part of the broader agency-wide Sentinel Initiative. FDA Catalyst is a program that uses the Sentinel System’s infrastructure to expand FDA's ability to actively gather information about the performance of regulated medical products following FDA approval.

This requirement seeks to preserve the foundation and accomplishments of the current Sentinel System and FDA-Catalyst and expand in the areas of enhancement, innovation and modernization of its foundational elements.

Glossary of Terms Italicized (See Attachment 1)

2. Objectives This Task Order is issued under the Sentinel Initiative IDIQ contract in pursuit of the establishment and maintenance of the Operations Center (OC). The OC will manage key operational aspects of the FDA “Sentinel Initiative” to enhance the foundation of the Sentinel System (data infrastructure, operations and technologies) and FDA-Catalyst enabling FDA to conduct post-market safety studies.

To this end, this task order includes the following program tasks:

· Task 1: Operations Center

· Task 2: Distributed Data Network

· Task 3: Medical Product Analyses

· Task 4: Analytic Tools

· Task 5: Training

· Task 6: Medical Record Review

· Task 7: Methods Development

· Task 8: FDA-Catalyst

3. Task 1: Operations Center (OC)

3.1. Establishment of an Operations Center

The Contractor shall establish an enterprise-level Executive Leadership Team to oversee all aspects of operations for the Operations Center (OC) and the Sentinel Data Partners Network (DPN). The Executive Leadership Team shall create the organizational structure, administrative processes and governance for the OC and its network of scientific collaborators and data partnerships. The Executive Leadership Team shall interface routinely with the FDA Sentinel Core Team, FDA Senior Leadership Team, Innovation Center (IC) and Community Building and Outreach Center (CBOC). In collaboration with FDA, the Executive Leadership Team shall participate in strategic development planning. The Executive Leadership Team shall meet programmatic goals and oversee contract deliverables, finances and program management activities.

The OC shall leverage organizational partnerships to build and maintain a diverse team with expertise in epidemiology, clinical medicine, pharmacy, statistics, health informatics, data science (specifically, artificial intelligence (natural language processing, machine learning)), network operations, and training to achieve the goals of this requirement.

The OC shall obtain all required Institutional Review Board (IRB) approvals for activities that involve interventions or interactions with health plan members or providers (e.g., FDA Catalyst).

3.2. Posting Analytic Packages and Results

Under the direction of FDA, the OC shall establish the means to make publicly available all analytic results and key analytic packages to meet transparency objectives of the Sentinel Initiative. The analytic packages shall contain the specifications, parameters, and codes used in the analysis and be in an executable format to enable other investigators to directly replicate FDA Sentinel analyses on any dataset formatted in the Sentinel Common Data Model (SCDM). The OC shall work with the FDA to establish communication and transparency timelines and processes to facilitate the public posting of analytic packages and results, either through the Sentinel Initiative website, a Science and Technology Developers Community Public website, or another venue.

3.3. Sentinel Initiative Public Website

The OC shall develop and maintain a public website for the Sentinel Initiative to include content provided by the OC, IC, CBOC and other systems within the Sentinel Initiative (e.g., Biologics Effectiveness and Safety (BEST) system). The website shall be hosted in a Federal Information Security Management Act (FISMA) compliant environment and be routinely updated to meet FDA’s needs. The Sentinel Initiative website is a critical communication tool for industry and the public and is one of the primary vehicles used to meet transparency mandates associated with FDA policy objectives and user fee commitments.

The Sentinel Initiative website shall be compliant with Section 508 of the Rehabilitation Act, user friendly, and allow stakeholders to easily obtain updates on the Sentinel Initiative. The Sentinel Initiative website shall offer intuitive and effective search functions, the ability to broadcast or push new information to subscribers, and enable viewers to download and review presentations, results and other information. The assigned webmaster shall track and analyze website traffic to provide statistics on the location of end-users, page hits for each subpage, and other descriptive data.

The Sentinel Initiative website shall also enable interactive online features and analytics, when requested by FDA, using JavaScript, HTML5, Qlik sense or other relevant software (e.g., see FDA FAERS Public Dashboard as reference). All online material shall be reviewed and cleared by the Sentinel Core Team and other FDA staff (as needed), unless other arrangements are made. The OC shall provide ready access to any necessary professional-grade photos and graphic design software and expertise to meet communication needs. There shall be pre-production and production environments of the Sentinel Initiative website.

3.4. Science and Technology Developers Community Public Website The OC shall create and maintain an organizational structure that promotes scientific collaborations. To this end, the OC shall develop and maintain an online public developer community website, such as GitHub.com, that hosts analytic tool codes, a synthetic publicly available dataset formatted in the SCDM and includes appropriate user guides and support documentation. The OC shall collaborate with the IC, CBOC and other FDA initiatives to promote the development of a Sentinel scientific community, as agreed upon with FDA.

3.5. Website Data Management Plan

The OC shall establish a data management and support team to maintain the Sentinel Initiative website and the Science and Technology Developers Community Public website and develop a ‘Website Data Management Plan’ to routinely track and report on the clearance and posting of final reports, presentations, publications, analytic results and other documents online.

4. Task 2: Distributed Data Network Infrastructure

4.1. Formation and Management of Data Partner Network

The OC shall establish and maintain business partnerships and Data Use Agreements (DUAs) with different types of organizations including academia, healthcare providers, contract research organizations, and technology companies. The OC shall create an organizational and governance structure that effectively manages the data partnerships and the consortium of scientific collaborators. Electronic healthcare data shall be captured from at least 100 million patients to meet the mandate set forth in the FDA Amendments Act (FDAAA) 2007. The OC shall establish and maintain a DPN with access to the medical records from which the medical encounter claims were derived.

The OC, under the guidance of FDA, shall create policies and processes to support the wide variety of FDA needs, including:

· testing of analytic packages,

· drug use queries,

· inferential queries,

· RWE trials and studies of product effectiveness that may include interaction or intervention with patients or the participating healthcare systems,

· health outcome algorithm validation, and

· testing and evaluation of novel statistical methods.

The OC shall keep the Data Partners and Scientific Collaborators informed of current developments, regulatory outcomes and public health gains in medical product safety and effectiveness associated with Sentinel System and FDA-Catalyst analyses.

The OC shall establish communication among stakeholders within the Sentinel DPN as well as external entities disseminated via electronic newsletters, major events, and hosting Data Partner and Investigator Summits.

The OC shall develop and maintain the capacity to coordinate activities with external entities, such as:

· Reagan-Udall Foundation’s Innovation in Medical Evidence Development and Surveillance (RUF/IMEDS);

· US government agencies;

· International regulatory agencies;

· Other stakeholders at the direction of the FDA; and,

· Patient representative to the Sentinel System.

4.1.1. Data Source Requirements

The OC shall establish and maintain a reliable and efficient DDN to access a variety of clinical data sources. Healthcare data shall include commercially available data, public data (e.g., from state-based registries, or federally-held Medicare and Medicaid data sources), electronic health record data, and private healthcare data (e.g., national health insurers and integrated delivery systems).

The OC shall establish and maintain a DDN containing sufficiently large (at least 100 million patients) and demographically diverse longitudinal populations in the United States to enable FDA to study rare diseases (defined as affecting fewer than 200,000 people), rare outcomes (<10 per 100,000 person-years), and cover time periods that enable FDA to study both recently approved medical products and those that have been on the market for more than 10 years.

The OC shall establish and maintain access to longitudinal healthcare data on important special populations of regulatory interest, including infants linked with mothers, the elderly population, and the pediatric population.

The OC shall have access to medical records, particularly those that are linked to administrative claims. Medical records may be electronic health records (EHR), paper-based medical charts, or a combination. Datasets with extensive linkages between administrative claims data and EHR are highly desireable.

The OC shall provide a description of each of the data sources in the DDN including sufficient detail so that FDA has the ability to assess the potential contribution of each data source to the assessment of risk factors associated with the occurrence of medical events of interest. The OC shall develop, maintain and update a Data Characteristics Table describing each of its proposed data resource(s). See Attachment 2 – Data Characteristics Table. The OC shall provide yearly updates to the Data Characteristics Table.

The OC shall provide the means to conduct and support pragmatic clinical trials or other hybrid study designs in real-world settings and access to real-world data (RWD) to generate real-world evidence (RWE) for efficacy demonstration projects (FDA-Catalyst). This includes the ability to capture patient-generated data (e.g., via the FDA Mystudies Mobile App), patient-reported outcomes, and the ability to randomize patients within health plans.

The OC shall establish and renew DUAs to ensure continuity of critical Sentinel System functions and access to the following data sources:

· Longitudinal data from both private (e.g., national health insurers, integrated delivery systems) and public sources (e.g., Medicare data) to support ARIA analyses using Level 1, 2, and 3 tools.

· Medical records (paper and/or EHR) from the originating medical encounter.

· De-identified patient-level data formatted with the Patient Episode Profile Retrieval (PEPR) tool, when deemed necessary to follow up an important analytic finding or support algorithm development.

· Mother-infant linked dataset to support evaluations of the safety of medical therapies in pregnant women.

· Data in a format necessary to operate TreeScan and other signal detection analysis tools.

· In collaboration with FDA, the OC shall pursue access to rare disease registries and national and state registry linkages (e.g., National Death Index (NDI) database and Surveillance, Epidemiology, and End Results (SEER) Program registries).

The OC shall establish DUAs to ensure continuity of critical FDA-Catalyst activities conducted under IRB approval:

· Pragmatic clinical trials:

· Implementation of a Randomized Controlled Trial to Improve Treatment with Oral Anticoagulants in Patients with Atrial Fibrillation (IMPACT-AFib) and

· FDA-Catalyst COPD, Asthma, And Respiratory Disease Effectiveness (CARE) For 21st Century Cures: Feasibility Assessments for Comparative Effectiveness Studies

· Trials, noninterventional studies, and/or registries involving the collection of patient generated data (e.g., from mobile technology) and subsequent linkage to electronic health data or electronic research data sets:

· FDA-Catalyst MyStudies App Alignment with Pragmatic Trials and/or Registries.

4.1.2. Data Sources Management Plan

The OC shall develop and maintain a ‘Data Sources Management Plan’ to include: classification of data sources; data characterization, transformation, curation, and refresh schedules; plan to address DDN infrastructure security reporting requirements; routine schedule of technical environment assessments of storage and computational needs; and communication plan for data expansion activities, progress and timelines.

The OC shall develop a data source optimization plan by establishing a tiered data source classification approach weighing features including: cost, database size, ability to link to electronic health records and trace back to medical records (paper), and reliability of routine transformation into analysis-ready datasets. The purpose of this plan is to characterize the data sources to enable FDA to balance the cost and capabilities of the system to meet the analytic goals, including decisions about which data sources to use for analytic package testing, rapid study feasibility assessments, fine-tuning of cohort identification approaches, full-scale safety assessments, outcome validation, and other potential regulatory needs.

The plan shall account for relevant data use restrictions (e.g., administrative services only, traceability back to the patient medical record) or unique features (e.g., access to free text EHR, access to special disease registries, linkages to state-based death registries, laboratory or vital signs information, or other local resources at the data partner), and other important properties of the source data.

4.1.3. Data Characterization, Transformation, Curation, and Refresh The OC shall create a large, linked DDN by collaborating with each Data Partner to transform their raw source data into the SCDM format and quality check their data to ensure that it is analysis-ready. Although other common data models may be proposed for eventual use, the current analytic platform shall be adopted to capitalize on the government’s prior long-term investment in the SCDM analytic platform and to ensure continuity of operations at an acceptable level of performance. The OC shall leverage existing ETL code (Extract, Transform, and Load) to minimize disruption to services, and develop new code as necessary to incorporate different data sources. Each proposed Sentinel dataset ETL version shall be characterized and curated routinely to ensure consistency, validity, and model compliance. The OC shall make its data quality review and data characterization process publicly available on the Sentinel Initiative website in a thoroughly annotated document.

4.1.4. Distributed Data Network Infrastructure Security

The OC shall develop a DDN that is FISMA compliant and addresses the confidentiality and proprietary concerns of the participating data partners. The OC shall work with FDA on all necessary FISMA compliance reporting requirements, including routinely tracking and auditing user-access.

The OC shall distribute queries to data partners to be executed behind their firewalls. When data partners return aggregated results, they shall be combined across the various Data Partners in the Sentinel System. The OC shall implement secure query distribution software with appropriate access controls and activity enforcement. Each data holder shall maintain physical and operational control of its data and directly authorize and manage the use of its data for analyses.

The OC shall implement physical, technical, and administrative security features into its network operations that can be administered at scale across a diverse set of data partnerships that include administrative claims sources, as well as, electronic health record data. The network shall facilitate distributed analyses with aggregated datasets but also enable the secure sharing of de-identified patient-level data when necessary (e.g., for PEPR tool).

4.2. Data Expansion Activities

The OC shall expand the DDN by enhancing the infrastructure and by obtaining access to new data sources. The OC shall work with FDA to develop, prioritize and implement these data expansion activities.

Expansion of the DDN Infrastructure: The OC shall expand the Sentinel System’s data infrastructure under the guidance of FDA. The OC shall develop and maintain the capacity to initiate and sustain four to five different major infrastructure enhancements simultaneously. Such infrastructure activities include the addition of new Data Partners, new common data model tables and elements (e.g., inpatient pharmacy and vital signs), development and testing of novel data linkages (e.g., mother-infant linkage), and the integration of federally-held data sources (e.g., Medicare data).

The OC shall advance FDA’s strategic priorities to advance the incorporation, ingestion and processing of EHR data for evidence generation. These activities shall require strategic decisions concerning about how to evolve the SCDM and whether raw unstructured data or standardized information should be incorporated. The integrated nature of the analytic platform requires the OC to ensure that any potential SCDM changes are also accompanied by the requisite changes to the transformation code and to the quality assurance programs that work in concert to convert raw data into quality-checked, analysis-ready datasets at each Data Partner.

Complementary Data Sources: In collaboration with FDA, the IC, and CBOC, the OC shall identify new and/or develop data sources including commercially available datasets, registries, and other data sources to complement the network of Data Partner relationships, for use in a production environment.

5. Task 3: Medical Product Analyses

5.1. Tool-based Analyses

The OC shall maintain the current Sentinel System suite of analysis tools developed with numerous capabilities. These pre-programmed, reusable, and parameterizable analysis tools include Summary Tables (ST), Level 1, 2, 3 tools, the PEPR and TreeScan.

5.2. Service Level Agreement for Medical Product Analyses

The OC shall provide a Service Level Agreement (SLA) for 12 month ordering periods accounting for the following representative requirements:

· In order to maintain the capacity and the capabilities of the fully activated Sentinel System, the contractor shall have the capacity to support a minimum of 10 Level 1, 5 Level 1+, and 2 Level 2+ queries per quarter, unless otherwise directed by FDA.

· Specify the exact number of queries, by query type, to be run on a quarterly basis within 12-month ordering periods, in conjuction with FDA, based upon Agency surveillance needs.

· Specify the total amount of custom programming allowed per quarter.

· Establish contractual agreements with Data Partners inclusive of these goals.

· SLAs shall be renewed at least 2-months prior to the start of the next ordering period on an annual basis to ensure sufficient time for the Data Partners to appropriately staff to meet the volume and variety of analyses, sustain the requisite response times for the rapid analytics platform, and prepare for the rapid tool enhancements (e.g., L1+/L2+).

The OC shall provide various SLAs, that differ by cost and service level, for FDA to select to operationalize the rapid analytics platform in the Sentinel System; agreements can be different from the existing ones. These SLAs shall set timetables for responses at various stages of query development (e.g., planning/specification, program testing, query distribution and execution, and result compilation, verification and reporting to FDA). These timetables shall vary by analysis type and provide FDA with a fully loaded per-query cost (See Attachment 9 - Pricing Matrix).

FDA reserves the right to retire certain analysis types (e.g., Summary Tables) if it is in its best interest during the annual renewal period for the SLAs. Any special costs that are external to the provision of services related to query development and execution, such as Summary Table creation by Data Partners, should be highlighted separately here as well. Moreover, any additional queries that are deemed necessary to be run by the OC alone as part of routine quality checking, erroneous query specification, query re-runs secondary to newly identified programming anomalies or discrepancies, or to meet other network operational needs, should be included separately here as well.

The following tables are provided as technical reference only for the current level of service. A version of these tables is to be populated in the business proposal (see Volume VI - Task Order #1 Price Proposal).

Table 1: Current Service Level Agreement Query Schedule

FDA Queries Per Quarter FDA Queries Per Year

Ω Note: The numbers in this table are an estimate based on current contract quantities. Future ordered quantities may either increase or decrease.

Table 2: Schedule of Response Times for the Current Service Level Agreement (mean time in weeks)

[Stage 1] Query Planning* Date concept brief submitted by FDA to when FDA approves the test specifications [Stage 2] Query Testing & Refinement* Test specifications approved to Date distributed to the DPs

[Stage 3] ¥ Query Distribution, Execution, Compilation & Quality Checking and Reporting Date distributed to DPs to when the report is sent to FDA

* Note: The average times provided for stages 1 and 2 are for information purposes only and not intended to be contractually enforced response timelines.

¥ Note: The contractor shall meet these time targets in Stage 3 for 90% of all queries or another mutually agreed upon target. This 10% allowance was created to give FDA the flexibility to rapidly shift priorities to meet its regulatory timelines. For example, FDA may choose to pause an ongoing query at a certain stage to accelerate another priority query through the analytic stages.

Table 3: Operation Center Queries*

* Operations Center Queries are a number of queries reserved for the Operating Center to be run to meet their own quality control or other needs. These requests do not count towards the Service Level Agreement (SLA). This table is provided for information purposes only to help inform offeror proposals.

5.3. Query Execution Process and Query Stage Definitions

The OC shall establish a staged process for query execution with well-defined sequential stages as follows:

· The query execution process shall begin with the development of a “Sentinel concept brief”. The concept brief is an abbreviated study protocol, written by FDA investigators, which contains the regulatory background motivating the query, analytic objectives, shell tables, and study design requirements for each analysis.

· In collaboration with FDA, the OC staff shall specify, parameterize and translate these designs into the analytic programs.

· To ensure fidelity in the translation process from concept brief to analytic package, all query packages shall undergo testing in a test dataset prior to distribution to the DPN. The OC shall acquire the requisite test dataset formatted in the SCDM. The test dataset shall contain healthcare data that reasonably corresponds to the data in the Sentinel System. The test results shall be shared with FDA, who may adjust the parameters and code based on these results to ensure that the correct quantities of interest are specified. This may require one or more rounds of iteration, prior to actual query distribution to the Sentinel System Data Partners, to achieve the desired study specifications.

· Upon FDA approval, the query shall be distributed to Data Partners to run on each of their SCDM-formatted datasets unless otherwise specified by FDA.

· Once Data Partners return results to the OC, the OC shall quality check the data and compile the results into a report for the FDA.

The OC shall establish SLAs and maintain a rapid analysis system. The OC shall include sufficient project management and data analyst staffing, and implement efficient workflow support programs and standardized processes to track and audit analyses from start to finish. Highly efficient quality assurance programs for a rapid analysis system shall include standard templates, standardized output formats, and the ability to capture the date, time and content of key analytic decisions made with the FDA investigation teams across the lifetime of a query.

5.4. Protocol Based Assessments

The OC shall conduct protocol-based analyses (PBA) and assemble teams of investigators to either lead or co-lead analyses with FDA investigators.

PBA analyses are distinct from routine queries by a combination of features. Not every PBA has all these features and some PBAs have additional features. However, PBAs generally share at least some of the following features:

· Novel study designs

· Novel statistical methods

· Novel data sources (e.g., linkages to registries)

· Extensive / full custom programming

· Access to data sources not routinely formatted in the SCDM

· Formation of a long-term workgroup of FDA and external collaborative investigators

· Chart validation of outcomes or other study design elements

· Development of formal study protocol that is posted on the Sentinel website

· Study protocol and final report written by a non-FDA investigator

PBA analyses are often conducted when the pre-programmed tools are not sufficient to meet the requisite study design to address a regulatory question. For each PBA, FDA will initiate analyses using work orders.

5.5. Data Retention Policies

The OC shall develop ‘Data Retention Policies’ for the Sentinel System. The schedule shall be proposed by the OC and approved by the Data Partners and FDA. At a minimum, the schedule shall include the duration of retention for:

· Analytic datasets at each Data Partner to allow post-hoc analyses for each query type

· Data refreshes (i.e., the prior ETL, current ETL, and future ETL that is undergoing quality checking)

The data retention schedule shall balance the storage costs with the relative importance of the queries; the highest priority given to retaining the dataset for the Level 2 analyses or other complex analyses. Lower priority analyses, such as Level 1 analyses, will only be held for brief time periods or even not retained at all. The ‘Data Retention Policies’ document shall include the establishment of such policies and address the storage requirements associated with implementation of such data retention policies.

5.6. Business Informatics, Workflow Tracking and Reporting

To meet FDA’s program tracking needs, the OC shall establish, maintain and improve a ‘Business Informatics Workflow Tracking and Reporting System’ for:

· All routine queries including ST, L1, L2, L3, TreeScan and other signal detection tools, and PEPR.

· All infrastructure expansion and tool enhancement projects.

· All methods development projects.

· FDA clearance and approval of deliverables and results for posting on the Sentinel website.

· FDA clearance and approval of abstracts, manuscripts, posters, final reports for public presentation and/or submission to peer-review journals.

· All Sentinel publications.

· All FDA regulatory outcomes emerging from analyses completed by Sentinel System.

The Business Informatics Workflow Tracking and Reporting System shall meet the following requirements:

· Automatically update, publish and distribute status reports on analyses in a “dashboard” or other desired format.

· Accept data entry from staff from the OC, FDA, and investigators from the Sentinel System’s network of collaborators.

· Support rapid descriptive analytics to help respond to inquiries from Congress, to assist with user fee or executive reporting requirements, and assist with public or mass media responses.

· Track FDA’s determinations of ARIA sufficiency and create automated reports of this critical program outcome measure.

· Include improvements to the types of data collected, the user interface, automated reporting enhancements, and custom analytics as necessary.

· Ensure FISMA compliant and have controlled access accounts whose privileges can be tailored to predefined user profiles.

· Sustain at least 125 FDA users, in addition to all necessary OC staff, select Sentinel investigators, and have capacity to be expanded as needed.

6. Task 4: Analytic Tools

6.1. Tool Maintenance Activities

The Sentinel System currently possesses a wide array of rapid analysis tools capable of both descriptive and inferential analyses and serve both hypothesis-generation and hypothesis-testing purposes. Even with sophisticated testing and validation of all programming code, no code is perfectly created, and new anomalies and discrepancies will surface as the programs are challenged against the wide spectrum of FDA’s analytic designs and needs. The OC shall provide the requisite programming and data analysis experience to maintain and enhance these parameterizable analytic tools.

The OC shall maintain the analytic tools through continuous checking, profiling, and debugging of these analytic programs. Patches shall be released routinely in successive waves that are version-controlled and timed to avoid major disruptions in capabilities and when necessary, to achieve FDA’s time sensitive analytic needs.

6.2. Rapid Tool Enhancements

The OC shall work with FDA investigators to identify differences between the desired analysis and current tool capabilities. In situations where these gaps can be remedied with small amounts of custom programming, the OC shall modify the existing tools with custom code to achieve the desired analytic endpoint. These analyses are termed “Level 1 plus” or “Level 2 plus” (i.e., L1+ or L2+, respectively). The OC shall designate programmer analysts who can be available for at least 80 hours per quarter to facilitate these semi-custom, rapid-analytics to sustain this highly important capability.

6.3. Routine Integration of Rapid Enhancements

The OC shall routinely select and integrate small, discrete amounts of custom coding used to tailor queries into the routine analysis tools. When it becomes clear that these one-time enhancements have a wider application and are repeatedly requested by FDA investigators, they shall be automatically incorporated into the standard toolkit as new features. Once integrated, FDA’s use of these new features will fall under a traditional L1/L2 analysis and not require the L1+/L2+ designation.

6.4. Version Control of Routine Tools

The OC shall implement a version control system to track the change-history of these improvements, document merges and branches to the program development tree, and connect to project management and debugging software tracking that can record the intent of the change and enable forensics and root cause analyses to be conducted.

6.5. New Tool Development

In collaboration with the IC, innovative new methods will be transformed into analysis tools and major enhancements to existing tools shall be developed by the OC under the guidance of FDA leadership for use in a production environment. The OC shall develop, test, evaluate and operationalize new tools into a rapid analysis platform for use in a production environment. Recent examples of new tools have included the TreeScan signal detection tool and the propensity score matching sequential (Level 3) analysis tool. Tools may be descriptive, inferential and predictive in nature to provide the FDA with the most robust suite of risk analysis for drug safety and advance the boundaries the RWE development.

The OC staff shall possess and maintain expertise in adapting statistical methods to a distributed database setting where analytic datasets contain only aggregate level information consisting of counts and coefficients, rather than patient-level data with one row per patient.

7. Task 5: Training

7.1. FDA

The OC, in collaboration with FDA, IC and CBOC shall establish a routine training program to train and update FDA epidemiologists, pharmacists, medical officers, statisticians, and other FDA staff on new data resources and tool capabilities and the use of the SCDM. The content of the training shall be directed and reviewed by FDA Sentinel program staff. This internal training may consist of multiple parts, to be negotiated with FDA, and include:

· Monthly, FDA-wide, 1-hour webinars on topics that emerge from recently completed methods projects or studies, or special training topics of interest to FDA (e.g., updates to ICD-10 coding). These are done remotely and are not associated with travel.

· Quarterly 1-day visits to the FDA White Oak Campus by OC epidemiologists and data analysts to collaborate on pending queries and provide a two-hour presentation of system-level updates.

· Ad hoc training, as needed, on new query development processes, changes to the query tracking interface software, or any other topic of interest to FDA. These trainings may be in-person or be conducted remotely.

7.2. External Stakeholders

The OC, in collaboration with FDA, IC and CBOC shall establish a routine external training program to train and update other stakeholders that may include staff at Sentinel System Data Partners, FDA-regulated industry, international regulators, and other federal agencies on new data resources and tool capabilities as to the use of the SCDM as they relate to regulatory and public health decision making. The content of the training shall be directed and reviewed by FDA Sentinel Core Team staff. This external training may consist of multiple parts, to be negotiated with FDA, and include annual public trainings conducted in association with the Sentinel Annual Meeting.

These training resources may also be used to educate and train interested members of the academic community, consumers, and the public, as part of activities of the CBOC.

8. Task 6: Medical Record Review

8.1. Traditional Chart Review Activity

The OC shall establish routine and standardized administrative, financial and scientific processes to ascertain medical records (electronic and/or paper) across the Data Partners in the DDN. The OC shall provide the requisite tracking software, database development, abstraction teams, and clinical adjudication teams needed to complete these activities in support of FDA’s analytic needs. The capability to access medical records shall be required for a variety of regulatory reasons, such as:

· Larger sample (e.g., 150–250 charts across several data partners) chart review, for example, to validate an important health outcome algorithm for use in future analyses, or as part of a follow-up investigation for a Level 2 analysis or TreeScan query.

· Smaller sample (e.g., 25–50 charts from a single data partner) chart review, for example, to assist with outcome algorithm refinement, or to fine tune a medication error algorithm based on real world coding practices.

· Specific charts needed for rare events.

Given that the costs for chart review projects have many dependencies, including the number and specific Data Partners involved, the number of charts ascertained, and the vendor used by each Data Partner, FDA will initiate chart review activities using Work Orders. The OC shall work with FDA to establish a standard SLA to support medical chart review activities.

The OC shall obtain and meet all required review, approval and standard processes that comply with all relevant federal, state, and local laws and regulations to conduct the scope of work.

8.2. Enhanced Chart Review Activities

Under the direction of FDA, the OC shall develop and implement new technology-assisted approaches to automate labor intensive steps and reduce medical chart review costs in the production environment, in collaboration with the IC. This shall include implementation of optical character recognition, natural language processing or machine learning technologies to enhance the speed, accuracy and cost-effectiveness of traditional medical chart review. This scope of activities also includes non-technological improvements such as administrative workflow improvements, standardization of processes, changes to the SCDM to record chart review confirmation status, and other opportunities. The OC shall develop and maintain a plan to improve the efficiency of medical chart review, reduce costs, and provide a roadmap for promoting automation of labor-intensive steps in medical chart validation. FDA will initiate automation and technology assisted medical chart review activities using Work Orders.

9. Task 7: Methods Prioritization and Implementation For project and priority areas identified, in collaboration with FDA, the IC and the CBOC, the OC shall assist with identifying and matching the scientific community’s expertise with FDA’s methodologic needs. The OC shall provide the expertise in the areas as described below. FDA will initiatve Work Orders to initiate methods projects under this Task Order.

· The OC shall have the expertise and ability to develop and implement a strategy to reach out to stakeholders that can include FDA-regulated industry, other Federal Agencies and international regulatory agencies.

· The OC shall supervise and assist with the conversion of diverse data sources to the SCDM.

· The OC shall have expertise in advanced analytics (e.g., machine learning, natural language processing, artificial intelligence) for purposes of implementing and operating tools based on such methods in a production environment.

· The OC shall develop the administrative processes and appropriate governance system to issue scientific workgroup opportunities for investigators in the Sentinel network.

The OC shall apply advanced analytic methods to EHR data to solve epidemiologic problems, such as improving the identification of populations of interest, covariates, or health outcomes.

10. Task 8: FDA Catalyst FDA Catalyst is designed to use the Sentinel System to support effectiveness research. Study designs include randomized trials and patient registries with prospective data collection in addition to observational research. Expertise in clinical trial design and human subjects protection is required. FDA-Catalyst activities may involve contact with patients or providers for purposes including but not limited to recruitment, eligibility, and outcome assessment. FDA-Catalyst activities will also include the use of mobile technology including apps and wearables. FDA shall initiate work within FDA Catalyst using Work Orders.

The OC shall provide assurance that FDA Catalyst activities that involve interventions or interactions with health plan members or providers will undergo initial and continuing review by an appropriate Institutional Review Board (IRB) and will comply with Office of Management and Budget (OMB) requirements (e.g., Paper Reduction Act (PRA)). The OC shall establish DUAs to ensure continuity of critical FDA Catalyst activities conducted under IRB approval.

The OC shall integrate the publicly available FDA MyStudies App (mobile app) into the Sentinel infrastructure in support of ongoing and future FDA-Catalyst projects.[footnoteRef:2] [2: https://www.fda.gov/Drugs/ScienceResearch/ucm624785.htm, links to publicly available technical descriptions]

11. Period of Performance