AoU_CLAD_ROA-OTA-23-003 Appendix B - Data Quality.docx (1).pdf

APPENDIX B: ALL OF US RESEARCH PROGRAM CONCEPTUAL FRAMEWORK

FOR DATA QUALITY

The overarching objective of CLAD is to broaden the All of Us Research Program’s research utility by passively acquiring new data streams through linkage. CLAD will also provide additional methods to augment existing data, such as electronic health records (EHR). Inherent in linking in these new data are a variety of issues that may affect the quality and validity of the data.

The All of Us Research Program uses an extensive operational process for the curation of EHR data prior to dissemination on the Researcher Workbench (RW). All of Us EHR data is acquired from many healthcare provider organizations (HPO) with variability in data quality. There are various underlying EHR vendor systems used by the HPOs. Consequently, consolidation into a single dataset is quite challenging. Moreover, there is great variability in how investigators will use the data in the RW.

Currently, the All of Us Research Program uses several steps in the data curation pipeline to assess EHR data quality. First, the HPOs perform a quality control check prior to uploading data into a bucket for transfer to the Data and Research Center (DRC). These checks are done with a local validation script and assess conformance with the Observational Medical Outcomes Partnership (OMOP) common data model, presence of required data, and consistency in file naming conventions. In the second step, the DRC uses the Automated Characterization of Health Information at Large-scale Longitudinal Evidence System (ACHILLES) to flag data errors, participant ID verification, concept encoding correctness, currency of data points, and match counts with recruitment data. Thirdly, after all data has been aggregated into a single dataset it is passed through a variety of cleaning rules, de-identification, and de-duplication checks. Finally, an integrity check is conducted for OMOP domains (e.g., Person, Visits, etc.)

In addition to this work, the All of Us Research Program uses two other elements, outside of the data curation pipeline, to assess data quality. First, a periodic data characterization report is produced. This report generates basic descriptive statistics for data completeness by race and ethnicity, sex assigned at birth, self-reported gender identity, self-reported age, educational attainment, and self-reported income. Other aspects of this report include assessments of data availability by OMOP domain and HPO site; benchmarking against other national data sources (e.g., the National Health and Nutrition Examination Survey) where appropriate; and data element distributions.

The second element uses a conceptual framework of five data quality dimensions (DQD) that capture the fitness of use for All of Us EHR data. The DQDs are mutually exclusive and parsimonious. These DQDs are being operationalized to assess specific phenotypes. Currently, assessments have been conducted on ductal carcinoma in situ (DCIS), type 2 diabetes, and diabetic retinopathy. The All of Us Research Program anticipates expanding this to develop baseline quality profiles for additional phenotypes. The table below provides information on how these dimensions are defined and examples of how they can be applied.

Notwithstanding how the quality of the provided data is monitored and assessed by the offerors, it should be able to withstand scrutiny by researchers and other end-users. The All of Us EHR data quality steps, elements, and DQDs may be adapted by offerors, in whole or in part, to assess the quality of CLAD data linkages. Offerors may also choose to define a different framework to assess the quality of CLAD data linkages.

Dimension Definition Application

Completeness Sufficient health-related data/information has been captured/is available for patients within the dataset.

Understanding the core data elements needed for each unique data set is essential. For example, type two diabetes requires the presence of specific lab values.

Concordance Similarity or agreement in EHR data values and elements

EHR data may include numerous encounters for an individual with the healthcare system. Data such as physical measurements may need to agree across these encounters.

Conformance Equivalent data values and elements are represented in the same way.

The representation of EHR data is standardized across multiple locations and settings. For example, specific data values are defined using the same clinical terminology. Another example is lab values are represented using accepted publicly available code values.

Plausibility EHR data values and elements are believable.

Most phenotypes follow a known biological trajectory. For example, certain malignancies are rarely diagnosed in younger individuals.

Temporality EHR data values and elements have a valid start time, end time, and duration, and follow expected temporal patterns.

Most healthcare practices follow a specific agreed-upon pathway. For example, to diagnose a malignancy, imaging and biopsy are typically followed by treatment.