Developing an unbiased machine learning tool for prediction of acute coronary syndrome

This Project Grant from the National Institute for Minority Health and Health Disparities, part of the Department of Health and Human Services National Institutes of Health, provides $769,755 to develop an unbiased machine learning tool for the prediction of acute coronary syndrome. The tool aims to minimize bias in predictions between patient demographic groups, as measured by equal opportunity difference and the Zemel statistic, to ensure machine learning algorithms do not exacerbate existing health inequities. Specifically, Dascena, Inc. will develop an algorithm for early acute coronary syndrome prediction through aim one and compare its performance to commonly used risk stratification scores through aim two. The algorithm will be evaluated for accuracy and bias when predicting outcomes for white versus non-white and male versus female emergency department patients. This work supports the goals of the Minority Health and Health Disparities Research program to reduce health disparities through research, information dissemination, and community outreach.

Abstract

Significance: racial and SEX disparities in the diagnosis and care of acute coronary syndrome (ACS) patients are well documented. As machine learning algorithms (MLA) become more common in healthcare settings, IT is imperative to ensure that these methods do not contribute to disparities through biased predictions or differential accuracy across racial and SEX groups. Research question: can a MLA be trained to be more accurate and less biased than commonly used risk stratification systems for ACS prediction? Prior work: the research team developed a preliminary gradient boosted tree model for myocardial infarction (MI) prediction using retrospective data from electronic health records. On a hold-out test set, the algorithm classifier attained an area under the receiver operating characteristic curve (AUROC) value of 0.92 when tested for the detection of MI at any point during a patient's hospital stay. Other prior work by the research team involved development of a MLA to minimize bias in inpatient mortality predictions between white and non-white patient groups. The model was found to be unbiased as measured by the equal opportunity difference (EOD = 0.016, P = 0.204) and outperformed commonly used severity scoring systems mews, saps-II, and apache in respect to bias and accuracy. Specific aims: in aim 1, an unbiased model for early ACS prediction will be developed. Preprocessing the MLA training data will remove aspects of the data that reflect systemic health inequities while maintaining the aspects of the data that reflect relevant patient measurements and outcomes. Assessment of equal opportunity difference (EOD) and the zemel statistic will provide a means to evaluate the MLA's ability to operate without SEX or racial bias. In aim 2, the model's performance will be compared to three commonly used ACS risk stratification scores. Evaluating model performance and bias against these systems will allow for comparison of the unbiased MLA to the current ACS standard of care. Methods: aim 1: an ACS prediction algorithm that will be demonstrated to be unbiased when comparing performance accuracy on white VS. Non-white and male VS. Female emergency department patients will be developed. The model's performance will be assessed with regard to the EOD and zemel statistic, which measure the difference in false negative results and average predicted risk, respectively, between white and non-white and male and female patients under the null hypothesis of no difference. Aim 2: model performance will be compared to modified versions of three other commonly used ACS risk stratification scores: the global registry of acute coronary events (GRACE) score; the platelet glycoprotein IIB/iiia in unstable angina: receptor suppression using integrilin (EPTIFIBATIDE) therapy (PURSUIT) score; and the thrombolysis in myocardial infarction (TIMI) score, some of which have been shown to perform differentially across gender and race. EOD and the zemel statistic will also be assessed as a measure of bias for the MLA, GRACE, PURSUIT and TIMI scores. Future directions: the MLA will be implemented in live hospital settings for prospective evaluation.