A Caltech Library Service

Real-time electronic health record mortality prediction during the COVID-19 pandemic: a prospective cohort study

Sottile, Peter D. and Albers, David and DeWitt, Peter E. and Russell, Seth and Stroh, J. N. and Kao, David P. and Adrian, Bonnie and Levine, Matthew E. and Mooney, Ryan and Larchick, Lenny and Kutner, Jean S. and Wynia, Matthew K. and Glasheen, Jeffrey J. and Bennett, Tellen D. (2021) Real-time electronic health record mortality prediction during the COVID-19 pandemic: a prospective cohort study. Journal of the American Medical Informatics Association, 28 (11). pp. 2354-2365. ISSN 1067-5027. doi:10.1093/jamia/ocab100.

[img] PDF - Submitted Version
See Usage Policy.

[img] MS Word (Supplementary data) - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


Objective: To rapidly develop, validate, and implement a novel real-time mortality score for the COVID-19 pandemic that improves upon sequential organ failure assessment (SOFA) for decision support for a Crisis Standards of Care team. Materials and Methods: We developed, verified, and deployed a stacked generalization model to predict mortality using data available in the electronic health record (EHR) by combining 5 previously validated scores and additional novel variables reported to be associated with COVID-19-specific mortality. We verified the model with prospectively collected data from 12 hospitals in Colorado between March 2020 and July 2020. We compared the area under the receiver operator curve (AUROC) for the new model to the SOFA score and the Charlson Comorbidity Index. Results: The prospective cohort included 27 296 encounters, of which 1358 (5.0%) were positive for SARS-CoV-2, 4494 (16.5%) required intensive care unit care, 1480 (5.4%) required mechanical ventilation, and 717 (2.6%) ended in death. The Charlson Comorbidity Index and SOFA scores predicted mortality with an AUROC of 0.72 and 0.90, respectively. Our novel score predicted mortality with AUROC 0.94. In the subset of patients with COVID-19, the stacked model predicted mortality with AUROC 0.90, whereas SOFA had AUROC of 0.85. Discussion: Stacked regression allows a flexible, updatable, live-implementable, ethically defensible predictive analytics tool for decision support that begins with validated models and includes only novel information that improves prediction. Conclusion: We developed and validated an accurate in-hospital mortality prediction score in a live EHR for automatic and continuous calculation using a novel model that improved upon SOFA.

Item Type:Article
Related URLs:
URLURL TypeDescription Paper
Sottile, Peter D.0000-0002-1071-730X
Albers, David0000-0002-5369-526X
DeWitt, Peter E.0000-0002-6391-0795
Russell, Seth0000-0002-2436-1367
Stroh, J. N.0000-0003-4844-1983
Levine, Matthew E.0000-0002-5627-3169
Mooney, Ryan0000-0002-5401-0891
Wynia, Matthew K.0000-0003-3307-069X
Bennett, Tellen D.0000-0003-1483-4236
Alternate Title:Real-Time Mortality Prediction During COVID-19
Additional Information:© The Author(s) 2021. Published by Oxford University Press on behalf of the American Medical Informatics Association. This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model ( Received 23 February 2021; Revised 19 April 2021; Editorial Decision 4 May 2021; Accepted 6 May 2021. Published: 02 September 2021. We would like to acknowledge Sarah Davis, Michelle Edelmann, and Michael Kahn at Health Data Compass. PS is supported by NIH K23 HL 145001; DA and ML by NIH R01 LM012734; DK by NIH K08 HL125725; TB by NIH UL1 TR002535 and NIH UL1 TR002535 - 03S2. Author Contributions: TDB had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. PDS, DA, PED, SR, JS, and TDB contributed substantially to the study design, data acquisition, data analysis and interpretation, and the writing of the manuscript. DPK, BA, RM, and LL contributed substantially to data acquisition, verifying data integrity, and the writing of the manuscript. MEK contributed substantially to study design and the writing of the manuscript. DA, MEL, and PDS conceptualized and initially designed the statistical modeling framework. JSK, MKW, and JJG contributed substantially to the study design, data acquisition, and the writing of the manuscript. Conflict of Interest Statement: None declared.
Funding AgencyGrant Number
NIHK23 HL145001
NIHR01 LM012734
NIHK08 HL125725
NIHUL1 TR002535
NIHUL1 TR002535-03S2
Subject Keywords:crisis triage, mortality prediction, COVID-19, decision support systems, clinical, machine learning
Issue or Number:11
Record Number:CaltechAUTHORS:20210120-124914573
Persistent URL:
Official Citation:Peter D Sottile, David Albers, Peter E DeWitt, Seth Russell, J N Stroh, David P Kao, Bonnie Adrian, Matthew E Levine, Ryan Mooney, Lenny Larchick, Jean S Kutner, Matthew K Wynia, Jeffrey J Glasheen, Tellen D Bennett, Real-time electronic health record mortality prediction during the COVID-19 pandemic: a prospective cohort study, Journal of the American Medical Informatics Association, Volume 28, Issue 11, November 2021, Pages 2354–2365,
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:107593
Deposited By: Tony Diaz
Deposited On:20 Jan 2021 23:22
Last Modified:18 Nov 2021 17:56

Repository Staff Only: item control page