If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author at: Alan A. Kubey, MD, Division of Hospital Medicine, Thomas Jefferson University Hospital, 833 Chestnut Street, Suite 701, Philadelphia, PA 19107, USA
Division of Hospital Medicine, Department of Medicine, Thomas Jefferson University Hospital, Philadelphia, PA, USADivision of Hospital Internal Medicine, Department of Internal Medicine, Rochester, MN, USA
Identifying patients at risk for mortality from COVID-19 is crucial to triage, clinical decision-making, and the allocation of scarce hospital resources. The 4C Mortality Score effectively predicts COVID-19 mortality, but it has not been validated in a United States (U.S.) population. The purpose of this study is to determine whether the 4C Mortality Score accurately predicts COVID-19 mortality in an urban U.S. adult inpatient population.
Methods
This retrospective cohort study included adult patients admitted to a single-center, tertiary care hospital (Philadelphia, PA) with a positive SARS-CoV-2 PCR from 3/01/2020 to 6/06/2020. Variables were extracted through a combination of automated export and manual chart review. The outcome of interest was mortality during hospital admission or within 30 days of discharge.
Results
This study included 426 patients; mean age was 64.4 years, 43.4% were female, and 54.5% self-identified as Black or African American. All-cause mortality was observed in 71 patients (16.7%). The area under the receiver operator characteristic curve of the 4C Mortality Score was 0.85 (95% confidence interval, 0.79-0.89).
Conclusions
Clinicians may use the 4C Mortality Score in an urban, majority Black, U.S. inpatient population. The derivation and validation cohorts were treated in the pre-vaccine era so the 4C Score may over-predict mortality in current patient populations. With stubbornly high inpatient mortality rates, however, the 4C Score remains one of the best tools available to date to inform thoughtful triage and treatment allocation.
The management of patients with COVID-19 is challenging for front-line healthcare providers given limited validated, evidence-based clinical decision support. Determining patient mortality risk is critical for effective triage, management, and discharge decision making. Numerous COVID-19 risk prediction models of varying country of origin, population demographic, and robustness exist.
Xie J, Hungerford D, Chen H, et al. Development and external validation of a prognostic multivariable model on admission for hospitalized patients with COVID-19. medRxiv. 2020. doi:10.1101/2020.03.28.20045997.
The 4C Mortality Score, created by the International Severe Acute Respiratory and emerging Infections Consortium (ISARIC), is one of the largest-scale, high-performing predictive models published to date.
Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C mortality score.
The initial 4C Mortality Score study included adult patients hospitalized with COVID-19 across 260 hospitals in England, Wales, and Scotland, utilizing eight weighted variables – age, sex, select comorbidities, respiratory rate, peripheral oxygen saturation, Glasgow Coma Scale (GCS), blood urea nitrogen (BUN), and C-reactive protein (CRP) – to predict 30-day inpatient mortality. The area under receiver operator characteristic curve (AUC) of the derivation (n=35,463) and validation (n=22,361) cohorts was 0.79 (95% confidence interval [CI], 0.78-0.79) and 0.77 (95% CI, 0.76-0.77), respectively. This model outperformed 16 risk stratification scores, including SOFA, NEWS, CURB-65 and four novel COVID-19 risk scores.
Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C mortality score.
For United States clinicians to confidently utilize the 4C Mortality Score, however, the model must be validated in a population similar to its intended implementation, particularly in areas with minorities under-represented in prior research. The objective of this study is to determine whether the 4C Mortality Score is a clinically appropriate model to predict hospital mortality in United States patients with COVID-19.
Methods
Study population
Patients included in this study had a positive SARS-CoV-2 PCR test result and were admitted through the emergency department to Thomas Jefferson University Hospital (TJUH), a tertiary care academic medical center in Philadelphia, PA, between March 1st and June 6th, 2020. Symptomatic and asymptomatic patients were included. Patients transferred from outside hospitals were excluded. Patients under the age of 18, pregnant, and/or incarcerated were excluded from this study. This study was approved by the TJUH Institutional Review Board (IRB#: 20E.737).
Variables and outcome
The following 4C Mortality Score comorbidities were extracted from electronic health records (EHR) through manual chart review: chronic cardiac disease, chronic respiratory disease (excluding asthma), chronic kidney disease (estimated glomerular filtration rate ≤ 30 mL/min/1.73m2), liver disease, connective tissue disease, diabetes mellitus, HIV/AIDS, active malignancy, and obesity (body mass index [BMI] ≥ 30 kg/m2). These data were obtained from the last discharge summary, the emergency department note, the admission history and physical, the EHR “Past Medical History” and/or “Active Problems” sections. All comorbidities were retrieved by two separate researchers; a third independent reviewer adjudicated data discrepancies. The study database was created prior to the release of the 4C Mortality Score; therefore, two comorbidities – dementia and “chronic neurological disease ” – were not included in this study. Respiratory rate, peripheral oxygen saturation, GCS, BUN, and CRP, within 24 hours of initial hospital encounter were digitally exported. The initial hospital encounter was defined as the first time-value of vitals entered in the EHR for the admission of interest. The primary outcome was a composite outcome of mortality during admission or mortality within 30 days of discharge. Mortality status was determined by chart review which included deaths occurring at other hospitals using the same EHR (the three largest academic medical centers in Philadelphia are included). This study is reported consistently with the Transparent Reporting of a multivariate prediction model for Individual Prognosis or Diagnosis guidelines.
Statistical analysis
Data was missing for less than 1.4% of variables. Values were missing for four variables of the 4C Score: CRP, GCS, BUN, and obesity (Table 1). Missing data was assumed to be missing at random. Multiple imputation methods were applied to the 4C Score components using R (Version 4.0.4, R Project for Statistical Computing) and the Multivariate Imputation by Chained Equations package (MICE, v3.13.0). Twenty imputed datasets were created, using predictive mean matching for numerical variables (e.g., CRP) and logistic regression for binary variables (e.g., obesity).
TABLE 1Demographics, outcomes, and clinical variables of the study cohort and 4C Mortality Score cohort.
Adjusted 4C Mortality Scores were computed for each record using the data in each imputed dataset. Mortality probability for each 4C Score was sourced from the ISARIC online 4C calculator (https://isaric4c.net/risk/v2/). Receiver operating characteristic curves were calculated for each imputed dataset comparing the predicted 4C Score and actual mortality for each record. Area under the receiver operator curve and Brier score, another measure of goodness of fit (ranging from 0 to 1; smaller values indicate superior calibration), were calculated for each curve. AUC, corresponding 95% CI, and Brier score were pooled for all 20 imputation results using Rubin's rule.
There were 426 patients included in this study. Mean age was 64.4 years; 43.4% were female; 54.5% were Black or African American (hereafter “Black”). Cohort demographics, clinical variables, and outcomes are presented in Table 1. The primary outcome of mortality during admission or within 30 days from discharge occurred in 16.7% of patients (71/426 patients). Of the 71 deaths, 10 occurred after discharge. Of deaths occurring after the index admission, the average time from index admission to death was 13.9 days (minimum: 7 days; maximum 21 days). The incidence of other measures of clinical interest (i.e., ICU status, discharge to hospice) can also be seen in Table 1.
Performance of 4C mortality score
The AUC for the pooled dataset was 0.85 (95% CI, 0.79-0.89). The Brier score for the dataset was 0.246. The calibration curve for all imputed datasets is plotted in Fig. 1; calibration curves for each individual imputation are plotted in the Supplemental Materials.
FIGURE 1Predicted (4C Mortality Score) vs. actual mortality for all imputed datasets.
With an AUC of 0.85 (95% CI, 0.79-0.89) and Brier score of 0.246, the 4C Mortality Score performed well in a United States urban population with a majority Black patient cohort. These findings are consistent with the results of external validation studies in other countries.
We note a non-statistically significant trend toward overprediction of mortality, particularly visually noticeable in 4C scores <15 as seen in Fig. 1. This may be non-significant data variation, but suggests the need for further research in larger United States populations to investigate. We suspect that the 4C Score will tend to over-predict mortality in current populations given the increased availability of vaccines and effective outpatient and inpatient treatments relative to the study population (prior to June 7, 2020).
Group The RECOVERY Collaborative Tocilizumab in patients admitted to hospital with COVID-19 (RECOVERY): a randomised, controlled, open-label, platform trial.
Efficacy and safety of baricitinib for the treatment of hospitalised adults with COVID-19 (COV-BARRIER): a randomised, double-blind, parallel-group, placebo- controlled phase 3 trial.
Effect of noninvasive respiratory strategies on intubation or mortality among patients with acute hypoxemic respiratory failure and COVID-19: the RECOVERY-RS randomized clinical trial.
Effect of high-flow oxygen therapy vs conventional oxygen therapy on invasive mechanical ventilation and clinical recovery in patients with severe COVID-19: a randomized clinical trial.
Given stubbornly high inpatient mortality rates to date, however, this assumption may not hold true.
Notably, this dataset utilized manual chart review, which was shown to outperform automated EHR data export for the sensitivity of capturing comorbidities within this study population.
This may have led to higher scores in our validation cohort relative to the original derivation cohort or other validation cohorts. Further strengths of this study include the representation of the racial diversity of the center's patient population, with a majority of the study cohort identifying as Black (54.5%). Recent research into the association of race, socioeconomic status, and COVID-19 outcomes highlights the necessity for including under-represented minorities in the validation of risk models.
Association of race/ethnicity and socioeconomic status with COVID-19 30-day mortality at a Philadelphia medical center using a retrospective cohort study.
There are notable limitations to this study, including the limited generalizability of a validation performed in a single, tertiary care center. The absence of dementia and “chronic neurological disease” may have yielded lower risk scores for patients with 0-1 other comorbid conditions. Additionally, the original 4C Mortality Score used clinician-defined obesity, whereas this study used a BMI of > 30 kg/m2. We anticipate that the effect of missing comorbidities is opposed by the increased sensitivity of manual chart review in identifying comorbidities, mitigating any significant effect on model calibration, especially considering that comorbidities account for a small portion of the 4C Mortality Score. The primary outcome or the original 4C Mortality Score was 30-day in-hospital mortality, as compared to mortality during admission or within 30 days of discharge. As the intent of our validation of the 4C Mortality Score was to evaluate its clinical utility, we chose to include patients who died within 30 days of discharge under the assumption that they were likely to have died of complicated related to their index admission; that the average time from index admission to death was 13.9 days appears to support this assumption. Given these limitations, we believe the overall conclusion is still appropriate and reinforces the importance of clinicians and public health leaders thoughtfully applying the 4C Score to current specific scenarios.
The 4C Mortality Score is a clinically useful tool to assess risk for COVID-19 mortality as validated in an early 2020 United States urban tertiary care center. The validation of the 4C Mortality Score in a majority Black population is important for providers in many United States urban and rural centers to confidently use this tool for clinical decision making. The score is particularly useful for emergency department triage, as well as patient-specific treatment decision making (e.g., monoclonals, oral antivirals, remdesivir dosing, etc.) for scenarios in which treatment regimens are not clearly defined in national guidelines. Given the vaccination efforts and improved treatment options since this study cohort, the 4C Mortality Score should be applied thoughtfully by clinicians, carefully accounting for differences between an individual patient's clinical scenario and the populations and timing of our validation cohort. Further external validation of COVID-19 risk prediction tools in cohorts including vaccinated and unvaccinated patients receiving more recent treatment regimens would be of great value to clinicians. Ultimately, the 4C Mortality Score provides one of the best evidence-based approaches to date for patient triage, treatment choice and duration, and discharge planning to best serve patients afflicted by COVID-19.
Funding
None.
Conflicts of Interest
The authors have no conflicts of interest to disclose.
Xie J, Hungerford D, Chen H, et al. Development and external validation of a prognostic multivariable model on admission for hospitalized patients with COVID-19. medRxiv. 2020. doi:10.1101/2020.03.28.20045997.
Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C mortality score.
Efficacy and safety of baricitinib for the treatment of hospitalised adults with COVID-19 (COV-BARRIER): a randomised, double-blind, parallel-group, placebo- controlled phase 3 trial.
Effect of noninvasive respiratory strategies on intubation or mortality among patients with acute hypoxemic respiratory failure and COVID-19: the RECOVERY-RS randomized clinical trial.
Effect of high-flow oxygen therapy vs conventional oxygen therapy on invasive mechanical ventilation and clinical recovery in patients with severe COVID-19: a randomized clinical trial.
Association of race/ethnicity and socioeconomic status with COVID-19 30-day mortality at a Philadelphia medical center using a retrospective cohort study.
00211-7&id=gr1.jpg){kind=link}