If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author at: Catherine S. O'Neal, Louisiana State University Health Sciences Center, Internal Medicine Residency Program, 5246 Brittany Dr., Baton Rouge, LA 70808, USA.
Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USAOur Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA
Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USAOur Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA
Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USALouisiana State University Health Sciences Center, Emergency Medicine Residency Program, Baton Rouge, LA, USA
Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USALouisiana State University Health Sciences Center, Emergency Medicine Residency Program, Baton Rouge, LA, USA
Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USAOur Lady of the Lake Regional Medical Center, Baton Rouge, LA, USA
Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USAOur Lady of the Lake Regional Medical Center, Baton Rouge, LA, USABaton Rouge General Medical Center, Baton Rouge, LA, USA
Our Lady of the Lake Regional Medical Center, Baton Rouge, LA, USALouisiana State University Health Sciences Center, Emergency Medicine Residency Program, Baton Rouge, LA, USA
Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USAOur Lady of the Lake Regional Medical Center, Baton Rouge, LA, USABaton Rouge General Medical Center, Baton Rouge, LA, USA
Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USABaton Rouge General Medical Center, Baton Rouge, LA, USA
Louisiana State University Health Sciences Center, Internal Medicine Residency Program, Baton Rouge, LA, USAOur Lady of the Lake Regional Medical Center, Baton Rouge, LA, USABaton Rouge General Medical Center, Baton Rouge, LA, USA
This study examined three methods for retrospectively identifying infection in emergency department (ED) patients: modified objective definitions of infection (MODI) from the CDC/NHSN, physician adjudication determination of infection, and ED treating physician behavior.
Methods
This study used a subset of data from a prospective sepsis trial. We used Fleiss's Kappa to compare agreement between two physicians retrospectively adjudicating infection based on the patient's medical record, modified infection definition from the CDC/NHSN, and ED treating physician behavior.
Results
Overall, there was similar agreement between physician adjudication of infection and MODI criteria (Kappa=0.59) compared to having two physicians independently identify infection through retrospective chart review (Kappa=0.58). ED treating physician behavior was a poorer proxy for infection when compared to the MODI criteria (0.41) and physician adjudication (Kappa = 0.50).
Conclusions
Retrospective identification of infection poses a significant challenge in sepsis clinical trials. Using modified definitions of infection provides a standardized, less time consuming, and equally effective means of identifying infection compared to having multiple physicians adjudicate a patient's chart.
Unfortunately, there are no universally accepted definitions of infection and no reliable mechanism to differentiate patients with infectious versus noninfectious etiologies. Cultures are, at times, insensitive, and at other times, nonspecific, leading to a subjective clinical diagnosis. This subjectivity is underscored in the validation study of the new sepsis definition
that used physician behavior, defined as the combination of orders for body fluid cultures and antimicrobials, as a proxy for suspected infection. Many sepsis clinical trials use retrospective physician adjudication of the patients’ medical records to identify infection, a costly and time-consuming task with inherent subjectivity.
Since 1998, the National Healthcare Safety Network (NHSN) and the CDC have provided annually updated criteria for common infections for surveillance of hospital-acquired infections.
However, there is no literature regarding the application of these definitions for the retrospective identification of infection in emergency department (ED) patients for clinical trials research. We modified the NHSN definitions for infection to include only the information typically available for these infection types at the time of an ED visit and within the first 48 hours of admission with the intention to provide a standardized approach that may be used by all healthcare facilities to determine the presence of an infection.
The purpose of the current study is to determine whether modified objective definitions of infection (MODI) from the CDC/NHSN
performed comparably to ED treating physician behavior (e.g., combination of culture orders and ED antibiotic administration) and physician adjudication of the patient record. We hypothesized that there would be low concordance rates between physicians’ identification of infection and that MODI would provide a standardized means of identifying infected patients.
Methods
Patient population
This is a subset of data comparing the MODI criteria to physician adjudication and ED treating physician behavior. This data was retrospectively analyzed from a prospective, observational cohort of high acuity patients with suspected or confirmed infection. Data from this trial have previously been published.
The study was performed in the EDs of two academic medical centers between February and December 2016. Inclusion criteria for the prospective, observational trial were adults who provided informed consent with signs of infection defined as two of four SIRS criteria and at least one indication of organ dysfunction (e.g., elevated serum lactate (> 2mmol/L), hypoxia, hypotension, acute kidney injury, elevated total bilirubin, decreased platelet count, or elevated INR). Exclusion criteria were death expected within 24 hours, unable or unwilling to consent, hematologic malignancy or myelodysplastic, myeloproliferative disorder, or neutropenia. Data from all of the patients enrolled in prospective trial was retrospectively analyzed for the purposes of the current study. Both institutions’ Institutional Review Boards approved this study.
Data abstraction
Data abstracted from the medical record included demographic, physiologic, radiologic, laboratory, diagnostic, and discharge information as well as physician's orders for cultures and medication administered from enrollment to the third hospital day. Patients were identified as having physician suspected infection if the ED physician ordered cultures and antibiotics in the ED.
Adjudication
Two physicians from an adjudication committee retrospectively adjudicated patient charts through independent review of each patient's medical record to determine the presence of infection. If both adjudicators agreed, their answer was used as the final adjudication. In cases of disagreement, a third physician served as a tiebreaker with the majority decision, or consensus, determining the endpoint. Cases in which no physicians agreed were labeled as non-consensus. Whether the individual physician determined that the patient was infected or not, they were required to specify the organ system involved in the infection, insult or injury, respectively.
Modified objective definitions of Infection (MODI) from the CDC/NHSN
We derived the criteria for MODI from the CDC/NHSN surveillance definitions and criteria,
Tennessee Department of Health. About healthcare associated infections (HAI). 2017; Available at: https://tn.gov/health/cedep/hai.html. Accessed March 20, 2022.
Modifications included removing specific window-periods and requirements of hospital stays and interventions, so the criteria included in the MODI would reflect the data available during an ED visit, and, if the patient was hospitalized, the data collected during the first 48 hours of hospital admission. Additionally, we removed certain criteria specific to physician behavior, such as treatment for infection with antibiotics. We included only the most common sites of infection: respiratory, gastrointestinal (GI), urinary, cardiovascular, central nervous system (CNS), skin and soft tissue, and bone and joint in the current study. Authors created forms for abstracting MODI data for each of the major sites (online supplement 1). The definitions and modification for each site of infection are available in the online supplement 2. Finally, these criteria were operationalized in REDCap,
so that coordinators entered data directly into the electronic data capture system.
Trained research coordinators retrospectively abstracted MODI criteria using patients’ medical record. Data pertaining to the ED intake history and physical and review of symptoms was abstracted only for the duration of the ED encounter. All other data was collected for the first 48 hours from study enrollment in accordance to the MODI. Adjudicators were blinded to the MODI, and coordinators were blinded to adjudicators’ decisions.
Statistical analyses
Chi-square analyses were used to compare rates of agreement. Variation in diagnosing the presence or absence of infection was quantified using Fleiss’ Kappa (Fleiss et al 2003). Kappa values greater than .75 indicated strong agreement, between .40-.75 indicated fair to good agreement, and values less than .40 indicated poor agreement. We examined agreement between physician adjudicators’ diagnoses of infected, not infected, and indeterminate. Next, we included all indeterminate cases in the infected category in order to assess agreement between final physician adjudication (infected and not infected) and MODI (infected and not infected). Finally, we compared both retrospective means of identifying infection to ED treating physician behavior (e.g., orders for body fluid cultures and antibiotics). Analyses were run in SPSS Version 26.
Results
Three hundred and ninety-three subjects met inclusion criteria. (Table 1). The two initial adjudicators agreed in 301 (76.6%) of cases, indicating fair to good agreement, with Fleiss Kappa = 0.59. The 92 (23.4% of total) remaining cases required a third adjudicator, and in 71 of these cases, consensus was reached, with two of three adjudicators in agreement. The 21 remaining cases, in which one physician chose infected, one chose not infected, and the third chose indeterminate, were ultimately classified as infected as per the study protocol. In all, there were 173 patients adjudicated as infected and 220 patients adjudicated as not infected (Fig. 1).
Table 1Patient demographic information and infection classification.
Physician Adjudication
MODI Classification
Physician Behavior
Infected
Not Infected
Infected
Not infected
Cx - only
Abx - only
Cx & Abx
N
173
220
162
231
72
21
187
Age, m (SD)
62.8 (18.6)
62.5 (17.6)
65.0 (17.4)
61.0 (18.3)
60.8 (18.5)
62.4 (14.9)
64.1 (17.5)
Gender, n (%)
Male
87 (50.3)
111 (50.5)
82 (50.6)
116 (50.2)
39 (54.9)
12 (57.1)
89 (47.8)
Female
86 (49.7)
109 (49.5)
80 (49.4)
115 (49.8)
32 (45.1)
9 (42.9)
97 (52.2)
Race, n (%)
White
100 (59.5)
115 (52.3)
91 (56.2)
124 (53.7)
39 (54.2)
10 (47.6)
107 (57.2)
Black
68 (40.5)
100 (45.5)
69 (42.6)
99 (42.9)
31 (43.1)
9 (42.9)
76 (40.6)
Other
5 (2.9)
5 (2.3)
2 (1.2)
8 (3.5)
2 (2.8)
2 (9.5)
4 (2.1)
Physician Adjudication, n (%)
Infected
-
-
127 (78.4)
46 (19.9)
21 (29.2)
8 (38.1)
132 (70.6)
Not infected
-
-
35 (21.6)
185 (80.1)
51 (70.8)
13 (61.9)
55 (29.4)
MODI Classification, n (%)
Infected
-
-
-
-
23 (31.9)
7 (33.3)
117 (62.6)
Not Infected
-
-
-
-
49 (68.1)
14 (66.7)
70 (37.4)
Source of involvement, n (%)
Bone & Joint
8 (4.6)
4 (1.8)
6 (3.7)
CNS
1 (0.6)
11 (5)
0 (-)
Gastrointestinal
27 (15.6)
33 (15)
19 (11.7)
Genitourinary
27 (15.6)
7 (3.1)
18 (11.1)
Primary Blood
11 (6.3)
5 (2.2)
17 (10.5)
Respiratory
56 (32.4)
51 (23.2)
87 (53.7)
Skin-Skin / Wound
21 (12.1)
0 (-)
15 (9.3)
Multiple Sources
19 (26.0)
45 (20.4)
0 (-)
Other
2 (1.2)
64 (29)
0 (-)
Note: Cx only = Culture Orders Only; Abx Only = Antibiotic Orders Only; Cx & Abx = Culture and Antibiotic Orders.
Of the 393 patients included, 162 (41.2%) met MODI classification of infection. Fleiss's Kappa indicated fair to good agreement between the MODI and final physician adjudication (Kappa = 0.58; PA = 79.4%). There were 35 patients identified as infected by MODI criteria but adjudicated as not infected by physician adjudication. Of those, 32 met MODI criteria for a Respiratory infection and three for other sources for infection. Of the 46 patients adjudicated as infected by physicians but not classified as infected by MODI, source of infection listed by physician consensus was Respiratory for 14 cases, GI for 10, Skin for 10, 9 other cases had other sources listed. Three cases did not have a physician consensus for source of infection.
We compared physician adjudication to treating ED physician behavior and MODI classification to ED physician behavior. Agreement was fair to good (PA = 75.6%, Kappa = 0.50) when ED physician behavior was compared to physician adjudication. When MODI classification and treating ED physician behavior were compared, PA and kappa were slightly lower (70.7%, 0.41).
Discussion
The primary objective of the current investigation was to compare three methods of identifying infection for sepsis clinical trials: modified CDC/NHSN definitions of infection (MODI) for identification of infection, physician adjudication, and ED treating physician behavior. The current study highlights the complexity of diagnosing infection both in real time (ED treating physician behavior) and retrospectively (using the entire medical record). Over one in five patients required tiebreak. Further, three physicians could not reach consensus about the presence of infection in 21 cases. These findings are consistent with the current literature suggesting that disagreement between physicians on the diagnosis of infection or identification of the source of infection is high.
Physician agreement on the diagnosis of sepsis in the intensive care unit: estimation of concordance and analysis of underlying factors in a multicenter cohort.
MODI definitions are based on NHSN/CDC definitions of infection used widely and in a standardized fashion by infection preventionists to identify hospital acquired infections. Benefits of using a modified NHSN definition include that they are easily adopted across different facilities and can be followed by both clinical and data teams for abstraction. Unfortunately for identifying sepsis on presentation, some of the definitions require modification as not all clinical data is accounted for in the first 48 hours of the hospital visit. NHSN definitions also sometimes include physician behavior as part of definition. For the purpose of adjudication, physician behavior was removed so that criteria for infection included objective data. The MODI criteria for infection demonstrated comparable rates of concordance with final physician adjudication. The MODI criteria offer a standardized means of diagnosing infection for clinical trials that removes variability inherent in physician adjudication and saves physicians significant time and effort. The MODI criteria may also provide an objective and standardized means for the comparison of patient populations from different sites of a multi-centered study.
One of the Infectious Diseases Society of America (IDSA's) primary challenges of the Sepsis-3 guidelines was the Task Force's failure to acknowledge the inherent difficulty in identifying infection and differentiating sepsis from non-infectious syndromes.
The results of our current study demonstrate that treating ED physician behavior had a slightly lower percent agreement and kappa compared to MODI classification and physician adjudication.
Limitations include the lack of a gold standard for diagnosing sepsis or infection. As a result of the variability among physician adjudicators, it may be difficult to assess the exact utility of the MODI criteria. Since we relied on retrospective review of clinical data in the medical record, lack of specific documentation and key words may lead to misclassification. In addition, there is some degree of inter-relation among the three standards, as adjudicators and study personnel reviewing for MODI criteria were bound by data obtained during routine clinical treatment of individual cases. Finally, the data was collected at two institutions in a large metropolitan area in the Southeast United States, and both sites were adjudicated by the same research team. Additional studies will be needed to assess the generalizability of these results.
In summary, the current data validates the IDSA's concerns regarding the variability in identifying patients who are infected from those with noninfectious etiologies. Using MODI criteria was as good as other means of classifying objectively and has some benefits such as standardizing the definition of infection by removing some of the subjectivity inherent in clinical decision-making, as well as relieving physicians of the time and effort required to make similar decisions through retrospective assessments. Finally, these criteria are easily implemented in an electronic data capturing system, improving efficiency and making remote assessment for infection possible. The MODI criteria offer an alternative to using physician adjudication in sepsis clinical trials by standardizing definitions of infection and allowing for an objective comparison of subjects across sites.
Autor Contributions
CO and HO developed the idea for creating modified definitions of infection. MM and DH were primarily responsible for statistical analyses. All authors were responsible for making intellectual contributions regarding refining the definitions, collecting data, and editing the manuscript.
Source of funding
This is a subset of data from a prospective, observational cohort that was funded by CytoVale, Inc, San Francisco, CA.
Declaration of Competing Interest
The authors have no conflicts of interest to declare.
Acknowledgements
The authors would also like to express their gratitude to the Office of Research at Our Lady of the Lake Regional Medical Center and the personnel who worked tirelessly to screen and enroll patients: Jennifer Daigle, BSN; Brittany Rishell, BSN; Katie Vance, PhD; JoAnne Hollingsworth, BSN, Allie Piehet, BS, and Emily Perdue, RN. Lastly, the authors would like to recognize the contributions of the team at Cytovale for their work in the set-up and completion of the prospective study from which the data from this project was collected: Roya Sheybani, Henry Tse, Ajay Shah, Anne Jensen, Audun Johnson, Bliss Lambert, Chris Dahlberg, Hon Seng, Kylie Ortego, Lionel Guillou, Maddy Hem and Nicholas Martinez.
Tennessee Department of Health. About healthcare associated infections (HAI). 2017; Available at: https://tn.gov/health/cedep/hai.html. Accessed March 20, 2022.
Physician agreement on the diagnosis of sepsis in the intensive care unit: estimation of concordance and analysis of underlying factors in a multicenter cohort.
00080-5&id=gr1.jpg){kind=link}