Beyond Reporting Early Warning Score Sensitivity: The Temporal Relationship and Clinical Relevance of “True Positive” Alerts that Precede Critical Deterioration
BACKGROUND: Clinical deterioration is difficult to detect in hospitalized children. The pediatric Rothman Index (pRI) is an early warning score that incorporates vital signs, laboratory studies, and nursing assessments to generate deterioration alerts.
OBJECTIVES: (1) Evaluate the timing of pRI alerts and clinicians recognizing deterioration or escalating care prior to critical deterioration events (CDEs) and (2) determine whether the parameters triggering alerts were clinically related to deterioration.
DESIGN: CDEs are unplanned transfers to the intensive care unit with noninvasive ventilation, tracheal intubation, and/or vasopressor infusion in the 12 hours after transfer. Using one year of data from a large freestanding children’s hospital without the pRI, we analyzed CDEs that would have been preceded by pRI alerts. We (1) compared the timing of pRI alerts to time-stamped notes describing changes in patient status and orders reflecting escalations of care and (2) identified score component(s) that caused alerts to trigger and determined whether these were clinically related to CDE etiology.
RESULTS: Fifty CDEs would have triggered pRI alerts if the pRI had been in use (sensitivity 68%). In 90% of CDEs, the first clinician note reflecting change in patient status and/or the first order reflecting escalation of care preceded the first pRI alert. All of the vital sign and laboratory components of the pRI and 51% of the nursing components were clinically related to the etiology of the CDE.
CONCLUSIONS: Evidence that clinicians were aware of deterioration preceded pRI alerts in most CDEs that generated alerts in the preceding 24 hours.
© 2018 Society of Hospital Medicine.
Patients at risk for clinical deterioration in the inpatient setting may not be identified efficiently or effectively by health care providers. Early warning systems that link clinical observations to rapid response mechanisms (such as medical emergency teams) have the potential to improve outcomes, but rigorous studies are lacking.1 The pediatric Rothman Index (pRI) is an automated early warning system sold by the company PeraHealth that is integrated with the electronic health record. The system incorporates vital signs, labs, and nursing assessments from existing electronic health record data to provide a single numeric score that generates alerts based on low absolute scores and acute decreases in score (low scores indicate high mortality risk).2 Automated alerts or rules based on the pRI score are meant to bring important changes in clinical status to the attention of clinicians.
Adverse outcomes (eg, unplanned intensive care unit [ICU] transfers and mortality) are associated with low pRI scores, and scores appear to decline prior to such events.2 However, the limitation of this and other studies evaluating the sensitivity of early warning systems3-6 is that the generated alerts are assigned “true positive” status if they precede clinical deterioration, regardless of whether or not they provide meaningful information to the clinicians caring for the patients. There are two potential critiques of this approach. First, the alert may have preceded a deterioration event but may not have been clinically relevant (eg, an alert triggered by a finding unrelated to the patient’s acute health status, such as a scar that was newly documented as an abnormal skin finding and as a result led to a worsening in the pRI). Second, even if the preceding alert demonstrated clinical relevance to a deterioration event, the clinicians at the bedside may have been aware of the patient’s deterioration for hours and have already escalated care. In this situation, the alert would simply confirm what the clinician already knew.
To better understand the relationship between early warning system acuity alerts and clinical practice, we examined a cohort of hospitalized patients who experienced a critical deterioration event (CDE)7 and who would have triggered a preceding pRI alert. We evaluated the clinical relationship of the alert to the CDE (ie, whether the alert reflected physiologic changes related to a CDE or was instead an artifact of documentation) and identified whether the alert would have preceded evidence that clinicians recognized deterioration or escalated care.
