An Automated Electronic Tool to Assess the Risk of 30-Day Readmission: Validation of Predictive Performance

From the Divisions of Hospital Medicine (Drs. Dawson, Chirila, Bhide, and Burton) and Biomedical Statistics and Informatics (Ms. Thomas), Mayo Clinic, Jacksonville, FL, and the Division of Hospital Medicine, Mayo Clinic, Phoenix, AZ (Dr. Cannon).

Abstract

• Objective: To validate an electronic tool created to identify inpatients who are at risk of readmission within 30 days and quantify the predictive performance of the readmission risk score (RRS).
• Methods: Retrospective cohort study including inpa-tients who were discharged between 1 Nov 2012 and 31 Dec 2012. The ability of the RRS to discriminate between those who did and did not have a 30-day urgent readmission was quantified by the c statistic. Calibration was assessed by plotting the observed and predicted probability of 30-day urgent readmission. Predicted probabilities were obtained from generalized estimating equations, clustering on patient.
• Results: Of 1689 hospital inpatient discharges (1515 patients), 159 (9.4%) had a 30-day urgent readmission. The RRS had some discriminative ability (c statistic: 0.612; 95% confidence interval: 0.570–0.655) and good calibration.
• Conclusions: Our study shows that the RRS has some discriminative ability. The automated tool can be used to estimate the probability of a 30-day urgent readmission.

Hospital readmissions are increasingly scrutinized by the Center for Medicare and Medicaid Services and other payers due to their frequency and high cost. It is estimated that up to 25% of all patients discharged from acute care hospitals are readmitted within 30 days [1]. To address this problem, the Center for Medicare and Medicaid Services is using these rates as one of the benchmarks for quality for hospitals and health care organizations and has begun to assess penalties to those institutions with the highest rates. This scrutiny and the desire for better patient care transitions has resulted in most hospitals implementing various initiatives to reduce potentially avoidable readmissions.

Multiple interventions have been shown to reduce readmissions [2,3]. These interventions have varying effectiveness and are often labor intensive and thus costly to the institutions implementing them. In fact, no one intervention has been shown to be effective alone [4], and it may take several concurrent interventions targeting the highest risk patients to improve transitions of care at discharge that result in reduced readmissions. Many experts do recommend risk stratifying patients in order to target interventions to the highest risk patients for effective use of resources [5,6]. Several risk factor assessments have been proposed with varying success [7–13]. Multiple factors can limit the effectiveness of these risk stratification profiles. They may have low sensitivity and specificity, be based solely on retrospective data, be limited to certain populations, or be created from administrative data only without taking psychosocial factors into consideration [14].

An effective risk assessment ideally would encompass multiple known risk factors including certain comorbidities such as malignancy and heart failure, psychosocial factors such as health literacy and social support, and administrative data including payment source and demographics. All of these have been shown in prior studies to contribute to readmissions [7–13]. In addition, availability of the assessment early in the hospitalization would allow for interventions throughout the hospital stay to mitigate the effect of these factors where possible. To address these needs, our institution formed a readmission task force in January 2010 to review published literature on hospital 30-day readmissions and create a readmission risk score (RRS). The aim of this study was to quantify the predictive performance of the RRS after it was first implemented into the electronic medical record (EMR) in November 2012.

Methods

Study Design and Cohort

All consecutive adult inpatients who were discharged between 1 November 2012 and 31 December 2012 were included in this retrospective cohort study. This narrow time frame corresponded to the period from RRS tool implementation to the start of readmission interventions. We excluded hospitalizations if the patient died in the hospital.

Outcome Measures

The primary outcome was a 30-day urgent readmission, which included readmissions categorized as either emergency, urgent, or semi-urgent. Secondary outcomes included any 30-day readmission and 30-day death. Only readmissions to Mayo Clinic were examined.

Predictors

In collaboration with the information technology department, an algorithm was written to extract data from the EMR for each patient within 24 hours of admission to the hospital. This data was retrieved from existing repositories of patient information, such as demographic information, payer source, medication list, problem list, and past medical history. In addition, each patient was interviewed by a nurse at the time of admission, and the nurse completed an “admission profile” in the EMR that confirmed or entered past medical history, medications, social support at home, depression symptoms, and learning styles, among other information (Table 1). The algorithm was able to extract data from this evaluation also, so that each element of the risk score was correlated to at least one data source in the EMR. The algorithm then assigned the correct value to each element, and the total score was electronically calculated and placed in a discrete cell in each patient’s record. The algorithm was automatically run again 48 hours after the initial scoring in order to assure completeness of the information. If the patient had a length of stay greater than 5 days, an additional score was generated to include the length of stay component.

Statistical Analysis

The predictive performance of the RRS was assessed by evaluating the discrimination and calibration. Discrimination is the ability of the RRS to separate those who had a 30-day urgent readmission and those who did not. Discrimination was quantified by the c statistic, which is equivalent to the area under the receiver operating characteristic curve in this study owing to the use of binary endpoints. A c statistic of 1.0 would indicate that the RRS perfectly predicts 30-day urgent readmission while a c statistic of 0.5 would indicate the RRS has no apparent accuracy in predicting 30-day urgent readmission. Calibration assesses how closely predicted outcomes agree with observed outcomes. The predicted probability of 30-day urgent readmission was estimated utilizing a generalized estimating equation model, clustering on patient, with RRS as the only predictor variable. Inpatient discharges were divided into deciles of the predicted probabilities for 30-day urgent readmission. Agreement of the predicted and observed outcomes was displayed graphically according to decile of the predicted outcomes. All analyses were performed using SAS (version 9.3, SAS Institute, Cary, NC) and R statistical software (version 3.1.1, R Foundation for Statistical Computing, Vienna, Austria).