Comparison of clinical scores for predicting stroke-associated pneumonia after intracerebral hemorrhage (ICH): potential tools for personalized care and clinical trials for ICH

Introduction

Spontaneous intracerebral hemorrhage (ICH) accounts for approximately 15–20% of all strokes and is one of the leading causes of mortality and morbidity worldwide (1,2). Despite advances being made in medical knowledge, treatment for ICH remains strictly supportive, with few evidence-based interventions currently available (3,4).

Stroke-associated pneumonia (SAP) is a common medical complication after stroke and has a significant impact on patient outcomes. Evidence has shown that SAP not only increases the length of stay (LOS) in hospital and medical costs, but is also an important risk factor for poststroke mortality and morbidity (5). In addition, pneumonia has been found to increase the risk of several non-pneumonia-related medical complications, such as deep vein thrombosis, gastrointestinal bleeding, and atrial fibrillation (6). In a previous study, SAP was also found to be more common in patients with ICH than in those with acute ischemic stroke (AIS) (6,7). These data point to the need for more aggressive SAP prophylaxis among patients with ICH.

Several clinical scores have been developed for predicting SAP after ICH, such as the Veterans Health Administration (VHA) risk score (8), intracerebral hemorrhage-associated pneumonia score (ICH-APS; both ICH-APS-A and ICH-ASP-B) (9), the integer-based pneumonia risk score (ISAN) (10), the ACCD4 (age ≥75 years =1; congestive heart failure =1; dysarthria =1; dysphagia =4) (11), and the Preventive Antibiotics in Stroke Study (PASS) score (12). In the current study, the ISAN (10), ACCD4 (11), and PASS (12) scores were used for ICH patients in the derivation cohort, while the ICH-APS-A (9) and ICH-APS-B (9) scores were used exclusively for the ICH validation cohort (Tables S1,S2). Although a few of these ICH risk models have been internally or externally validated, none are universally accepted or consistently used in routine clinical practice or clinical research. In addition, with many grading systems available, it is becoming increasingly difficult for clinicians and researchers to determine which risk model provides the optimal predictability and reliability in clinical practice and clinical trials. Therefore, we believe it is necessary to conduct a head-to-head comparison of these models in an independent cohort.

In this study, we aimed to systematically compare the discrimination and calibration of 5 clinical scores for in-hospital SAP after ICH following the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) guideline (13,14). We present the following article in accordance with the Standards for Reporting Diagnostic Accuracy Studies (STARD) reporting checklist (available at https://atm.amegroups.com/article/view/10.21037/atm-21-4046/rc).

Methods

Validation cohort

This was a multicenter, prospective, observational cohort study, with the validation cohort being derived from the Beijing Registration of Intracerebral Hemorrhage (15). A total of 13 hospitals in the Beijing area participated. To be eligible, patients had to meet the following criteria: (I) age 18 years or older; (II) hospitalized with a primary diagnosis of spontaneous ICH and confirmed by brain computed tomography (CT) or magnetic resonance imaging (MRI); (III) direct admission to hospital from a physician’s clinic or emergency department; and (IV) written informed consent from patients or their legal representatives. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study protocol was approved by the Institutional Review Board (IRB) of the Beijing Tiantan Hospital (No. KY2014-023-02), and informed consent was given by all patients.

Data collection and definition of variables

A standardized electronic case report form (eCRF) was used for data collection. Participating centers collected data and submitted it online to the coordinating center at Beijing Tiantan Hospital. For this study, the following candidate variables were analyzed: (I) demographics; (II) time from onset to hospitalization (hours); (III) stroke risk factors, including hypertension, diabetes mellitus, dyslipidemia, atrial fibrillation, history of stroke/transient ischemic attack (TIA), myocardial infarction, heart failure, and current smoking or alcohol consumption; (IV) prestroke modified Rankin Scale (mRS) score; (V) preadmission antithrombotic medications; (VI) admission stroke severity based on the National Institutes of Health Stroke Scale (NIHSS) score and the Glasgow Coma Scale (GCS) score; (VII) admission blood pressure (mmHg); (VIII) admission laboratory tests; (IX) neuroimaging variables, including hematoma volume (measured using the ABC/2 method) (16), hematoma location (supratentorial or infratentorial ICH), intraventricular extension (presence or absence), and subarachnoid extension (presence or absence); (X) etiology diagnosis (primary or secondary ICH); (XI) surgical treatment (craniotomy evacuation, minimally invasive surgical therapy, or brain ventricle puncture and drainage); (XII) withdrawal of medical care; and (XIII) LOS in hospital.

Diagnosis of SAP

In-hospital SAP was diagnosed by the treating physician according to the Center for Disease Control and Prevention’s criteria for hospital-acquired pneumonia (17,18). This included a basis of clinical and laboratory indices of respiratory tract infection (fever, cough, new purulent sputum, and auscultatory respiratory crackles) and was supported by typical chest X-ray findings. In this study, the occurrence of prestroke pneumonia was not considered.

Statistical analysis

For our statistical analysis, categorical variables are summarized as proportions, while continuous variables are summarized with mean and SD, or median and interquartile range (IQR). Chi-square or Fisher’s exact test was used to compare categorical variables, and a Mann-Whitney test or independent t-test was employed to compare continuous variables between groups. Through a systematic search, we identified 6 clinical scores which could be used to predict SAP after ICH. However, the VHA score (8) could not be validated in the study as we did not have information on “found down at symptom onset”. This left us with 5 clinical scores for our study: ICH-APS-A (9), ICH-ASP-B (9), ISAN (10), ACCD4 (11), and PASS (12). Discrimination was assessed by calculating the area under the receiver operating characteristic curve (AUROC) (13,14). Pairwise AUROC was compared with Delong’s method (19). Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated at each risk model’s maximum Youden index. Calibration, on the other hand, was evaluated using a Hosmer-Lemeshow goodness-of-fit test, and observed versus predicted risk was plotted according to 10 deciles of the predicted risk. We were then able to calculate the Cox and Snell R² and Nagelkerke R² (13,14). As the Hosmer-Lemeshow test has been shown to be overly sensitive (20), we also used Pearson correlation coefficient to calculate the observed and predicted risk. All tests were 2-tailed, and statistical significance was determined at a P level of .05. Statistical analysis was conducted using SAS 9.1 (SAS Institute Inc., Cary, NC, USA), SPSS 26.0 (IBM Corp., Armonk, NY, USA), and MedCalc 12.3 (MedCalc Software Ltd., Ostend, Belgium) software.

Results

Patient characteristics

From December 2014 to September 2016, a total of 1964 patients were enrolled in the Beijing Registration of Intracerebral Hemorrhage. Their clinical characteristics are shown in Table 1. The mean age was 56.8±14.4 years, and 67.6% were male. The median time from onset to hospitalization was 4.0 h (IQR, 1.90–11.1 h), and the median NIHSS and GCS score on admission was 11 (IQR, 3–21) and 14 (IQR, 8–15), respectively. The median hematoma volume on CT was 15.8 cm³ (IQR, 6.0–38.6 cm³), while the median LOS was 16 days (IQR, 8–22 days). A total of 575 (29.2%) patients were diagnosed with in-hospital SAP after ICH. Compared to patients without in-hospital SAP, those with in-hospital SAP after ICH were older and had a higher proportion of dysphagia, dysarthria, hematoma intraventricular extension, and subarachnoid extension. Furthermore, they had received surgical treatment and had a higher NIHSS score (at admission), blood pressure, blood glucose level, and hematoma volume, as well as a longer LOS (Table 1).

LOS and in-hospital SAP after ICH

Previous studies have shown that predicting SAP with a clinical score is more effective for patients who survive 48–72 h after ICH. For this reason, we performed a sensitivity analysis by LOS. Patient characteristics stratified by LOS (≤72 vs. >72 h) are shown in Table 2. Compared to patients with an LOS >72 h, those with an LOS ≤72 h had a significantly higher NIHSS score (P<0.001) upon admission, a lower GCS score (P<0.001), and a larger hematoma volume (P<0.001). However, the proportion of in-hospital SAP between the 2 groups was not statistically significant (30.2% vs. 29.2%; P=0.75). It was also found that patients with an LOS ≤72 h had a significantly higher rate of in-hospital mortality (40.6% vs. 7.4%; P<0.001) and withdrawal of medical care (18.4% vs. 5.7%; P<0.001; Table 2).

Comparison of model discrimination for in-hospital SAP

Figure 1 shows the discrimination of the 5 clinical scores regarding in-hospital SAP after ICH. The AUROC of the 5 scores ranged from 0.732 to 0.800 (Table 3). The sensitivity, specificity, PPV, NPV, and maximum Youden index for predicting in-hospital SAP after ICH are shown in Table 3. We found that the ICH-SAP-B had the maximum value in the Youden index. In the comparison of scores, the ICH-APS-B (0.800; 95% CI: 0.780–0.820; P<0.001) also showed a statistically better discrimination than did the other risk models for in-hospital SAP after ICH (all P<0.001). However, all risk models showed a much better discrimination in patients with an LOS >72 h (Table 3). Among these patients with an LOS >72 h, the ICH-APS-B (0.827; 95% CI: 0.806–0.848; P<0.001) still showed statistically better discrimination for in-hospital SAP after ICH than did the other risk models (all P<0.001).

Figure 1 Predictive performance of clinical scores regarding in-hospital SAP after ICH (n=1,964). PASS, Preventive Antibiotics in Stroke Study Score; ISAN, integer-based pneumonia risk score; ACCD4, an 8-point pneumonia prediction scale (age ≥75 years =1; congestive heart failure =1; dysarthria =1; dysphagia =4); ICH-APS, intracerebral hemorrhage-associated pneumonia score; SAP, stroke-associated pneumonia.

Comparison of model calibration for in-hospital SAP

We plotted the predicted and observed risk according to 10 deciles of the predicted risk of in-hospital SAP after ICH (Figure S1). The results of the Hosmer-Lemeshow test are shown in Table 4. In the overall cohort, the Hosmer-Lemeshow test showed that the ICH-APS-B had a significance level greater than .05, indicating that the observed values were not statistically different from the expected values. The ICH-SAP-B had the largest Cox and Snell R². Similar results were found for patients with an LOS >72 h. As the Hosmer-Lemeshow test has been shown to be overly sensitive to trivial deviations from the ideal fit, Pearson correlation coefficient was also used to calculate the predicted and observed risk. All the correlation coefficients of the 5 models were greater than 0.90 (all P<0.001), and the ICH-SAP-B had the largest Pearson correlation coefficient.

Discussion

In this study, we systematically compared the discrimination and calibration of 5 clinical scores regarding in-hospital SAP after ICH. The AUROC ranged from 0.732 to 0.800 in the overall cohort. Through a comparative analysis, we found the ICH-APS-B showed statistically better discrimination than did the other risk models in terms of in-hospital SAP after ICH. Furthermore, all risk models showed much better discrimination among patients with an LOS >72 h. The ICH-APS-B still showed statistically better discrimination than did the other risk models for in-hospital SAP after ICH in patients with an LOS >72 h. The Hosmer-Lemeshow test also revealed that the ICH-SAP-B had the largest Cox and Snell R² among patients with an LOS >72 h.

Previous studies have shown that clinical scores for predicting SAP performed better in sensitivity analyses for patient survival beyond 48–72 h after ICH (9,10). We found the potential reasons for this to be worth investigating. To this end, we compared the baseline characteristics of patients with an LOS ≤72 h and those with an LOS >72 h and found that patients with a shorter LOS had a significantly more severe neurological deficit on admission as indicated by a higher NIHSS score, lower GCS score, and larger hematoma volume. However, these factors did not correspond to an increased risk of in-hospital SAP after ICH. The rates of in-hospital SAP between the 2 groups were not statistically different. Furthermore, we found that patients with a shorter LOS had a significantly higher proportion of in-hospital mortality and withdrawal of medical care (Table 2). For our study, limited data were available about the time course of SAP after stroke. However, previous studies have reported a 3-day (IQR, 2–5) median time from onset to diagnosis of SAP after ICH), a 4-day median for AIS (IQR, 2–7), and a 5-day median for subarachnoid hemorrhage (SAH; IQR, 3–7) (21). Therefore, based on these data, we conjectured that the contradiction between neurological severity and risk of in-hospital SAP after ICH in patients with an LOS ≤72 h might be due to the patient dying or leaving hospital before the pneumonia occurred. For this patient subgroup of in-hospital pneumonia after ICH, this might be a potential reason for the lower sensitivity of these clinical scores. Despite advances in medical knowledge, treatment for ICH remains strictly supportive, with few evidence-based interventions currently available (3,4). Evidence has shown that SAP not only increases LOS and medical costs, but is also an important risk factor for poststroke mortality and morbidity. In addition, a previous study has shown the rate of SAP to be higher among patients with ICH than those with AIS (6). These data suggest that compared to patients with AIS, patients with ICH require a more aggressive SAP prophylaxis. Several clinical scores have been developed for predicting SAP after ICH. Although some risk models have been internally or externally validated, none of them have been universally accepted or consistently used in routine clinical practice and clinical research. In this study’s large ICH cohort (n=1,964), the ICH-APS-B showed a statistically better discrimination than did the other risk models regarding in-hospital SAP after ICH. It also displayed the largest Cox and Snell R² through the Hosmer-Lemeshow goodness-of-fit test, which was used to predict in-hospital SAP after ICH. Although these results are promising, caution needs to be taken when interpreting the results, as the cohorts used to develop these clinical scores were different. In other words, the ISAN, ACCD4, and PASS scores were only used for ICH patients in the derivation cohort, while the ICH-APS-A and ICH-APS-B scores were used exclusively for the ICH validation cohort. Furthermore, the baseline characteristics between these 2 study groups were different, and there might be complex genetic, social, and economic factors, as well as regional management philosophies and preferences, that are difficult to account for when risk models are developed or applied to a distinct population. Therefore, these clinical scores need to be further validated by a study with a larger sample size.

In 2 recent large, randomized trials [STROKE-INF (22) and PASS (23)], preventive antibiotic therapy did not improve functional outcomes after stroke. These trials selected patients based on symptoms and signs, with a “one-size-fits-all” prevention strategy being developed for the average person. This approach gave less consideration to the differences in SAP risk between individuals. In this way, it was inevitable that they would include patients with an unbalanced, too-high, or too-low risk of developing SAP in their clinical trials. In our study, we found the clinical scores we validated could be used to stratify patients by risk of SAP after stroke, meaning they could aid the testing of prophylactic antibiotics in different risk stratifications. Clinical trials conducted in this way would allow one to more accurately clarify which prophylactic antibiotics will work better for different SAP stratifications. However, whether preventive antibiotic therapy in a high-risk subgroup of SAP patients can improve functional outcomes after ICH requires further investigation.

It is also important to note that our study has certain limitations. First, like all observational studies, we cannot rule out the possibility that additional baseline variables (unmeasured confounders), such as use of angiotensin-converting enzyme inhibitors (24), acid-suppressive medications (25), oral hygiene (26), and biomarkers for stroke-induced immunodepression (27,28), might have some impact on the development of in-hospital SAP after ICH. Second, we could not include all elements required for all risk models. For example, we did not have information about the “found down at symptom onset”, and the VHA score (8) could not be validated. Third, our study included only hospitalized patients, and patients who died in the emergency room or who were treated in outpatient clinics were not included. Furthermore, like most registries, our registry required informed consent, and selection bias was therefore inevitable (29). Lastly, the validation cohorts originated from an Asian population, and thus the ICH models need to be further validated in different populations.

Conclusions

In summary, among the 5 models for predicting SAP after ICH, the ICH-APS-B showed the best predictive performance. This indicates that the model can be applied in personalized care and clinical trials of SAP after ICH.

Acknowledgments

Funding: This study was sponsored by Beijing Municipal Science & Technology Commission (Z131107002213009), the Nova Program of Beijing Science and Technology Commission (2008B30), the National Natural Science Foundation of China (81471208 and 81641162), the Beijing High-Level Healthy Human Resource Project (014-3-033), and the Shandong Province Key Innovation Project (2019JZZY020901).

Footnote

Reporting Checklist: The authors have completed the STARD reporting checklist. Available at https://atm.amegroups.com/article/view/10.21037/atm-21-4046/rc

Data Sharing Statement: Available at https://atm.amegroups.com/article/view/10.21037/atm-21-4046/dss

Peer Review File: Available at https://atm.amegroups.com/article/view/10.21037/atm-21-4046/prf

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://atm.amegroups.com/article/view/10.21037/atm-21-4046/coif). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study protocol was approved by the Institutional Review Board (IRB) of the Beijing Tiantan Hospital (No. KY2014-023-02). Informed consent was given by all patients involved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Virani SS, Alonso A, Benjamin EJ, et al. Heart Disease and Stroke Statistics-2020 Update: A Report From the American Heart Association. Circulation 2020;141:e139-596. [Crossref] [PubMed]
2. Wang W, Jiang B, Sun H, et al. Prevalence, Incidence, and Mortality of Stroke in China: Results from a Nationwide Population-Based Survey of 480 687 Adults. Circulation 2017;135:759-71. [Crossref] [PubMed]
3. Hemphill JC 3rd, Greenberg SM, Anderson CS, et al. Guidelines for the Management of Spontaneous Intracerebral Hemorrhage: A Guideline for Healthcare Professionals From the American Heart Association/American Stroke Association. Stroke 2015;46:2032-60. [Crossref] [PubMed]
4. Steiner T, Al-Shahi Salman R, Beer R, et al. European Stroke Organisation (ESO) guidelines for the management of spontaneous intracerebral hemorrhage. Int J Stroke 2014;9:840-55. [Crossref] [PubMed]
5. Teh WH, Smith CJ, Barlas RS, et al. Impact of stroke-associated pneumonia on mortality, length of hospitalization, and functional outcome. Acta Neurol Scand 2018;138:293-300. [Crossref] [PubMed]
6. Ji R, Wang D, Shen H, et al. Interrelationship among common medical complications after acute stroke: pneumonia plays an important role. Stroke 2013;44:3436-44. [Crossref] [PubMed]
7. Kwan J, Pickering RM, Kunkel D, et al. Impact of stroke-associated infection on long-term survival: a cohort study. J Neurol Neurosurg Psychiatry 2013;84:297-304. [Crossref] [PubMed]
8. Chumbler NR, Williams LS, Wells CK, et al. Derivation and validation of a clinical system for predicting pneumonia in acute stroke. Neuroepidemiology 2010;34:193-9. [Crossref] [PubMed]
9. Ji R, Shen H, Pan Y, et al. Risk score to predict hospital-acquired pneumonia after spontaneous intracerebral hemorrhage. Stroke 2014;45:2620-8. [Crossref] [PubMed]
10. Smith CJ, Bray BD, Hoffman A, et al. Can a novel clinical risk score improve pneumonia prediction in acute stroke care? A UK multicenter cohort study. J Am Heart Assoc 2015;4:e001307. [Crossref] [PubMed]
11. Kumar S, Marchina S, Massaro J, et al. ACDD4 score: A simple tool for assessing risk of pneumonia after stroke. J Neurol Sci 2017;372:399-402. [Crossref] [PubMed]
12. Westendorp WF, Vermeij JD, Hilkens NA, et al. Development and internal validation of a prediction rule for post-stroke infection and post-stroke pneumonia in acute stroke patients. Eur Stroke J 2018;3:136-44. [Crossref] [PubMed]
13. Moons KG, Altman DG, Reitsma JB, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1-73. [Crossref] [PubMed]
14. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015;350:g7594. [Crossref] [PubMed]
15. Ji R, Wang W, Liu X, et al. Head-to-head comparison of prognostic models of spontaneous intracerebral hemorrhage: tools for personalized care and clinical trial in ICH. Neurol Res 2022;44:146-55. [Crossref] [PubMed]
16. Kothari RU, Brott T, Broderick JP, et al. The ABCs of measuring intracerebral hemorrhage volumes. Stroke 1996;27:1304-5. [Crossref] [PubMed]
17. Garner JS, Jarvis WR, Emori TG, et al. CDC definitions for nosocomial infections, 1988. Am J Infect Control 1988;16:128-40. [Crossref] [PubMed]
18. Smith CJ, Kishore AK, Vail A, et al. Diagnosis of Stroke-Associated Pneumonia: Recommendations From the Pneumonia in Stroke Consensus Group. Stroke 2015;46:2335-40. [Crossref] [PubMed]
19. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988;44:837-45. [Crossref] [PubMed]
20. Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited. Crit Care Med 2007;35:2052-6. [Crossref] [PubMed]
21. Liu X, Wang L, Li G, et al. Characteristics of Stroke Associated Pneumonia by Stroke Subtypes: Incidence, Time Course, Potential Risk and Microbiological Pathogen. Int J Cerebrovasc Dis Stroke 2021;4:129.
22. Kalra L, Irshad S, Hodsoll J, et al. Prophylactic antibiotics after acute stroke for reducing pneumonia in patients with dysphagia (STROKE-INF): a prospective, cluster-randomised, open-label, masked endpoint, controlled clinical trial. Lancet 2015;386:1835-44. [Crossref] [PubMed]
23. Westendorp WF, Vermeij JD, Zock E, et al. The Preventive Antibiotics in Stroke Study (PASS): a pragmatic randomised open-label masked endpoint clinical trial. Lancet 2015;385:1519-26. [Crossref] [PubMed]
24. Caldeira D, Alarcão J, Vaz-Carneiro A, et al. Risk of pneumonia associated with use of angiotensin converting enzyme inhibitors and angiotensin receptor blockers: systematic review and meta-analysis. BMJ 2012;345:e4260. [Crossref] [PubMed]
25. Marchina S, Doros G, Modak J, et al. Acid-suppressive medications and risk of pneumonia in acute stroke patients: A systematic review and meta-analysis. J Neurol Sci 2019;400:122-8. [Crossref] [PubMed]
26. Wagner C, Marchina S, Deveau JA, et al. Risk of Stroke-Associated Pneumonia and Oral Hygiene. Cerebrovasc Dis 2016;41:35-9. [Crossref] [PubMed]
27. Mengel A, Ulm L, Hotter B, et al. Biomarkers of immune capacity, infection and inflammation are associated with poor outcome and mortality after stroke - the PREDICT study. BMC Neurol 2019;19:148. [Crossref] [PubMed]
28. Nam KW, Kim TJ, Lee JS, et al. High Neutrophil-to-Lymphocyte Ratio Predicts Stroke-Associated Pneumonia. Stroke 2018;49:1886-92. [Crossref] [PubMed]
29. Tu JV, Willison DJ, Silver FL, et al. Impracticability of informed consent in the Registry of the Canadian Stroke Network. N Engl J Med 2004;350:1414-21. [Crossref] [PubMed]