Pneumonia Severity Assessment Tools for Predicting Mortality in Patients with Healthcare-Associated Pneumonia: A Systematic Review and Meta-Analysis

Healthcare-associated pneumonia (HCAP) is a category of pneumonia that was proposed in the American Thoracic Society (ATS)/Infectious Diseases Society of America (IDSA) guidelines in 2005 [1]. Recent reports have shown that 17.3-38.0% of outpatients with pneumonia suffer from HCAP [2,3,4,5,6], and its mortality is about 20%, which is approximately twice that of community-acquired pneumonia (CAP) [7,8,9,10].

Various assessment tools for evaluating the severity of pneumonia are used in clinical practice, and the PORT score and CURB-65 are commonly recommended as severity assessment tools for making decisions regarding the site of care, general management, and the choice of antibiotics in CAP patients in various countries [11,12]. The PORT score, which consists of 20 items, was primarily developed as a useful prediction rule for identifying low-mortality-risk patients who can be safely treated as outpatients and/or other adverse outcomes in patients with CAP [7,11]. However, this scoring system is not substantially utilized in clinical settings, due to its relative complexity and lack of physicians' awareness [13]. CURB-65, which consists of 5 items, has also been developed as a predictive tool for patients with CAP based on the risk factors predicting the mortality of pneumonia proposed by the British Thoracic Society (BTS), and this tool is significantly easier to use than the PORT score [12].

Although various studies regarding severity assessment tools for HCAP patients have been reported thus far [7,8,9,10,14,15,16,17], no specific severity assessment tools for predicting the mortality in patients with HCAP have been developed. Some reports have suggested that the PORT score and CURB-65 may be able to be used to predict the mortality in the management of patients with HCAP [7,9]; however, the lack of usefulness of these scoring systems has also been reported, particularly with respect to predicting mortality in low risk classes [18]. These controversial results may be explained by differences in the subjects evaluated and the social backgrounds in each study. Regardless, given that an increase in the number of HCAP patients is expected as the population ages [19], accurate severity assessment of HCAP patients is an important issue in real-world clinical practice.

In this study, we performed a systematic review and meta-analysis of the published literature assessing these scoring tools and evaluated whether or not the scoring tools used for predicting the mortality for CAP patients were useful for predicting the mortality in patients with HCAP.

This systematic review and meta-analysis was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and the Meta-Analyses (PRISMA) statement [20].

A PubMed search with the following search words was performed: (1) “pneumonia (MeSH Terms)” AND (“healthcare associated pneumonia” OR “nursing home acquired pneumonia”) AND (“severity score” OR “predict” OR “prognosis” OR “mortality score”), (2) “pneumonia (MeSH Terms)” AND (“healthcare-associated pneumonia” OR “nursing home acquired pneumonia”) AND (“pneumonia severity index” OR “PORT score” OR “fine score” OR “A-DROP” OR “I-ROAD”) (accessed on July 16, 2015) [21].

The inclusion criteria for eligible studies were as follows: prospective or retrospective studies, diagnosing patients with HCAP and/or nursing home-associated pneumonia (NHAP) according to ATS/IDSA guidelines in 2005 [1], evaluating severity scores and reporting the outcomes of mortality and appropriate data, written in English as original research articles. Studies on nursing and healthcare-associated pneumonia [22], which have been adjusted as the same concept as HCAP in Japan, were excluded. Only studies with HCAP and/or NHAP patients treated in-hospital were included, and studies with patients diagnosed and treated in nursing homes were excluded.

Two reviewers (S.N. and T.K.) independently assessed the articles. After the PubMed search using the keywords, the nonrelevant studies were first excluded based on the titles and abstracts. Second, the full texts of potentially appropriate titles and abstracts were further reviewed. The quality of each study was assessed as “good,” “moderate,” and “poor” according to Hayden's criteria [23].

The following information was collected from each study: geographic location, design, sample size, mean age of participants, type of severity score, common outcome, and mortality rate.

The outcome in this study was mortality (30- or 28-day).

Paired forest plots were described using the Reviewer manager 5.3 software program (Cochrane Collaboration, Oxford, UK), and the pooled sensitivities, specificities, positive likelihood ratio (PLR) and negative likelihood ratio (NLR) were calculated using the STATA 14 software program (StataCorp LP, College Station, TX, USA). To assess the overall accuracy, the diagnostic odds ratios (DORs) and summary receiver operator characteristic curves (sROCs) were constructed to describe the relationship between the sensitivity and specificity across the included studies using STATA 14 software program. Statistical heterogeneities for the DORs among included studies were assessed using the I² statistic; I² <25, 25-50, >50% indicates low, moderate, and severe heterogeny, respectively.

A total of 354 articles were identified in the initial search through PubMed database. Nineteen articles were potentially eligible after screening the titles and abstracts, 11 of which were excluded after reviewing the full texts, leaving 8 articles eventually selected [7,8,9,10,14,15,16,17]. The flowchart for the selection process is shown in Figure 1, and the characteristics of the included studies are shown in Table 1. The qualities of the included studies were good in 1, moderate in 5, and poor in 2. The size of the included studies ranged from 74 to 767 patients, and the mortality rate ranged from 12.4 to 24.1%. Two of the 8 studies were prospective observational studies [8,15], and the other 6 were retrospective observational studies [7,9,10,14,16,17]. Seven of the 8 studies described 30-day mortality [7,8,9,10,14,15,16], while the other described 28-day mortality [17] as the study outcome. Four studies evaluated patients with HCAP [7,8,9,10], and the rest studied patients with NHAP [14,15,16,17].

The PORT score and CURB-65 were included in 7 [7,8,9,10,15,16,17] and 8 studies [7,8,9,10,14,15,16,17], respectively, and other severity scores were included in less than 3 studies (Table 1). Therefore, the PORT score and CURB-65 were analyzed, and forest plots of the raw data and the estimated sensitivities and specificities from each study for these parameters are shown in Figure 2. There were significant heterogeneities for the pooled estimates.

Seven of the 8 studies were included in the meta-analysis for the PORT score [7,8,9,10,15,16,17]. Using a cutoff value of ≥IV, the pooled sensitivity, pooled specificity, PLR, and NLR for mortality were 0.97 (0.93-0.99), 0.13 (0.08-0.21), 1.12 (1.05-1.19), and 0.21 (0.10-0.44), respectively (Table 2), and the DOR was 5.28 (2.49-11.17). Using a cutoff value of V, the pooled sensitivity, pooled specificity, PLR, and NLR for mortality were 0.63 (0.54-0.72), 0.69 (0.63-0.74), 2.01 (1.78-2.28), and 0.54 (0.44-0.65), respectively, and the DOR was 3.76 (2.88-4.92). Using cutoff values of ≥IV and V, the areas under the curve (AUCs) using the sROC curve were 0.68 (0.64-0.72) and 0.71 (0.67-0.75), respectively (Fig. 3a, c).

Five of the 7 studies evaluating the PORT score were conducted in nonimmunocompromised patients [8,9,15,16,17]. In these 5 studies, using a cutoff value of ≥IV, the DOR and AUC were 6.87 (2.62-18.0) and 0.71 (0.67-0.75), respectively (Table 2; Fig. 3b), and using a cutoff value of V, the DOR and AUC were 4.77 (3.27-6.96) and 0.74 (0.70-0.78), respectively (Table 2; Fig. 3d).

Eight studies were included in the meta-analysis for CURB-65 [7,8,9,10,14,15,16,17]. Using a cutoff value of ≥2, the pooled sensitivity, pooled specificity, PLR, and NLR for mortality were 0.90 (0.80-0.95), 0.28 (0.18-0.42), 1.25 (1.12-1.38), and 0.37 (0.25-0.55), respectively (Table 2), and the DOR was 3.35 (2.26-4.97). Using a cutoff value of value ≥3, the pooled sensitivity, pooled specificity, PLR, and NLR for mortality were 0.61 (0.45-0.75), 0.63 (0.50-0.75), 1.65 (1.42-1.91), and 0.62 (0.50-0.78), respectively, and the DOR was 2.65 (2.05-3.43). Using cutoff values of ≥2 and ≥3, the AUCs were 0.65 (0.61-0.69) and 0.66 (0.62-0.70), respectively (Fig. 3e, g).

Six of the 8 studies with CURB-65 were conducted in nonimmunocompromised patients [8,9,14,15,16,17]. In these 6 studies, using a cutoff value of ≥2, the DOR and AUC were 4.40 (2.22-8.74) and 0.65 (0.61-0.69), respectively (Table 2 and Fig. 3f), and using a cutoff value of ≥3, the DOR and AUC were 3.09 (2.19-4.37) and 0.68 (0.64-0.72), respectively (Table 2; Fig. 3h).

This systematic review and meta-analysis evaluating the severity assessment tools for predicting the mortality in patients with HCAP found that PORT score is a more useful tool than CURB-65 for predicting the mortality in patients with HCAP. In addition, these severity assessment tools, especially PORT score, have strong power in nonimmunocompromised patients with HCAP, according to the DOR and AUC.

HCAP patients are extremely heterogeneous, and many of them are elderly and have variable clinical presentations with comorbid diseases. In addition, due to their advanced age and poor general condition, these patients do not always receive intensive care, unlike CAP patients, despite the severity grade. Whether or not a higher rate of multidrug-resistant (MDR) pathogens relates to the pathogenesis of HCAP [4,6,24,25,26] and the severity assessment score correlates with the isolation rates of MDR pathogens [3,27] remains controversial. Simply applying pneumonia severity score systems for predicting the mortality of HCAP is therefore difficult, as various factors influence the treatment strategies for pneumonia and the mortality in patients with HCAP.

Regarding the tools available for assessing the severity in HCAP patients, some reports have described the PORT score as more useful for predicting the mortality than CURB-65 [7,8,9,16,17], but other reports have stated that the predictive value of CURB-65 for mortality was equal to or better than that of the PORT score [15,17,28]. In this meta-analysis, the DOR and AUC of the PORT score for predicting mortality were higher than those of CURB-65; as such, the PORT score may be a more useful tool for predicting mortality in HCAP patients than CURB-65. Compared to CURB-65, which is not sufficiently accurate for detecting patients with multiple comorbidities [29], the PORT score has several benefits, in that this system can evaluate comorbidities and clinical parameters. However, the PORT score also is limited in that it may overestimate the pneumonia severity in elderly patients with comorbidities and have relative complexity. The AUC is generally considered to have moderate discriminating power when the value exceeds 0.70; Fang et al. [7] and Lee et al. [16] reported AUC values of 0.703 using a cutoff value of ≥IV and 0.73 using a cutoff value of V for the PORT score, respectively, whereas the AUC values were less than 0.70 in 5 other reports [8,9,10,15,17]. In this meta-analysis, the AUC values were 0.68 and 0.71 using cutoff values of ≥IV and V for the PORT score, respectively, and the results may not conclusively demonstrate that the PORT score is better for predicting mortality in HCAP patients than CURB-65.

In the meta-analysis, increased values of DOR and AUC of the PORT score and CURB-65, especially the PORT score, for predicting mortality in nonimmunocompromised patients were observed compared to those in all patients with HCAP [8,9,14,15,16,17]. Fang et al. [7] reported that the patients treated with systemic chemotherapy for cancer and/or hospitalized for ≥2 days had a high mortality, and the differences in the mortality by subcategory among patients with HCAP should be considered when determining the optimum method for managing HCAP patients. Indeed, Carrabba et al. [8] showed that these scoring systems were useful for nonimmunocompromised patients. Whether or not immunocompromised patients are considered the same or a different group in HCAP patients remains controversial [8,30], and an immunocompromised condition is not described as a risk factor in the guidelines of HCAP [1]. Also controversial is whether or not NHAP should be managed like HCAP, although NHAP is included in the HCAP criteria [17]. Man et al. [15] and Porfyridis et al. [28] reported that these scoring tools were useful for the identification of patients with NHAP. Conversely, however, Ugajin et al. [17] found that these scoring systems had less prognostic value for NHAP than for CAP. Only 3 observational studies of NHAP were included in the present meta-analysis [15,16,17]; therefore, we were unable to evaluate the usefulness of the severity assessment tools in patients with NHAP specifically. In addition, regarding the items used in the scoring system, Shindo et al. reported that 5 items (serum albumin ≤30 mg/dL, nonambulatory status, pH ≤7.35, respiratory rate ≥30/min, blood urea nitrogen ≥7.14 mmol/L) were independent risk factors for 30-day mortality in pneumonia patients who received appropriate initial antibiotics [31]. Thus, creating as homogeneous groups as possible among HCAP patients and evaluating the severity assessment scores as well as investigating new combinations of items in each appropriately categorized group might be useful when implementing severity assessment tools for HCAP.

A number of systematic reviews and meta-analyses evaluating the severity assessment tools for predicting mortality in patients with CAP have been published [21,32,33,34]. Chalmers et al. [21] reported that the DOR and AUC were 9.6 and 0.82 for the PORT score (≥IV), respectively, and 8.3 and 0.81 for CURB-65 (≥2), respectively. Loke et al. [32] reported similar results in a meta-analysis of patients with CAP. No meta-analyses have evaluated the utility of severity assessment tools for predicting the mortality in HCAP patients, although some reports have shown that the power of these scoring systems for predicting the mortality was lower in patients with HCAP than that in those with CAP [8,9,18]. In our meta-analysis, the DOR and AUC values for HCAP patients were obviously low, and our meta-analysis demonstrated that the powers of the PORT score and CURB-65 for predicting mortality in HCAP patients were low compared with those in CAP patients. The discerning power in HCAP patients tends to be lower than in CAP patients because severity scores are typically higher in patients with HCAP due to their older age and increased number of comorbidities, although Fabio and Carrabba [27] reported that age might not be a relevant factor for assessing pneumonia severity in patients with HCAP.

Other severity assessment tools for predicting mortality besides the PORT score and CURB-65 have been reported. The SOAR (systolic blood pressure, oxygenation, age, and respiratory rate) criteria were included in 3 reports in this study [7,14,16] and were proposed as an alternative tool for evaluating severe CAP patients with advanced age [35]. In this severity scoring system, consciousness is not included in the assessment items, because “confusion” may be overemphasized in older-age patients who often have dementia or Alzheimer disease or who are receiving psychotropic medication, especially in the emergency room [14]. Whereas “confusion” was included as one of 3 major criteria (hypotension, tachypnea, and altered mentation) in the quick sequential organ failure assessment (qSOAR) score that have recently been published in predicting the mortality in patients with sepsis [36]. Thus, an evaluation of altered consciousness in elderly patients with pneumonia should be carefully considered as one of important issues for clinicians in estimating the prognosis of pneumonia patients.

In addition, the following severity scoring systems were included in only 1 or 2 studies each: IDSA/ATS [7], SCAP [7,8], M-ATS [7,15], R-ATS [15], SMART-CO [7], SMART-COP [7], España rule [15], NHAP model score [16], A-DROP [10,17], and I-ROAD [10]. Fang et al. [7] reported that SCAP score had better statistical power (AUC 0.71) in predicting mortality in the criteria of IDSA/ATS, SCAP, M-ATS, SMART-CO, SMART-COP, CURB-65, and PORT score, and blood pH was significantly associated with poor outcomes in HCAP patients. Man et al. [15] reported that M-ATS and R-ATS are useful for excluding severe pneumonia but these tools might underestimate the severity of pneumonia in elderly patients and/or patients with poor functional status because these criteria include “mechanical ventilation” as one of major criteria. As such, further data accumulation is needed to clarify the usefulness of these scoring systems for predicting mortality in HCAP patients, although various scoring tools have been evaluated in predicting the mortality in patients with HCAP.

There are several limitations associated with the present systematic review and meta-analysis that should be considered. First, this meta-analysis had large heterogeneity among the included studies, although large heterogeneities are commonly seen in systematic reviews of diagnostic test accuracy studies [37]. Second, we could not assess the differences in the severity assessment tools for each country, study design, or category of pneumonia due to the limited data and small sample size. Third, the definition of “immunocompromised condition” was not exactly the same among the reports evaluated in this study.

In conclusion, this systematic review indicates that the PORT score and CURB-65 have less power for predicting mortality in HCAP patients than for predicting mortality in CAP patients, and the PORT score is a superior tool for predicting mortality compared to CURB-65 in HCAP patients, although it is not easy to use in real-world clinical practice. In addition, this meta-analysis suggests that these scoring systems, especially the PORT score, may be useful severity assessment tools for predicting the mortality in nonimmunocompromised patients with HCAP. In the future, the proper adjustment and selection of patient grouping and assessment tools in each country are also important, given the differences in patient characteristics, bacterial etiology, healthcare, and health insurance systems by country.

This systematic review and a meta-analysis was performed in the process of composing the Japanese Respiratory Society Practical Guideline for Pneumonia. The authors appreciate the contributions of all of the members of the Guidelines Committee.

All authors declare no conflicts of interest.