Performance and Analysis of Pediatric Index of Mortality 3 Score in a Pediatric ICU in Latifa Hospital, Dubai, UAE

Mortality rate is one of the key quality indicators of an intensive care unit (ICU). However, the severity of -patients’ illnesses, comorbidities, and demographics all strongly affect the mortality rate [1]. Thus, a comparison of unadjusted mortality rate among ICUs without consideration of the severity of illness or other case-mix variables might provide an incorrect estimate of the quality of ICU care. Mortality data for ICU patients are usually adjusted using mortality prediction models. In general, for pediatric ICUs (PICUs), the Pediatric Index of Mortality (PIM) 3 and Pediatric Risk of Mortality (PRISM) III are the -commonly used mortality prediction models [2-4].

The PIM 3 score is one of several severity scoring systems used for predicting the outcome of pediatric patients admitted to the PICU based on data collected within first hour of admission. An advantage of the PIM 3 is that the formula and coefficients of predicted mortality are presented and are freely available [4], compared to the mortality prediction formula of PRISM III which is commercially patented and documents only incomplete scores in the report. The PIM 3 consists of 10 accessible variables that predict mortality before patients receive advanced therapy, and it is simple and inexpensive to perform.

The PIM 3 is an updated version of the PIM 2 for predicting mortality in the PICU. It provides an international standard based on a large contemporary dataset, for the comparison of risk-adjusted mortality among children admitted to intensive care [4].

PICU patients often have unclear prognoses, and mortality rates may be dependent on the PICU staff and procedures. A scoring system for illness severity is therefore useful for objectively predicting the outcomes and prognoses of PICU patients [2-5].

The PIM 3 is an updated model that was built using a larger dataset with more ICUs and greater representation across countries. This may improve the generalizability of the model to populations outside this study group; however, population differences in admission thresholds, case-mix, resourcing, and the process of care should be considered when assessing model performance in different populations [5, 6].

The objective of this study was to evaluate the usefulness of the PIM 3 for predicting mortality and to validate PIM 3 in children admitted to a single PICU in the tertiary care Latifa Women and Children’s Hospital. Additionally, we aimed to determine other factors strongly correlated with the predicted mortality rate.

This was a retrospective cohort study of children admitted to the PICU at Latifa Women and Children’s Hospital, Dubai, between January 2016 and October 2018. The probability of mortality for each admitted child was calculated within the first hour of admission and the mortality result was recorded at PICU discharge.

All patients between 1 month old and 15 years of age admitted to the PICU at Latifa Women and Children’s Hospital, Dubai, during the study period were included in the study. Of 590 patients, 7 were excluded from the study because of incomplete data, so that the final sample consisted of 583 patients.

The children included in the study were younger than 15 years old at the time of admission. During the study period, there were 583 admissions to the PICU in the Latifa Hospital that met the inclusion criteria. Readmissions were treated as new admissions, and the probability of death was estimated based on the characteristics at the time of the new admission. Data were collected from the computerized ICU registry database. The admission route was via the general ward, emergency room, or operating room.

Demographic data were collected from all study participants, including age, sex, diagnosis. PIM 3 variables such as systolic blood pressure and pupillary reaction to bright light were measured. Other variables assessed were partial oxygen tension (PaO₂) and FiO₂ (at the same time if oxygen was given by endotracheal tube); noninvasive ventilation; base excess in arterial blood gas analysis; the type of mechanical ventilation at any time during the first hour of PICU admission; elective admission to PICU;, recovery from surgery or the procedure that was the main reason for ICU admission; and a low-risk, high-risk, or very high-risk diagnosis. Definitions of these variables and the scoring method were according to the PIM 3 developers’ guidelines [5].

Scores were calculated using the PIM 3 calculator application. The variables used to evaluate the PIM 3 score, using a multiple logistic model, are listed in Table 8. Data were entered into Microsoft Excel 2013 and analyzed using SPSS v24.0. We analyzed the association between PIM 3 score and mortality. The performance of PIM3 score was assessed by calibration and discrimination. Calibration evaluated PIM 3 at different risks of mortality and was assessed with the χ² goodness-of-fit test and the standardized mortality ratio (SMR, calculated as the ratio of observed-to-expected death rates). Discrimination evaluated how well PIM 3 distinguished between patients who survived and died and was assessed using the area under the curve from a receiver-operating characteristics (ROC) plot.

Over the study period, 583 patients were included, 46 of whom (7.9%) died. The mean age of the patients was 37 months (median 12.5 [range 1–180] months). The majority of the patients were male (61.4%). The most common underlying cause for PICU admission was respiratory (29.9%), followed by postoperative (21.1%), neurological (19%), metabolic (10.5%), cardiovascular (5.7%), sepsis (3.6%), and others (9.8%). The demographic features and clinical course of the patients related to the outcome are provided in Table 1.

The number of patients in the years 2016, 2017, and 2018 were 219 (37.6), 220 (37.8%), and 143 (24.6%), respectively. The range of the probability score was 0.2–95.3% in survivors and 0.2–99.2% in nonsurvivors, with a mean and median score of 12.8 and 14.3% among survivors, respectively, and 39.8 and 30.7% among nonsurvivors, respectively (Table 2). Most subjects were in the 5–15% score interval. A higher PIM 3 score indicated a higher probability of mortality. Table 3 shows the PIM 3 score intervals and subjects’ outcomes.

Table 4 shows the SMR calibration of the PIM 3 model based on the PIM 3 score intervals of <5%, 5–14.99%, and ≥15%. SMR was 0.33 in the p < 14.3% group and 0.72 in the p > 14.3%, respectively. The expected mortality rate based on PIM 3 scores was 9.2% in the p < 14.3% group and 37.5% in the p >14.3% group, respectively. The overall accumulated expected mortality was 15% as compared to the observed mortality of 7.9% and SMR of 0.53. AUC analysis showed a good discriminatory ability of the PIM 3 score in the interval group to distinguish between survivors and nonsurvivors (AUC >70%). The AUC of the PIM 3 score was 0.44 in the p < 14.3% group and overall AUC was 0.78 (95% confidence interval [CI] 0.69–0.87) (Table 5).

Table 6 shows that SMRs in the overall demographic and the clinical course group were <1, except for predicting mortality in sepsis where it was 2.1.Table 7 and Figure 1 show the AUC of the PIM 3 score ROC analysis (AUC >0.7 is considered to be acceptable for predicting death and survival).Table 8 shows the PIM 3 variables and formula for the calculation of the probability of death.

The inclusion of patient data from children admitted to general ICUs may help improve the performance of our model among children admitted to nonpediatric ICUs. When monitoring PICU outcomes, it is desirable to monitor performance using both the international standard (PIM 3) as well as a locally calibrated version of the model, where the overall SMR for the local population = 1. A locally calibrated model will allow ICUs to compare their performance with local standards of care. If performed regularly, e.g., at intervals of 1 or 2 years, local calibration also overcomes the issue of calibration drift due to improvements in quality of care and changes in case-mix. To standardize the nomenclature, we recommend that the region and the final year of data be added to PIM 3; PIM3-ANZ [7] denotes a model calibrated using data from Australia and New Zealand, corresponding to the study period reported here. Assessment against an international standard is also important as there may be factors in the health care system that affect the outcomes of children in the region more generally. These factors will not be appreciated if the only assessment is a local comparison. The use of a larger and more geographically diverse patient population in the future may help to improve the generalizability of the model to other settings.

Our study investigated 583 PICU patients at Latifa Women and Children’s Hospital, Dubai, UAE, between January 2016 and October 2018 (3 years), in order to evaluate the performance of PIM 3, in terms of its calibration and discrimination ability, compared with what has been observed in developed countries. The study showed that the PIM 3 had an acceptable discrimination ability (AUC 0.78 [95% CI 0.69–0.87]). This finding was similar to an Italian validation study which reported that PIM 3 more accurately predicted mortality risk than PIM 2 [8]. Therefore, the PIM 3 score may be a reasonable choice for pediatric ICUs in Dubai, UAE, despite its additional complexity.

The prevalence of mortality in our PICU in the 3-year period was 15%, much lower than that of Honna et al. [9] at 45.7%. The prevalence of PICU mortality was also lower than that reported from India in 2011 (46.2%) [10] and Pakistan in 2006 (28.7%) [11].

We observed a high SMR of 2.1 with sepsis, followed by 0.96 with cardiovascular, 0.61 with respiratory, and 0.54 with metabolic causes. However, an SMR >1 suggests that the performance of the ICU is poor. The high unadjusted mortality may be explained by the severity of illness of patients in the ICU or by the poor performance of the ICU team.

All SMR values were <1, suggesting a good ICU performance, except for the 1–5% probability group which had an SMR value of 2.67, indicating that the actual mortality was 2.67 times higher than expected, and the sepsis group.

To clarify the actual cause, the SMR, which is the observed mortality rate divided by the cumulated predicted mortality, can be calculated. The SMR is the most commonly used parameter of ICU quality in western Europe and is mandated by some countries [12]. It can also be used to compare mortality data, follow changes over time, and evaluate the effect of interventions or events [13-16]. To calculate the SMR, the predicted mortality must first be determined. In this study, the observed mortality was lower than the predicted mortality. When the illness is severe, severity can affect the physiological parameters recorded at the time of ICU admission, and eventually the SMR.

Variation in the predicted mortality according to the scoring system can also influence the SMR [17]. In Latin America, PIM 2 was reported to be inadequate because of poor calibration [18]. Some countries have tried to develop a calibrated mortality prediction model because of regional differences among ICUs [5, 19]. Despite its defects, the SMR based on predicted mortality rates can provide valuable information. For example, serial measurements of ICU SMRs can be used to monitor internal quality improvements in hospitals. At the national level, the SMR of PICUs can be measured every year to evaluate the improvement of the quality of PICUs in Dubai, UAE.

Multiple factors can influence SMR, such as a poor referral system, delayed initial therapy, or complications which can change the outcome, such as hospital-acquired infection, malnutrition caused by hospitalization, or ventilator-associated pneumonia. The overall SMR in this study was 0.53, which meant that the PIM 3 model overpredicted deaths in our facility. Other studies in developing countries like India [10], Pakistan [11], and Egypt [20] have reported SMRs from PIM 2 scores of 3.3, 1.57, and 1.92, respectively. On the other hand, a Japanese study reported a PIM 2 SMR <1 (0.77), which means that the score overpredicted mortality [19].

The discrimination was evaluated by AUC. Discrimination is considered to be very good when the ROC curve is >0.9, good when it is 0.80–0.90, and fair when it is 0.70–0.80 [21]. The AUC was calculated to be 0.781 (95% CI 0.69–0.87), lower than the AUC in the original places where the PIM 3 studies were undertaken. Many studies using PIM 2 in developing countries have shown good discrimination with AUC values, i.e., 0.81 (95% CI 0.75–0.87) in Pakistan [11], 0.795 (95% CI 0.715–0.875) in Iran [22], and 0.841 (95% CI 0.78–0.90) in Africa [21]. However, AUCs from developed countries were found to be as follows: 0.91 in Australia, 0.90–0.93 in New Zealand, 0.85 in the UK, and 0.84–0.86 in Scotland [23].

All the p values for the χ² goodness-of-fit test showed that the observed mortality numbers were significantly less than the expected mortality numbers, which indicates that our ICU unit did very well.

Our study has several limitations. First, it was a single-center study conducted at a tertiary hospital. The findings may therefore not be generalizable to the entire pediatric population of the UAE. Dubai is a cosmopolitan state comprising about 200 nationalities. Multicenter studies that include primary and secondary hospitals are needed. The second limitation was the retrospective data collection, although every effort was made to validate the data thoroughly. Presently, the process of incorporating the PIM 3 score into our electronic data system is underway to obtain prospective data.

We found that the PIM 3 score had good calibration and was a consistent guide to the performance of our PICU in a pediatric population <15 years. Great care has to be taken to ensure that the data needed to calculate PIM are accurate. The model incorporates the quality of retrieval services in its assessment as well as adjusting for the presence of important premorbid conditions. We recommend that the PIM 3 be used routinely as a mortality prediction model for pediatric intensive care.

We thank the PICU nursing staff, especially Ms. Elizabeth, for their assistance in data collection.

All data were collected and analyzed retrospectively in this study. The study was approved by the Dubai Scientific Research Ethics Committee (DSREC) of the Dubai Health Authority (DHA) vide reference No. DSREC-12/2018_04.

The authors have no potential conflicts of interest to disclose.

There was no funding.

D.M.: Design, data acquisition, manuscript drafting, critical analysis, and final approval.

N.N.: Conception, analysis, manuscript drafting, critical analysis, and final approval.

M.E.H.: Analysis, manuscript drafting, critical analysis, and final approval.

M.Z.: Data analysis, manuscript drafting, critical analysis, and final approval.