Poor discriminatory performance of the Pediatric Risk of Mortality (PRISM) score in a South African intensive care unit

Objective: The use of the Pediatric Risk of Mortality (PRISM) score or other scoring systems in the intensive care unit (ICU) is of great importance for evaluating the efficacy and efficiency of a particular ICU. However, the PRISM score was developed and validated in the United States and subsequently validated in Europe, but has not been evaluated in a less affluent society. In general, scoring systems should be used only in populations similar to the reference population in which the prediction model was developed. We set out to determine the applicability of the PRISM score at Baragwanath Hospital, South Africa. Design: Prospective, descriptive study. Setting: Twenty-four- bed multidisciplinary ICU. Patients: We analyzed 1,528 consecutive pediatric admissions from January 1989 to June 1994. Interventions: None. Measurements and Main Results: PRISM scores, Therapeutic Intervention Scoring System scores, demographic, and clinical data collected prospectively were entered and stored by means of a commercial software package at the time of admission of each patient. The prediction of actual mortality by PRISM scoring was evaluated by the Hosmer and Lameshow goodness-of-fit test (chl2[8 degrees of freedom]). Receiver operating characteristic curves were constructed and compared with those curves from pediatric ICU populations in the United States and Europe. Individual receiver operating characteristic curves were constructed for surgical and nonsurgical patients, age categories, and diagnostic categories. Compared with European and American ICU populations, our patients were younger, were mostly nonsurgical emergency admissions, stayed longer in the ICU, and were more severely ill with a higher admission PRISM score and overall mortality rate. Respiratory and septic diagnoses predominated, with very few surgical cases admitted. The Hosmer and Lemeshow goodness-of-fit test showed a significant failure of the PRISM scoring system to accurately predict mortality over a wide range of expected mortality rates (chl2[8 degrees of freedom] = 465, p = 0). Similarly, receiver operating characteristic analysis indicated a poor predictive power (A = 0.73 ± 0.01 [SEM]), with an area under the curve significantly less than that for the PRISM reference population (p = 0). PRISM showed equally poor discriminatory function at all age groups and diagnostic categories. Conclusions: The PRISM score needs to be recalibrated or recalculated for our patient population in view of the high discrepancy and poor discriminatory function shown. Part of the inaccuracy may derive from different demographic characteristics of our ICU population and a different pattern of diseases. It appears that PRISM is not population independent.