Effect of training and strict guidelines on the reliability of risk adjustment systems in paediatric intensive care.
OBJECTIVE
Many risk-adjustment systems have significant interobserver variability in everyday clinical practice. This can be partly corrected by strict guidelines and training. These issues have not been well studied in the paediatric setting. We assessed the reliability of two widely used paediatric scoring systems, the Paediatric Risk of Mortality (PRISM) and Paediatric Index of Mortality (PIM), before and after a special training program.


METHODS
Prospective observational multi-centred cohort study.


METHODS
Twenty-seven physicians from five paediatric intensive care units (PICUs) assessed severity of illness in 20 selected patients using PRISM and PIM scores before and after a special training program. Physicians were divided according to level of PICU experience: intensivists (>3 years experience, n=12), fellows (6-30 months experience, n=6) and residents (<6 months experience, n=9). Intraclass correlation was used to compare scoring reliability before and after training.


RESULTS
Wide variability in PRISM and PIM scoring was observed before training (intraclass correlation for PRISM scores 0.24-0.73, intraclass correlation for PIM scores 0.16-0.33). Training and implementation of guidelines led to significant increases in interobserver agreement (intraclass correlation 0.74-0.86 for PRISM and 0.88-0.95 for PIM scores), although some variability remained.


CONCLUSIONS
Our results show that the reliability of PRISM and PIM risk adjustment systems in daily clinical practice is much lower than expected. Training and guidelines can significantly increase interobserver agreement. These factors should be taken into account when using these systems for benchmarking, or to compare quality of care between different PICUs.
Department of Paediatrics, VU University Medical Center, P.O. Box 7057, 1007 MB, Amsterdam, The Netherlands.
Jolanda G
JG
van Keulen
Reinoud J B J
RJ
Gemke
Kees H
KH
Polderman
D004522:Educational Status; D017408:Guidelines as Topic; D017052:Hospital Mortality; D006801:Humans; D015278:Intensive Care Units, Pediatric / Q000145:classification / Q000706:statistics & numerical data*; D008505:Medical Staff, Hospital; D015337:Multicenter Studies as Topic; D015588:Observer Variation; D015203:Reproducibility of Results; D020379:Risk Adjustment / Q000379:methods*; D012720:Severity of Illness Index