Reliability and validity of a scoring instrument for clinical performance during Pediatric Advanced Life Support simulation scenarios.
OBJECTIVE
To assess the reliability and validity of scoring instruments designed to measure clinical performance during simulated resuscitations requiring the use of Pediatric Advanced Life Support (PALS) algorithms.


METHODS
Pediatric residents were invited to participate in an educational trial involving simulated resuscitations that employ PALS algorithms. Each subject participated in a session comprised of four scenarios (asystole, dysrhythmia, respiratory arrest, shock). Video-recorded sessions were independently reviewed and scored by four raters using instruments designed to measure performance in terms of timing, sequence, and quality. Validity was assessed by two-factor analysis of variance with postgraduate year (PGY-1 versus PGY-2) as an independent variable. Reliability was assessed by calculation of overall interrater reliability (IRR) as well as a generalizability study to estimate variance components of individual measurement facets (scenarios, raters) and associated interactions.


RESULTS
20 subjects were scored by four raters. Based on a two-factor ANOVA, PGY-2s outperformed PGY-1s (p<0.05); significant differences in difficulty existed between the four scenarios, with dysrhythmia scores being the lowest. Overall IRR was high (0.81) and most variance could be attributed to subject (17%), scenario (13%), and the interaction between subject and scenario (52%); variance attributable to rater was minimal (1.4%).


CONCLUSIONS
The instruments assessed in this study measure clinical performance during PALS scenarios in a reliable and valid manner. Measurement error could be minimized further through the use of additional scenarios but additional raters, for a given scenario, would not improve reliability. Further studies should assess validity of measurement with respect to actual clinical performance during resuscitations.
Donoghue
Aaron
A
Division of Pediatric Critical Care Medicine, Children's Hospital of Philadelphia, PA 19104, United States. donoghue@email.chop.edu
Nishisaki
Akira
A
Sutton
Robert
R
Hales
Roberta
R
Boulet
John
J
D000465:Algorithms; D000704:Analysis of Variance; D001145:Arrhythmias, Cardiac / Q000628:therapy; D002648:Child; D002983:Clinical Competence / Q000592:standards*; D005163:Factor Analysis, Statistical; D006323:Heart Arrest / Q000628:therapy; D006801:Humans; D007396:Internship and Residency; D008021:Life Support Systems; D016544:Patient Simulation; D010372:Pediatrics / Q000193:education* / Q000379:methods; D011897:Random Allocation; D015203:Reproducibility of Results; D012131:Respiratory Insufficiency / Q000628:therapy; D012151:Resuscitation / Q000592:standards*; D012769:Shock / Q000628:therapy; D014743:Videotape Recording