title: Reliability and validity of a scoring instrument for clinical performance during Pediatric Advanced Life Support simulation scenarios.
abstract: OBJECTIVE
  To assess the reliability and validity of scoring instruments designed to measure clinical performance during simulated resuscitations requiring the use of Pediatric Advanced Life Support (PALS) algorithms.
  
  
  METHODS
  Pediatric residents were invited to participate in an educational trial involving simulated resuscitations that employ PALS algorithms. Each subject participated in a session comprised of four scenarios (asystole, dysrhythmia, respiratory arrest, shock). Video-recorded sessions were independently reviewed and scored by four raters using instruments designed to measure performance in terms of timing, sequence, and quality. Validity was assessed by two-factor analysis of variance with postgraduate year (PGY-1 versus PGY-2) as an independent variable. Reliability was assessed by calculation of overall interrater reliability (IRR) as well as a generalizability study to estimate variance components of individual measurement facets (scenarios, raters) and associated interactions.
  
  
  RESULTS
  20 subjects were scored by four raters. Based on a two-factor ANOVA, PGY-2s outperformed PGY-1s (p<0.05); significant differences in difficulty existed between the four scenarios, with dysrhythmia scores being the lowest. Overall IRR was high (0.81) and most variance could be attributed to subject (17%), scenario (13%), and the interaction between subject and scenario (52%); variance attributable to rater was minimal (1.4%).
  
  
  CONCLUSIONS
  The instruments assessed in this study measure clinical performance during PALS scenarios in a reliable and valid manner. Measurement error could be minimized further through the use of additional scenarios but additional raters, for a given scenario, would not improve reliability. Further studies should assess validity of measurement with respect to actual clinical performance during resuscitations.
authors.affiliation: Division of Pediatric Critical Care Medicine, Children's Hospital of Philadelphia, PA 19104, United States. donoghue@email.chop.edu
authors.forename: Aaron
authors.initials: A
authors.lastname: Donoghue
authors.forename: Akira
authors.initials: A
authors.lastname: Nishisaki
authors.forename: Robert
authors.initials: R
authors.lastname: Sutton
authors.forename: Roberta
authors.initials: R
authors.lastname: Hales
authors.forename: John
authors.initials: J
authors.lastname: Boulet
mesh_terms: D000465:Algorithms; D000704:Analysis of Variance; D001145:Arrhythmias, Cardiac / Q000628:therapy; D002648:Child; D002983:Clinical Competence / Q000592:standards*; D005163:Factor Analysis, Statistical; D006323:Heart Arrest / Q000628:therapy; D006801:Humans; D007396:Internship and Residency; D008021:Life Support Systems; D016544:Patient Simulation; D010372:Pediatrics / Q000193:education* / Q000379:methods; D011897:Random Allocation; D015203:Reproducibility of Results; D012131:Respiratory Insufficiency / Q000628:therapy; D012151:Resuscitation / Q000592:standards*; D012769:Shock / Q000628:therapy; D014743:Videotape Recording