Development and evaluation of high-fidelity simulation case scenarios for pediatric resident education.
OBJECTIVE
Pediatric residency programs need objective methods of trainee assessment. Patient simulation can contribute to objective evaluation of acute care event management skills. We describe the development and validation of 4 simulation case scenarios for pediatric resident evaluation.


METHODS
We created 4 pediatric simulation cases: apnea, asthma, supraventricular tachycardia, and sepsis. Each case contains a scenario and an unweighted checklist. Case and checklist development began by reaching expert consensus about case content followed by 92 pilot simulation sessions used for content revision and rater training. After development, 54 first-and second-year pediatric residents participated in 108 simulation test cases to assess the validity of data from these tools for our population. We report outcomes for interrater reliability, discriminant validity, and the impact of potential confounding factors on validity estimates.


RESULTS
Interrater reliability (kappa) ranged from 0.75 to 0.87. There were statistically and educationally significant differences in summary scores between first-and second-year residents for 3 of the 4 cases. Neither previous simulation exposure nor the order in which the cases were performed were found to be significant factors by multivariate analysis.


CONCLUSIONS
Simulation can be used to reliably measure and discriminate resident competencies in acute care management. Rigorous measurement development work is difficult and time-consuming. Done correctly, measurement development yields tangible and lasting benefits for trainees, faculty, and residency programs. Development studies that use systematic procedures and large trainee samples at multiple sites are the best approach to creating measurement tools that yield valid data.
Adler
Mark D
MD
Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, IL, USA. m-adler@northwestern.edu
Trainor
Jennifer L
JL
Siddall
Viva Jo
VJ
McGaghie
William C
WC
D015986:Confounding Factors, Epidemiologic; D004521:Educational Measurement; D006801:Humans; D007396:Internship and Residency; D008348:Manikins; D015588:Observer Variation; D016544:Patient Simulation; D010372:Pediatrics; D016730:Program Development; D015397:Program Evaluation; D015203:Reproducibility of Results