title: Validity and Feasibility Evidence of Objective Structured Clinical Examination to Assess Competencies of Pediatric Critical Care Trainees.
abstract: OBJECTIVE
  The purpose of this study was to provide validity and feasibility evidence for the use of an objective structured clinical examination in the assessment of pediatric critical care medicine trainees.
  
  
  METHODS
  This was a validation study. Validity evidence was based on Messick's framework.
  
  
  METHODS
  A tertiary, university-affiliated academic center.
  
  
  METHODS
  Seventeen pediatric critical care medicine fellows were recruited in 2012 and 2013 academic year.
  
  
  METHODS
  None. All subjects completed an objective structured clinical examination assessment.
  
  
  RESULTS
  Seventeen trainees were assessed. Simulation scenarios were developed for content validity by pediatric critical care medicine and education experts using CanMEDS competencies. Scenarios were piloted before the study. Each scenario was evaluated by two interprofessional raters. Inter-rater agreement, measured using intraclass correlations, was 0.91 (SE = 0.09) across stations. Generalizability theory was used to evaluate internal structure and reliability. Reliability was moderate (G-coefficient = 0.67, Φ-coefficient = 0.52). The greatest source of variability was from participant by station variance (40.6%). Pearson correlation coefficients were used to evaluate the relationship of objective structured clinical examination with each traditional assessment instruments: multisource feedback, in-training evaluation report, short-answer questions, and Multidisciplinary Critical Care Knowledge Assessment Program. Performance on the objective structured clinical examination correlated with performance on the Multidisciplinary Critical Care Knowledge Assessment Program (r = 0.52; p = 0.032) and multisource feedback (r = 0.59; p = 0.017), but not with the overall performance on the in-training evaluation report (r = 0.37; p = 0.143) or short-answer questions (r = 0.08; p = 0.767). Consequences were not assessed.
  
  
  CONCLUSIONS
  Validity and feasibility evidence in this study indicate that the use of the objective structured clinical examination scores can be a valid way to assess CanMEDS competencies required for independent practice in pediatric critical care medicine.
authors: 1Department of Critical Care Medicine, Hospital for Sick Children, Toronto, ON, Canada. 2Department of Paediatrics, Faculty of Medicine, University of Toronto, Toronto, ON, Canada. 3Department of Medical Education, University of Illinois at Chicago, Chicago, IL.
authors: Briseida
authors: B
authors: Mema
authors: Yoon Soo
authors: YS
authors: Park
authors: Afrothite
authors: A
authors: Kotsakis
mesh_terms: D000046:Academic Medical Centers; D002983:Clinical Competence / Q000592:standards*; D003422:Critical Care / Q000592:standards*; D004521:Educational Measurement / Q000379:methods* / Q000592:standards*; D000068156:Formative Feedback; D006801:Humans; D007396:Internship and Residency / Q000592:standards*; D010372:Pediatrics / Q000193:education*; D015203:Reproducibility of Results