title: Checklist content on a standardized patient assessment: an ex post facto review.
abstract: While checklists are often used to score standardized patient based clinical assessments, little research has focused on issues related to their development or the level of agreement with respect to the importance of specific items. Five physicians independently reviewed checklists from 11 simulation scenarios that were part of the former Educational Commission for Foreign Medical Graduate's Clinical Skills Assessment and classified the clinical appropriateness of each of the checklist items. Approximately 78% of the original checklist items were judged to be needed, or indicated, given the presenting complaint and the purpose of the assessment. Rater agreement was relatively poor with pairwise associations (Kappa coefficient) ranging from 0.09 to 0.29. However, when only consensus indicated items were included, there was little change in examinee scores, including their reliability over encounters. Although most checklist items in this sample were judged to be appropriate, some could potentially be eliminated, thereby minimizing the scoring burden placed on the standardized patients. Periodic review of checklist items, concentrating on their clinical importance, is warranted.
authors.affiliation: Educational Commission for Foreign Medical Graduates (ECFMG), Philadelphia, PA 19104-2685, USA. jboulet@ecfmg.org
authors.forename: John R
authors.initials: JR
authors.lastname: Boulet
authors.forename: Marta
authors.initials: M
authors.lastname: van Zanten
authors.forename: André
authors.initials: A
authors.lastname: de Champlain
authors.forename: Richard E
authors.initials: RE
authors.lastname: Hawkins
authors.forename: Steven J
authors.initials: SJ
authors.lastname: Peitzman
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D004521:Educational Measurement / Q000379:methods* / Q000592:standards; D006801:Humans; D007396:Internship and Residency / Q000379:methods / Q000592:standards; D008487:Medical History Taking / Q000592:standards; D016544:Patient Simulation; D010808:Physical Examination / Q000592:standards; D010820:Physicians / Q000592:standards; D015203:Reproducibility of Results; D013337:Students, Medical