title: Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence.
abstract: In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.
authors.affiliation: Department of Medicine, University of British Columbia, Suite 5907, Burrard Bldg, St. Paul's Hospital, 1081 Burrard St, Vancouver, BC, V6Z 1Y6, Canada. rhatala@mac.com.
authors.forename: Rose
authors.initials: R
authors.lastname: Hatala
authors.affiliation: Mayo Clinic Online Learning and Mayo Clinic Multidisciplinary Simulation Center, Mayo Clinic College of Medicine, Rochester, MN, USA.|Division of General Internal Medicine, Mayo Clinic, Rochester, MN, USA.
authors.forename: David A
authors.initials: DA
authors.lastname: Cook
authors.affiliation: Wilson Centre, University Health Network, Toronto, ON, Canada.
authors.forename: Ryan
authors.initials: R
authors.lastname: Brydges
authors.affiliation: Medical Education Programs, American Medical Association, Chicago, IL, USA.
authors.forename: Richard
authors.initials: R
authors.lastname: Hawkins
mesh_terms: D002983:Clinical Competence; D004521:Educational Measurement / Q000592:standards*; D006282:Health Personnel / Q000193:education*; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results
keywords: Assessment; OSATS; Systematic review; Validity argument