title: Validity evidence for two objective structured clinical examination stations to evaluate core skills of the shoulder and knee assessment.
abstract: BACKGROUND
  We developed two objective structured clinical examinations (OSCEs) to educate and evaluate trainees in the evaluation and management of shoulder and knee pain. Our objective was to examine the evidence for validity of these OSCEs.
  
  
  METHODS
  A multidisciplinary team of content experts developed checklists of exam maneuvers and criteria to guide rater observations. Content was proposed by faculty, supplemented by literature review, and finalized using a Delphi process. One faculty simulated the patient, another rated examinee performance. Two faculty independently rated a portion of cases. Percent agreement was calculated and Cohen's kappa corrected for chance agreement on binary outcomes. Examinees' self-assessment was explored by written surveys. Responses were stratified into 3 categories and compared with similarly stratified OSCE scores using Pearson's coefficient.
  
  
  RESULTS
  A multi-disciplinary cohort of 69 examinees participated. Examinees correctly identified rotator cuff and meniscal disease 88% and 89% of the time, respectively. Inter-rater agreement was moderate for the knee (87%; k = 0.61) and near perfect for the shoulder (97%; k = 0.88). No correlation between stratified self-assessment and OSCE scores were found for either shoulder (0.02) or knee (-0.07).
  
  
  CONCLUSIONS
  Validity evidence supports the continuing use of these OSCEs in educational programs addressing the evaluation and management of shoulder and knee pain. Evidence for validity includes the systematic development of content, rigorous control of the response process, and demonstration of acceptable interrater agreement. Lack of correlation with self-assessment suggests that these OSCEs measure a construct different from learners' self-confidence.
authors.affiliation: Salt Lake City Veterans Affairs Medical Center (SLC VAMC), 500 Foothill Drive, Salt Lake City, UT, 84148, USA. Michael.Battistone@va.gov.|Department of Medicine, Division of Rheumatology, University of Utah, Salt Lake City, USA. Michael.Battistone@va.gov.
authors.forename: Michael J
authors.initials: MJ
authors.lastname: Battistone
authors.affiliation: Salt Lake City Veterans Affairs Medical Center (SLC VAMC), 500 Foothill Drive, Salt Lake City, UT, 84148, USA.|Department of Family and Preventive Medicine, University of Utah, Salt Lake City, USA.
authors.forename: Andrea M
authors.initials: AM
authors.lastname: Barker
authors.affiliation: Salt Lake City Veterans Affairs Medical Center (SLC VAMC), 500 Foothill Drive, Salt Lake City, UT, 84148, USA.|Department of Orthopaedics, University of Utah, Salt Lake City, USA.
authors.forename: J Peter
authors.initials: JP
authors.lastname: Beck
authors.affiliation: Salt Lake City Veterans Affairs Medical Center (SLC VAMC), 500 Foothill Drive, Salt Lake City, UT, 84148, USA.|Department of Orthopaedics, University of Utah, Salt Lake City, USA.
authors.forename: Robert Z
authors.initials: RZ
authors.lastname: Tashjian
authors.affiliation: Salt Lake City Veterans Affairs Medical Center (SLC VAMC), 500 Foothill Drive, Salt Lake City, UT, 84148, USA.|Department of Medicine, Division of Rheumatology, University of Utah, Salt Lake City, USA.
authors.forename: Grant W
authors.initials: GW
authors.lastname: Cannon
mesh_terms: D057189:Checklist; D002983:Clinical Competence / Q000592:standards; D004504:Education, Medical, Undergraduate / Q000379:methods / Q000592:standards; D004521:Educational Measurement / Q000379:methods / Q000592:standards; D006801:Humans; D007717:Knee; D009140:Musculoskeletal Diseases / Q000175:diagnosis*; D009985:Orthopedics / Q000193:education*; D010808:Physical Examination / Q000592:standards; D018794:Problem-Based Learning / Q000592:standards; D015203:Reproducibility of Results; D012782:Shoulder; D014481:United States