title: Validity evidence for an OSCE to assess competency in systems-based practice and practice-based learning and improvement: a preliminary investigation.
abstract: OBJECTIVE
  To determine the psychometric properties and validity of an OSCE to assess the competencies of Practice-Based Learning and Improvement (PBLI) and Systems-Based Practice (SBP) in graduate medical education.
  
  
  METHODS
  An eight-station OSCE was piloted at the end of a three-week Quality Improvement elective for nine preventive medicine and endocrinology fellows at Mayo Clinic. The stations assessed performance in quality measurement, root cause analysis, evidence-based medicine, insurance systems, team collaboration, prescription errors, Nolan's model, and negotiation. Fellows' performance in each of the stations was assessed by three faculty experts using checklists and a five-point global competency scale. A modified Angoff procedure was used to set standards. Evidence for the OSCE's validity, feasibility, and acceptability was gathered.
  
  
  RESULTS
  Evidence for content and response process validity was judged as excellent by institutional content experts. Interrater reliability of scores ranged from 0.85 to 1 for most stations. Interstation correlation coefficients ranged from -0.62 to 0.99, reflecting case specificity. Implementation cost was approximately $255 per fellow. All faculty members agreed that the OSCE was realistic and capable of providing accurate assessments.
  
  
  CONCLUSIONS
  The OSCE provides an opportunity to systematically sample the different subdomains of Quality Improvement. Furthermore, the OSCE provides an opportunity for the demonstration of skills rather than the testing of knowledge alone, thus making it a potentially powerful assessment tool for SBP and PBLI. The study OSCE was well suited to assess SBP and PBLI. The evidence gathered through this study lays the foundation for future validation work.
authors.affiliation: Division of Preventive and Occupational Medicine, College of Medicine, Mayo Clinic, Rochester, Minnesota 55905, USA. varkey.prathibha@mayo.edu
authors.forename: Prathibha
authors.initials: P
authors.lastname: Varkey
authors.forename: Neena
authors.initials: N
authors.lastname: Natt
authors.forename: Timothy
authors.initials: T
authors.lastname: Lesnick
authors.forename: Steven
authors.initials: S
authors.lastname: Downing
authors.forename: Rachel
authors.initials: R
authors.lastname: Yudkowsky
mesh_terms: D002983:Clinical Competence; D003162:Competency-Based Education; D004503:Education, Medical, Graduate; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D011785:Quality Assurance, Health Care