title: Systems-based practice assessed with a performance-based examination simulated and scored by standardized participants in the health care system: feasibility and psychometric properties.
abstract: BACKGROUND
  Systems-based practice is one of the six general competencies proposed by the Accreditation Council for Graduate Medical Education in their Outcome Project. However, little has been published on its assessment--possibly because the systems-based practice competency has been viewed as difficult to define and measure.
  
  
  OBJECTIVE
  The purpose of this study was to determine whether a full performance-based examination of systems-based practice cases simulated and scored by standardized participants in the health care system could feasibly be constructed and implemented that would provide reliable and valid measurements.
  
  
  METHODS
  In the 1st year of the project (2008), four systems-based practice cases were developed and pilot tested with 13 residents. Videotapes of residents were studied to develop an instrument for subsequent assessment of performance by standardized participants. In the 2nd year (2009), the examination was expanded to a full 12 cases, which were completed by 11 second-year residents, and psychometric analyses were performed on the scores.
  
  
  RESULTS
  The generalizability coefficient for the full 12-case examination based on scoring by standardized participants was .71, which is nearly equal to that based on scoring by faculty physician observers, which was .78. The correlation between total scores obtained with standardized participants and physician observers was .78.
  
  
  CONCLUSIONS
  A performance-based examination can provide a feasible and reliable assessment of systems-based practice. However, attempts to evaluate convergent validity and discriminant validity-by correlating systems-based practice performance assessments with mean global ratings of residents on the 6 competencies by faculty throughout training-were unsuccessful, due to a lack of independence between the rated dimensions.
authors.affiliation: Department of Internal Medicine, Southern Illinois University School of Medicine, Springfield, Illinois 62702, USA. shingle@siumed.edu
authors.forename: Susan T
authors.initials: ST
authors.lastname: Hingle
authors.forename: Sherry
authors.initials: S
authors.lastname: Robinson
authors.forename: Jerry A
authors.initials: JA
authors.lastname: Colliver
authors.forename: Richard B
authors.initials: RB
authors.lastname: Rosher
authors.forename: Nancy
authors.initials: N
authors.lastname: McCann-Stone
mesh_terms: D000068:Accreditation; D002983:Clinical Competence / Q000592:standards*; D003695:Delivery of Health Care; D004503:Education, Medical, Graduate / Q000592:standards*; D004521:Educational Measurement / Q000379:methods / Q000592:standards*; D005240:Feasibility Studies; D007722:Health Knowledge, Attitudes, Practice; D006801:Humans; D011594:Psychometrics; D014743:Videotape Recording