title: Toward meaningful evaluation of clinical competence: the role of direct observation in clerkship ratings.
abstract: OBJECTIVE
  The lack of direct observation by faculty may affect meaningful judgments of clinical competence. The purpose of this study was to explore the influence of direct observation on reliability and validity evidence for family medicine clerkship ratings of clinical performance.
  
  
  METHODS
  Preceptors rating family medicine clerks (n = 172) on a 16-item evaluation instrument noted the data-source for each rating: note review, case discussion, and/or direct observation. Mean data-source scores were computed and categorized as low, medium or high, with the high-score group including the most direct observation. Analyses examined the influence of data-source on interrater agreement and associations between clerkship clinical scores (CCS) and scores from the National Board of Medical Examiners (NBME(R)) subject examination as well as a fourth-year standardized patient-based clinical competence examination (M4CCE).
  
  
  RESULTS
  Interrater reliability increased as a function of data-source; for the low, medium, and high groups, intraclass correlation coefficients were.29,.50, and.74, respectively. For the high-score group, there were significant positive correlations between CCS and NBME score (r =.311, p =.054); and between CCS and M4CCE (r =.423, p =.009).
  
  
  CONCLUSIONS
  Reliability and validity evidence for clinical competence is enhanced when more direct observation is included as a basis for clerkship ratings.
authors.affiliation: Department of Family Medicine, College of Medicine, University of Illinois at Chicago, 60612, USA. memoona@uic.edu
authors.forename: Memoona
authors.initials: M
authors.lastname: Hasnain
authors.forename: Karen J
authors.initials: KJ
authors.lastname: Connell
authors.forename: Steven M
authors.initials: SM
authors.lastname: Downing
authors.forename: Allan
authors.initials: A
authors.lastname: Olthoff
authors.forename: Rachel
authors.initials: R
authors.lastname: Yudkowsky
mesh_terms: D002568:Certification / Q000379:methods; D002982:Clinical Clerkship; D002983:Clinical Competence; D004521:Educational Measurement / Q000379:methods*; D005180:Faculty, Medical; D005194:Family Practice / Q000193:education*; D005260:Female; D006801:Humans; D007396:Internship and Residency; D007398:Interpersonal Relations; D007600:Judgment; D008297:Male; D015588:Observer Variation; D015203:Reproducibility of Results