Toward meaningful evaluation of clinical competence: the role of direct observation in clerkship ratings.
OBJECTIVE
The lack of direct observation by faculty may affect meaningful judgments of clinical competence. The purpose of this study was to explore the influence of direct observation on reliability and validity evidence for family medicine clerkship ratings of clinical performance.


METHODS
Preceptors rating family medicine clerks (n = 172) on a 16-item evaluation instrument noted the data-source for each rating: note review, case discussion, and/or direct observation. Mean data-source scores were computed and categorized as low, medium or high, with the high-score group including the most direct observation. Analyses examined the influence of data-source on interrater agreement and associations between clerkship clinical scores (CCS) and scores from the National Board of Medical Examiners (NBME(R)) subject examination as well as a fourth-year standardized patient-based clinical competence examination (M4CCE).


RESULTS
Interrater reliability increased as a function of data-source; for the low, medium, and high groups, intraclass correlation coefficients were.29,.50, and.74, respectively. For the high-score group, there were significant positive correlations between CCS and NBME score (r =.311, p =.054); and between CCS and M4CCE (r =.423, p =.009).


CONCLUSIONS
Reliability and validity evidence for clinical competence is enhanced when more direct observation is included as a basis for clerkship ratings.
Hasnain
Memoona
M
Department of Family Medicine, College of Medicine, University of Illinois at Chicago, 60612, USA. memoona@uic.edu
Connell
Karen J
KJ
Downing
Steven M
SM
Olthoff
Allan
A
Yudkowsky
Rachel
R
D002568:Certification / Q000379:methods; D002982:Clinical Clerkship; D002983:Clinical Competence; D004521:Educational Measurement / Q000379:methods*; D005180:Faculty, Medical; D005194:Family Practice / Q000193:education*; D005260:Female; D006801:Humans; D007396:Internship and Residency; D007398:Interpersonal Relations; D007600:Judgment; D008297:Male; D015588:Observer Variation; D015203:Reproducibility of Results