Relying on others' reliability: challenges in clinical teaching assessment.
BACKGROUND
The quality of the data generated from internally created faculty teaching instruments often draws skepticism. Strategies aimed at improving the reliability and validity of faculty teaching assessments tend to revolve around literature searches for a replacement instrument(s).


OBJECTIVE
The purpose was to test this "search-and-apply" method and discuss our experiences with it within the context of observational assessment practice.


METHODS
In a naturalistic pilot test, two previously validated faculty assessment instruments were paired with a global question. The reliability of both metrics was estimated.


RESULTS
Generalizability analyses indicated that for both pilot tested faculty teaching instruments, the global question was a more reliable measure of perceived clinical teaching effectiveness than a multiple-item inventory. Item analysis with Cronbach's coefficient alpha suggested redundant instrument content. Rater error accounted for the greatest proportion of the variance and straight-line responses occurred in approximately 28% of residents' appraisals.


CONCLUSIONS
The results of the present study draw attention to one of the common fallacies surrounding instrument-based assessment in medical education; the solution to improving one's assessment practice primarily involves identifying a previously published instrument from the literature. Academic centers need to invest in ongoing quality control efforts including the pilot testing of any proposed instruments.
Schulich School of Medicine & Dentistry, University of Western Ontario, London, Ontario, Canada. elaine.zibrowski@schulich.uwo.ca
Elaine M
EM
Zibrowski
Kathryn
K
Myers
Geoff
G
Norman
Mark A
MA
Goldszmidt
D001291:Attitude of Health Personnel; D002983:Clinical Competence / Q000592:standards / Q000706:statistics & numerical data*; D005180:Faculty, Medical / Q000592:standards / Q000706:statistics & numerical data*; D006801:Humans; D007398:Interpersonal Relations; D009864:Ontario; D010865:Pilot Projects; D011786:Quality Control; D015203:Reproducibility of Results; D012107:Research Design; D013223:Statistics as Topic; D013663:Teaching / Q000592:standards / Q000706:statistics & numerical data*