Interrater reliability for the Four Habits Coding Scheme as part of a randomized controlled trial.
OBJECTIVE
To describe the process for developing interrater reliability (IRR) for the Four Habits Coding Scheme (4HCS) for a heterogeneous material as part of a randomized controlled trial.


METHODS
Videotapes from 497 hospital encounters involving 71 doctors from most clinical specialties were collected. Four experienced psychology students were trained as raters. We calculated Pearson's r and the intraclass correlation (ICC) on the total score across consecutive samples of twenty videos, and Pearson's r on single videos across items in the initial coding phase.


RESULTS
After 18h of training and one rating session, the total score Pearson's r and ICC exceeded .70 for all pairs of raters. Across items within single videos, the Pearson's r was never below 0.60 after the first 50 videos. At item and habit level Pearson's r remained unsatisfactory for some rater pairs mostly due to low variance on some items.


CONCLUSIONS
Based on the evaluation of the effect of communication skills training via a total score, IRR was satisfactory for the 4HCS as applied to heterogeneous material. However, good reliability at item level was difficult to achieve.


CONCLUSIONS
4HCS may be used as an outcome measure for clinical communication skills in randomized controlled trials.
Fossli Jensen
Bård
B
HØKH, Research Centre, Akershus University Hospital, Lørenskog, Norway. b.f.jensen@medisin.uio.no
Gulbrandsen
Pål
P
Benth
Jurate Saltyte
JS
Dahl
Fredrik A
FA
Krupat
Edward
E
Finset
Arnstein
A
D000293:Adolescent; D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D002648:Child; D002675:Child, Preschool; D059019:Clinical Coding / Q000592:standards*; D003142:Communication; D018592:Cross-Over Studies; D005260:Female; D006184:Habits; D006769:Hospitals, General; D006801:Humans; D007223:Infant; D007231:Infant, Newborn; D008297:Male; D008875:Middle Aged; D009664:Norway; D015588:Observer Variation; D010819:Physician's Role / Q000523:psychology; D010817:Physician-Patient Relations; D011361:Professional Competence / Q000592:standards*; D015203:Reproducibility of Results; D011795:Surveys and Questionnaires; D014743:Videotape Recording; D055815:Young Adult