Inter-rater evaluation of a clinical scoring system in children with asthma.
BACKGROUND
Many clinical scores that measure the degree of asthma are used without adequate evaluation of inter-rater reliability. When reliability is tested, most often the Cohen K statistic is used, which limits the comparative results of only two raters at a time.


OBJECTIVE
To evaluate inter-rater agreement of a clinical asthma score using a multi-rater K statistic.


METHODS
Four raters administered a clinical asthma score to 17 children with clinical asthma. Five items were evaluated: O2 requirement, inspiratory breath sounds, accessory muscle use, expiratory wheeze, and cerebral function. For each, a score of zero indicated a normal state; one, moderate impairment; two, severe impairment. A multi-rater kappa statistic was used as a measure of agreement among all four raters simultaneously. This was applied using hand calculations then cross-checked by using a standard statistical syntax, a component of the Statistical Package for Social Sciences (SPSS 9.0).


RESULTS
Application of the multi-rater K statistic revealed strong agreement among raters on oxygenation (K = 0.759), moderate agreement for expiratory wheeze and cerebral function (K = 0.698), and poor agreement for accessory muscle use (K = 0.528) and inspiratory breath sounds (K = 0.316).


CONCLUSIONS
The level of agreement varied by item with the least subjective item, O2 requirement, demonstrating the highest inter-rater correlation. A multi-rater kappa statistic can be applied to data obtained from a clinical scoring instrument either manually or by using statistical syntax provided by SPSS.
Angelilli
Mary Lu
ML
Wayne State University School of Medicine, Pediatrics, Detroit, Michigan, USA. mangeli@med.wayne.edu
Thomas
Ronald
R
D000293:Adolescent; D001249:Asthma / Q000175:diagnosis* / Q000503:physiopathology; D002648:Child; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results; D012720:Severity of Illness Index; D013223:Statistics as Topic / Q000379:methods