Quality assessment of ordinal scale reproducibility: log-linear models provided useful information on scale structure.
OBJECTIVE
In health research, ordinal scales are extensively used. Reproducibility of ratings using these scales is important to assess their quality. This study aimed to compare two methods analyzing reproducibility: weighted Kappa statistic and log-linear models.


METHODS
Contributions of each method to the reproducibility assessment of ratings using ordinal scales were compared using intra- and interobserver data chosen in three different fields: Crow's feet scale in dermatology, dysplasia scale in oncology, updated Sydney scale in gastroenterology.


RESULTS
Both methods provided an agreement level. In addition, log-linear models allowed evaluation of the structure of agreement. For the Crow's feet scale, both methods gave equivalent high agreement levels. For the dysplasia scale, log-linear models highlighted scale defects and Kappa statistic showed a moderate agreement. For the updated Sydney scale, log-linear models underlined a null distinguishability between two adjacent categories, whereas Kappa statistic gave a high global agreement level.


CONCLUSIONS
Methods that can investigate level and structure of agreement between ordinal ratings are valuable tools, since they may highlight heterogeneities within the scales structure and suggest modifications to improve their reproducibility.
Valet
Fabien
F
Inserm U717, Département de Biostatistique et Informatique Médicale, Saint-Louis Hospital, 1 Avenue Claude Vellefaux, F-75010 Paris, University Paris 7, France. fabien.valet@paris7.jussieu.fr
Guinot
Christiane
C
Ezzedine
Khaled
K
Mary
Jean-Yves
JY
D000236:Adenoma / Q000473:pathology; D001706:Biopsy; D015179:Colorectal Neoplasms / Q000473:pathology; D003627:Data Interpretation, Statistical; D005260:Female; D016481:Helicobacter Infections / Q000473:pathology; D016480:Helicobacter pylori; D006801:Humans; D016014:Linear Models; D015588:Observer Variation; D015203:Reproducibility of Results; D015595:Skin Aging