Using relative improvement over chance (RIOC) to examine agreement between tests: three case examples using studies of developmental coordination disorder (DCD) in children.
Although statistics such as kappa and phi are commonly used to assess agreement between tests, in situations where the base rate of a disorder in a population is low or high, these statistics tend to underestimate actual agreement. This can occur even if the tests are good and the classification of subjects is adequate. Relative improvement over chance or RIOC is a statistic that can correct for this bias; however, it is not yet commonly used in the health sciences or disability fields. In this paper, we introduce RIOC and demonstrate its application using the results from 3 previously published studies, all of which assessed the agreement between tests commonly used to identify children with a neurodevelopmental disorder known as developmental coordination disorder (or DCD). The results illustrate the differences between kappa and RIOC under conditions where the distribution of scores in a 2 × 2 table is unbalanced. The introduction of this statistic in the area of developmental disabilities research is encouraged.
Cairney
John
J
Department of Family Medicine, Offord Centre for Child Studies and the CanChild Centre for Childhood Disability Research, McMaster University, Hamilton, ON, Canada. cairnej@mcmaster.ca
Streiner
David L
DL
D002648:Child; D003627:Data Interpretation, Statistical; D004185:Disability Evaluation; D006801:Humans; D015233:Models, Statistical; D019957:Motor Skills Disorders / Q000175:diagnosis* / Q000534:rehabilitation*; D011237:Predictive Value of Tests