title: How Well Do Raters Agree on the Development Stage of Caenorhabditis elegans?
abstract: The assessment of inter-rater reliability is a topic that is infrequently addressed in Caenorhabditis elegans research, despite the existence of sophisticated statistical methods and the strong interest in the field in obtaining reliable and accurate data. This study applies statistical modeling as a robust means of analyzing the performance of worm researchers measuring the stage of worm development in terms of the two independent factors that comprise "agreement", which are (1) accuracy, representing trueness, a lack of systematic differences, or lack of bias, and (2) precision, representing reliability or the extent to which random differences are small. In our study, multiple raters assessed the same sample of worms to determine the developmental stage of each animal, and we collected data linking each scorer with their assessment for each worm. To describe the agreement of the raters, we developed a structural equation model with latent variables and thresholds, which assumes that all the raters are jointly scoring each worm. This common factor model separately quantifies the two aspects of agreement. The stage-specific thresholds examine accuracy and characterize the relative biases of each rater during the scoring process. The factor loadings for each rater examine the precision and characterizes the random error of the rater. Within our group, we found that the overall agreement was good, while certain adjustments in particular raters would have decreased systematic differences. Hence, the use of developmental stage as an experimental outcome can be both accurate and precise.
authors.affiliation: Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America.
authors.forename: Annabel A
authors.initials: AA
authors.lastname: Ferguson
authors.affiliation: Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America; Department of Ophthalmology, School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America; Department of Orthodontics, School of Dental Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America.
authors.forename: Richard A
authors.initials: RA
authors.lastname: Bilonick
authors.affiliation: Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America; Center for Occupational Biostatistics and Epidemiology and Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America.
authors.forename: Jeanine M
authors.initials: JM
authors.lastname: Buchanich
authors.affiliation: Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America; Center for Occupational Biostatistics and Epidemiology and Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, Pennsylvania, United States of America.
authors.forename: Gary M
authors.initials: GM
authors.lastname: Marsh
authors.affiliation: Division of Geriatrics, Gerontology, and Palliative Medicine, Department of Medicine, UTHSCSA, San Antonio, Texas, United States of America; Center for Healthy Aging, UTHSCSA, San Antonio, Texas, United States of America; San Antonio GRECC, STVAHCS, San Antonio, Texas, United States of America.
authors.forename: Alfred L
authors.initials: AL
authors.lastname: Fisher
mesh_terms: D000818:Animals; D017173:Caenorhabditis elegans / Q000254:growth & development*; D015233:Models, Statistical; D015588:Observer Variation; D015203:Reproducibility of Results; D012106:Research