Statistical tools to improve assessing agreement between several observers.
In the context of assessing the impact of management and environmental factors on animal health, behaviour or performance it has become increasingly important to conduct (epidemiological) studies in the field. Hence, the number of investigated farms per study is considerably high so that numerous observers are needed for investigation. In order to maintain the quality and validity of study results calibration meetings where observers are trained and the current level of agreement is assessed have to be conducted to minimise the observer effect. When study animals were rated independently by the same observers by a categorical variable the exclusion test can be performed to identify disagreeing observers. This statistical test compares for each variable and each observer the observer-specific agreement with the overall agreement among all observers based on kappa coefficients. It accounts for two major challenges, namely the absence of a gold-standard observer and different data type comprising ordinal, nominal and binary data. The presented methods are applied on a reliability study to assess the agreement among eight observers rating welfare parameters of laying hens. The degree to which the observers agreed depended on the investigated item (global weighted kappa coefficients: 0.37 to 0.94). The proposed method and graphical description served to assess the direction and degree to which an observer deviates from the others. It is suggested to further improve studies with numerous observers by conducting calibration meetings and accounting for observer bias.
1 Department of Biometry, Epidemiology and Information Processing, WHO Collaborating Centre for Research and Training in Veterinary Public Health, University of Veterinary Medicine, Hannover, Germany.
I
I
Ruddat
2 Friedrich-Loeffler-Institut, Institute of Animal Welfare and Animal Husbandry, Celle, Germany.
B
B
Scholz
3 Department of Veterinary Science, Faculty of Veterinary Medicine, Chair of Animal Welfare, Ethology, Animal Hygiene and Animal Housing, Ludwig-Maximilians-University, Munich, Germany.
S
S
Bergmann
2 Friedrich-Loeffler-Institut, Institute of Animal Welfare and Animal Husbandry, Celle, Germany.
A-L
AL
Buehring
4 Institute for Animal Breeding and Genetics, University of Veterinary Medicine, Hannover, Germany.
S
S
Fischer
5 Department of Farm Animal Ethology and Poultry Science, University of Hohenheim, Stuttgart, Germany.
A
A
Manton
3 Department of Veterinary Science, Faculty of Veterinary Medicine, Chair of Animal Welfare, Ethology, Animal Hygiene and Animal Housing, Ludwig-Maximilians-University, Munich, Germany.
D
D
Prengel
3 Department of Veterinary Science, Faculty of Veterinary Medicine, Chair of Animal Welfare, Ethology, Animal Hygiene and Animal Housing, Ludwig-Maximilians-University, Munich, Germany.
E
E
Rauch
3 Department of Veterinary Science, Faculty of Veterinary Medicine, Chair of Animal Welfare, Ethology, Animal Hygiene and Animal Housing, Ludwig-Maximilians-University, Munich, Germany.
S
S
Steiner
6 Bavarian State Research Center for Agriculture, Kitzingen, Germany.
S
S
Wiedmann
1 Department of Biometry, Epidemiology and Information Processing, WHO Collaborating Centre for Research and Training in Veterinary Public Health, University of Veterinary Medicine, Hannover, Germany.
L
L
Kreienbrock
1 Department of Biometry, Epidemiology and Information Processing, WHO Collaborating Centre for Research and Training in Veterinary Public Health, University of Veterinary Medicine, Hannover, Germany.
A
A
Campe
D000822:Animal Husbandry / Q000706:statistics & numerical data*; D000818:Animals; D015588:Observer Variation; D015203:Reproducibility of Results