title: Reliability studies of diagnostic tests are not using enough observers for robust estimation of interobserver agreement: a simulation study.
abstract: OBJECTIVE
  Any attempt to generalize the performance of a subjective diagnostic method should take into account the sample variation in both cases and readers. Most current measures of the performance of a test, especially the indices of reliability, only tackle the variation of cases, and hence are not suitable for generalizing results across the population of readers. We attempted to study the effect of readers' variation on two measures of multireader reliability: pair-wise agreement and Fleiss' kappa.
  
  
  METHODS
  We used a normal hierarchical model with a latent trait (signal) variable to simulate a binary decision-making task by different number of readers on an infinite sample of cases.
  
  
  RESULTS
  It could be shown that both measures, especially Fleiss' kappa, have a large sample variance when estimated by a small number of readers, casting doubt on their accuracy given the number of readers typically used in current reliability studies.
  
  
  CONCLUSIONS
  The majority of the current agreement studies is likely limited by the number of readers and is unlikely to produce a reliable estimate of reader agreement.
authors.affiliation: Center for Health Evaluation and Outcome Research (CHEOS), University of British Columbia, Vancouver BC, Canada V6Z 1Y6. msafavi@interchange.ubc.ca
authors.forename: Mohsen
authors.initials: M
authors.lastname: Sadatsafavi
authors.forename: Mehdi
authors.initials: M
authors.lastname: Najafzadeh
authors.forename: Larry
authors.initials: L
authors.lastname: Lynd
authors.forename: Carlo
authors.initials: C
authors.lastname: Marra
mesh_terms: D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D003955:Diagnostic Tests, Routine; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results; D018401:Sample Size; D013223:Statistics as Topic