title: Variability in observer performance studies experimental observations.
abstract: OBJECTIVE
  The aim of the study is to assess variance components in observer performance studies and the possible impact on study results and conclusions.
  
  
  METHODS
  Two previously performed retrospective receiver operating characteristic-type observer performance studies to evaluate the performance of seven radiologists in detecting interstitial disease on conventional posteroanterior chest films and nine radiologists in detecting interstitial disease on a high-resolution workstation were reanalyzed by using the Beiden, Wagner, and Campbell nine-component model to estimate the different variance components. We estimated case-, reader-, and mode-related components of the variance for the group as a whole and after excluding (round robin) each reader. Overall variance was evaluated, and the effect of individual readers on overall study conclusions was assessed.
  
  
  RESULTS
  Overall results and conclusions of the reanalysis agreed with the original one in that, as a group, radiologists performed significantly better when using conventional films (P < .05) in both studies. Reader variability was large compared with all other components, and in one study, it was substantially larger for the workstation reading mode. Reader variability was affected substantially by one observer in each study, and in one study, reader-by-mode variability was affected by another reader who performed better on the workstation.
  
  
  CONCLUSIONS
  Estimates of variance components can shed light on the appropriateness of study design, as well as the sensitivity of results to the inclusion (or exclusion) of individual observers.
authors.affiliation: Department of Radiology, School of Medicine, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15213-3180, USA. gurd@upmc.edu
authors.forename: David
authors.initials: D
authors.lastname: Gur
authors.forename: Howard E
authors.initials: HE
authors.lastname: Rockette
authors.forename: Glenn S
authors.initials: GS
authors.lastname: Maitz
authors.forename: Jill L
authors.initials: JL
authors.lastname: King
authors.forename: Amy H
authors.initials: AH
authors.lastname: Klym
authors.forename: Andriy I
authors.initials: AI
authors.lastname: Bandos
mesh_terms: D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008171:Lung Diseases / Q000000981:diagnostic imaging*; D015588:Observer Variation; D011785:Quality Assurance, Health Care / Q000379:methods*; D012372:ROC Curve; D011859:Radiography; D015203:Reproducibility of Results; D012189:Retrospective Studies; D012680:Sensitivity and Specificity; D013647:Task Performance and Analysis; D014481:United States