title: Evaluation of lung MDCT nodule annotation across radiologists and methods.
abstract: OBJECTIVE
  Integral to the mission of the National Institutes of Health-sponsored Lung Imaging Database Consortium is the accurate definition of the spatial location of pulmonary nodules. Because the majority of small lung nodules are not resected, a reference standard from histopathology is generally unavailable. Thus assessing the source of variability in defining the spatial location of lung nodules by expert radiologists using different software tools as an alternative form of truth is necessary.
  
  
  METHODS
  The relative differences in performance of six radiologists each applying three annotation methods to the task of defining the spatial extent of 23 different lung nodules were evaluated. The variability of radiologists' spatial definitions for a nodule was measured using both volumes and probability maps (p-map). Results were analyzed using a linear mixed-effects model that included nested random effects.
  
  
  RESULTS
  Across the combination of all nodules, volume and p-map model parameters were found to be significant at P < .05 for all methods, all radiologists, and all second-order interactions except one. The radiologist and methods variables accounted for 15% and 3.5% of the total p-map variance, respectively, and 40.4% and 31.1% of the total volume variance, respectively.
  
  
  CONCLUSIONS
  Radiologists represent the major source of variance as compared with drawing tools independent of drawing metric used. Although the random noise component is larger for the p-map analysis than for volume estimation, the p-map analysis appears to have more power to detect differences in radiologist-method combinations. The standard deviation of the volume measurement task appears to be proportional to nodule volume.
authors.affiliation: Department of Radiology, School of Medicine, University of Michigan, 109 Zina Pitcher Place, Ann Arbor, MI 48109-2200, USA. cmeyer@umich.edu
authors.forename: Charles R
authors.initials: CR
authors.lastname: Meyer
authors.forename: Timothy D
authors.initials: TD
authors.lastname: Johnson
authors.forename: Geoffrey
authors.initials: G
authors.lastname: McLennan
authors.forename: Denise R
authors.initials: DR
authors.lastname: Aberle
authors.forename: Ella A
authors.initials: EA
authors.lastname: Kazerooni
authors.forename: Heber
authors.initials: H
authors.lastname: Macmahon
authors.forename: Brian F
authors.initials: BF
authors.lastname: Mullan
authors.forename: David F
authors.initials: DF
authors.lastname: Yankelevitz
authors.forename: Edwin J R
authors.initials: EJ
authors.lastname: van Beek
authors.forename: Samuel G
authors.initials: SG
authors.lastname: Armato
authors.forename: Michael F
authors.initials: MF
authors.lastname: McNitt-Gray
authors.forename: Anthony P
authors.initials: AP
authors.lastname: Reeves
authors.forename: David
authors.initials: D
authors.lastname: Gur
authors.forename: Claudia I
authors.initials: CI
authors.lastname: Henschke
authors.forename: Eric A
authors.initials: EA
authors.lastname: Hoffman
authors.forename: Peyton H
authors.initials: PH
authors.lastname: Bland
authors.forename: Gary
authors.initials: G
authors.lastname: Laderach
authors.forename: Richie
authors.initials: R
authors.lastname: Pais
authors.forename: David
authors.initials: D
authors.lastname: Qing
authors.forename: Chris
authors.initials: C
authors.lastname: Piker
authors.forename: Junfeng
authors.initials: J
authors.lastname: Guo
authors.forename: Adam
authors.initials: A
authors.lastname: Starkey
authors.forename: Daniel
authors.initials: D
authors.lastname: Max
authors.forename: Barbara Y
authors.initials: BY
authors.lastname: Croft
authors.forename: Laurence P
authors.initials: LP
authors.lastname: Clarke
mesh_terms: D001185:Artificial Intelligence; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008175:Lung Neoplasms / Q000000981:diagnostic imaging; D015588:Observer Variation; D010363:Pattern Recognition, Automated / Q000379:methods*; D010820:Physicians / Q000706:statistics & numerical data*; D011361:Professional Competence; D011871:Radiology; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D003074:Solitary Pulmonary Nodule / Q000000981:diagnostic imaging*; D013647:Task Performance and Analysis; D014057:Tomography, X-Ray Computed / Q000706:statistics & numerical data*