title: The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans.
abstract: OBJECTIVE
  The purpose of this study was to analyze the variability of experienced thoracic radiologists in the identification of lung nodules on computed tomography (CT) scans and thereby to investigate variability in the establishment of the "truth" against which nodule-based studies are measured.
  
  
  METHODS
  Thirty CT scans were reviewed twice by four thoracic radiologists through a two-phase image annotation process. During the initial "blinded read" phase, radiologists independently marked lesions they identified as "nodule >or=3 mm (diameter)," "nodule <3 mm," or "non-nodule >or=3 mm." During the subsequent "unblinded read" phase, the blinded read results of all four radiologists were revealed to each radiologist, who then independently reviewed their marks along with the anonymous marks of their colleagues; a radiologist's own marks then could be deleted, added, or left unchanged. This approach was developed to identify, as completely as possible, all nodules in a scan without requiring forced consensus.
  
  
  RESULTS
  After the initial blinded read phase, 71 lesions received "nodule >or=3 mm" marks from at least one radiologist; however, all four radiologists assigned such marks to only 24 (33.8%) of these lesions. After the unblinded reads, a total of 59 lesions were marked as "nodule >or=3 mm" by at least one radiologist. Twenty-seven (45.8%) of these lesions received such marks from all four radiologists, three (5.1%) were identified as such by three radiologists, 12 (20.3%) were identified by two radiologists, and 17 (28.8%) were identified by only a single radiologist.
  
  
  CONCLUSIONS
  The two-phase image annotation process yields improved agreement among radiologists in the interpretation of nodules >or=3 mm. Nevertheless, substantial variability remains across radiologists in the task of lung nodule identification.
authors.affiliation: The University of Chicago, Department of Radiology, MC 2026, The University of Chicago, 5841 S. Maryland Avenue, Chicago, IL 60637, USA. s-armato@uchicago.edu
authors.forename: Samuel G
authors.initials: SG
authors.lastname: Armato
authors.forename: Michael F
authors.initials: MF
authors.lastname: McNitt-Gray
authors.forename: Anthony P
authors.initials: AP
authors.lastname: Reeves
authors.forename: Charles R
authors.initials: CR
authors.lastname: Meyer
authors.forename: Geoffrey
authors.initials: G
authors.lastname: McLennan
authors.forename: Denise R
authors.initials: DR
authors.lastname: Aberle
authors.forename: Ella A
authors.initials: EA
authors.lastname: Kazerooni
authors.forename: Heber
authors.initials: H
authors.lastname: MacMahon
authors.forename: Edwin J R
authors.initials: EJ
authors.lastname: van Beek
authors.forename: David
authors.initials: D
authors.lastname: Yankelevitz
authors.forename: Eric A
authors.initials: EA
authors.lastname: Hoffman
authors.forename: Claudia I
authors.initials: CI
authors.lastname: Henschke
authors.forename: Rachael Y
authors.initials: RY
authors.lastname: Roberts
authors.forename: Matthew S
authors.initials: MS
authors.lastname: Brown
authors.forename: Roger M
authors.initials: RM
authors.lastname: Engelmann
authors.forename: Richard C
authors.initials: RC
authors.lastname: Pais
authors.forename: Christopher W
authors.initials: CW
authors.lastname: Piker
authors.forename: David
authors.initials: D
authors.lastname: Qing
authors.forename: Masha
authors.initials: M
authors.lastname: Kocherginsky
authors.forename: Barbara Y
authors.initials: BY
authors.lastname: Croft
authors.forename: Laurence P
authors.initials: LP
authors.lastname: Clarke
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D016208:Databases, Factual; D006801:Humans; D008175:Lung Neoplasms / Q000000981:diagnostic imaging; D015588:Observer Variation; D010363:Pattern Recognition, Automated / Q000379:methods*; D011361:Professional Competence / Q000706:statistics & numerical data*; D011856:Radiographic Image Enhancement / Q000379:methods; D011857:Radiographic Image Interpretation, Computer-Assisted / Q000379:methods*; D011871:Radiology / Q000706:statistics & numerical data; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D003074:Solitary Pulmonary Nodule / Q000000981:diagnostic imaging*; D014481:United States