The Lung Image Database Consortium (LIDC): an evaluation of radiologist variability in the identification of lung nodules on CT scans.
OBJECTIVE
The purpose of this study was to analyze the variability of experienced thoracic radiologists in the identification of lung nodules on computed tomography (CT) scans and thereby to investigate variability in the establishment of the "truth" against which nodule-based studies are measured.


METHODS
Thirty CT scans were reviewed twice by four thoracic radiologists through a two-phase image annotation process. During the initial "blinded read" phase, radiologists independently marked lesions they identified as "nodule >or=3 mm (diameter)," "nodule <3 mm," or "non-nodule >or=3 mm." During the subsequent "unblinded read" phase, the blinded read results of all four radiologists were revealed to each radiologist, who then independently reviewed their marks along with the anonymous marks of their colleagues; a radiologist's own marks then could be deleted, added, or left unchanged. This approach was developed to identify, as completely as possible, all nodules in a scan without requiring forced consensus.


RESULTS
After the initial blinded read phase, 71 lesions received "nodule >or=3 mm" marks from at least one radiologist; however, all four radiologists assigned such marks to only 24 (33.8%) of these lesions. After the unblinded reads, a total of 59 lesions were marked as "nodule >or=3 mm" by at least one radiologist. Twenty-seven (45.8%) of these lesions received such marks from all four radiologists, three (5.1%) were identified as such by three radiologists, 12 (20.3%) were identified by two radiologists, and 17 (28.8%) were identified by only a single radiologist.


CONCLUSIONS
The two-phase image annotation process yields improved agreement among radiologists in the interpretation of nodules >or=3 mm. Nevertheless, substantial variability remains across radiologists in the task of lung nodule identification.
Armato
Samuel G
SG
The University of Chicago, Department of Radiology, MC 2026, The University of Chicago, 5841 S. Maryland Avenue, Chicago, IL 60637, USA. s-armato@uchicago.edu
McNitt-Gray
Michael F
MF
Reeves
Anthony P
AP
Meyer
Charles R
CR
McLennan
Geoffrey
G
Aberle
Denise R
DR
Kazerooni
Ella A
EA
MacMahon
Heber
H
van Beek
Edwin J R
EJ
Yankelevitz
David
D
Hoffman
Eric A
EA
Henschke
Claudia I
CI
Roberts
Rachael Y
RY
Brown
Matthew S
MS
Engelmann
Roger M
RM
Pais
Richard C
RC
Piker
Christopher W
CW
Qing
David
D
Kocherginsky
Masha
M
Croft
Barbara Y
BY
Clarke
Laurence P
LP
D000465:Algorithms; D001185:Artificial Intelligence; D016208:Databases, Factual; D006801:Humans; D008175:Lung Neoplasms / Q000000981:diagnostic imaging; D015588:Observer Variation; D010363:Pattern Recognition, Automated / Q000379:methods*; D011361:Professional Competence / Q000706:statistics & numerical data*; D011856:Radiographic Image Enhancement / Q000379:methods; D011857:Radiographic Image Interpretation, Computer-Assisted / Q000379:methods*; D011871:Radiology / Q000706:statistics & numerical data; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D003074:Solitary Pulmonary Nodule / Q000000981:diagnostic imaging*; D014481:United States