Evaluation of lung MDCT nodule annotation across radiologists and methods.
OBJECTIVE
Integral to the mission of the National Institutes of Health-sponsored Lung Imaging Database Consortium is the accurate definition of the spatial location of pulmonary nodules. Because the majority of small lung nodules are not resected, a reference standard from histopathology is generally unavailable. Thus assessing the source of variability in defining the spatial location of lung nodules by expert radiologists using different software tools as an alternative form of truth is necessary.


METHODS
The relative differences in performance of six radiologists each applying three annotation methods to the task of defining the spatial extent of 23 different lung nodules were evaluated. The variability of radiologists' spatial definitions for a nodule was measured using both volumes and probability maps (p-map). Results were analyzed using a linear mixed-effects model that included nested random effects.


RESULTS
Across the combination of all nodules, volume and p-map model parameters were found to be significant at P < .05 for all methods, all radiologists, and all second-order interactions except one. The radiologist and methods variables accounted for 15% and 3.5% of the total p-map variance, respectively, and 40.4% and 31.1% of the total volume variance, respectively.


CONCLUSIONS
Radiologists represent the major source of variance as compared with drawing tools independent of drawing metric used. Although the random noise component is larger for the p-map analysis than for volume estimation, the p-map analysis appears to have more power to detect differences in radiologist-method combinations. The standard deviation of the volume measurement task appears to be proportional to nodule volume.
Meyer
Charles R
CR
Department of Radiology, School of Medicine, University of Michigan, 109 Zina Pitcher Place, Ann Arbor, MI 48109-2200, USA. cmeyer@umich.edu
Johnson
Timothy D
TD
McLennan
Geoffrey
G
Aberle
Denise R
DR
Kazerooni
Ella A
EA
Macmahon
Heber
H
Mullan
Brian F
BF
Yankelevitz
David F
DF
van Beek
Edwin J R
EJ
Armato
Samuel G
SG
McNitt-Gray
Michael F
MF
Reeves
Anthony P
AP
Gur
David
D
Henschke
Claudia I
CI
Hoffman
Eric A
EA
Bland
Peyton H
PH
Laderach
Gary
G
Pais
Richie
R
Qing
David
D
Piker
Chris
C
Guo
Junfeng
J
Starkey
Adam
A
Max
Daniel
D
Croft
Barbara Y
BY
Clarke
Laurence P
LP
D001185:Artificial Intelligence; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008175:Lung Neoplasms / Q000000981:diagnostic imaging; D015588:Observer Variation; D010363:Pattern Recognition, Automated / Q000379:methods*; D010820:Physicians / Q000706:statistics & numerical data*; D011361:Professional Competence; D011871:Radiology; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D003074:Solitary Pulmonary Nodule / Q000000981:diagnostic imaging*; D013647:Task Performance and Analysis; D014057:Tomography, X-Ray Computed / Q000706:statistics & numerical data*