The Lung Image Database Consortium (LIDC) data collection process for nodule detection and annotation.
OBJECTIVE
The Lung Image Database Consortium (LIDC) is developing a publicly available database of thoracic computed tomography (CT) scans as a medical imaging research resource to promote the development of computer-aided detection or characterization of pulmonary nodules. To obtain the best estimate of the location and spatial extent of lung nodules, expert thoracic radiologists reviewed and annotated each scan. Because a consensus panel approach was neither feasible nor desirable, a unique two-phase, multicenter data collection process was developed to allow multiple radiologists at different centers to asynchronously review and annotate each CT scan. This data collection process was also intended to capture the variability among readers.


METHODS
Four radiologists reviewed each scan using the following process. In the first or "blinded" phase, each radiologist reviewed the CT scan independently. In the second or "unblinded" review phase, results from all four blinded reviews were compiled and presented to each radiologist for a second review, allowing the radiologists to review their own annotations together with the annotations of the other radiologists. The results of each radiologist's unblinded review were compiled to form the final unblinded review. An XML-based message system was developed to communicate the results of each reading.


RESULTS
This two-phase data collection process was designed, tested, and implemented across the LIDC. More than 500 CT scans have been read and annotated using this method by four expert readers; these scans either are currently publicly available at http://ncia.nci.nih.gov or will be in the near future.


CONCLUSIONS
A unique data collection process was developed, tested, and implemented that allowed multiple readers at distributed sites to asynchronously review CT scans multiple times. This process captured the opinions of each reader regarding the location and spatial extent of lung nodules.
McNitt-Gray
Michael F
MF
Department of Radiology, David Geffen School of Medicine at UCLA, Suite 650, 924 Westwood Blvd, Los Angeles, CA 90095-1721, USA. mmcnittgray@mednet.ucla.edu
Armato
Samuel G
SG
Meyer
Charles R
CR
Reeves
Anthony P
AP
McLennan
Geoffrey
G
Pais
Richie C
RC
Freymann
John
J
Brown
Matthew S
MS
Engelmann
Roger M
RM
Bland
Peyton H
PH
Laderach
Gary E
GE
Piker
Chris
C
Guo
Junfeng
J
Towfic
Zaid
Z
Qing
David P-Y
DP
Yankelevitz
David F
DF
Aberle
Denise R
DR
van Beek
Edwin J R
EJ
MacMahon
Heber
H
Kazerooni
Ella A
EA
Croft
Barbara Y
BY
Clarke
Laurence P
LP
D003625:Data Collection / Q000379:methods*; D003628:Database Management Systems; D019992:Databases as Topic; D003936:Diagnosis, Computer-Assisted; D006801:Humans; D051188:Knowledge Bases; D008175:Lung Neoplasms / Q000000981:diagnostic imaging*; D015588:Observer Variation; D013902:Radiography, Thoracic; D011871:Radiology; D011873:Radiology Information Systems; D003074:Solitary Pulmonary Nodule / Q000000981:diagnostic imaging; D014057:Tomography, X-Ray Computed