The Lung Image Database Consortium (LIDC) data collection process for nodule detection and annotation.
OBJECTIVE
The Lung Image Database Consortium (LIDC) is developing a publicly available database of thoracic computed tomography (CT) scans as a medical imaging research resource to promote the development of computer-aided detection or characterization of pulmonary nodules. To obtain the best estimate of the location and spatial extent of lung nodules, expert thoracic radiologists reviewed and annotated each scan. Because a consensus panel approach was neither feasible nor desirable, a unique two-phase, multicenter data collection process was developed to allow multiple radiologists at different centers to asynchronously review and annotate each CT scan. This data collection process was also intended to capture the variability among readers.


METHODS
Four radiologists reviewed each scan using the following process. In the first or "blinded" phase, each radiologist reviewed the CT scan independently. In the second or "unblinded" review phase, results from all four blinded reviews were compiled and presented to each radiologist for a second review, allowing the radiologists to review their own annotations together with the annotations of the other radiologists. The results of each radiologist's unblinded review were compiled to form the final unblinded review. An XML-based message system was developed to communicate the results of each reading.


RESULTS
This two-phase data collection process was designed, tested, and implemented across the LIDC. More than 500 CT scans have been read and annotated using this method by four expert readers; these scans either are currently publicly available at http://ncia.nci.nih.gov or will be in the near future.


CONCLUSIONS
A unique data collection process was developed, tested, and implemented that allowed multiple readers at distributed sites to asynchronously review CT scans multiple times. This process captured the opinions of each reader regarding the location and spatial extent of lung nodules.
Department of Radiology, David Geffen School of Medicine at UCLA, Suite 650, 924 Westwood Blvd, Los Angeles, CA 90095-1721, USA. mmcnittgray@mednet.ucla.edu
Michael F
MF
McNitt-Gray
Samuel G
SG
Armato
Charles R
CR
Meyer
Anthony P
AP
Reeves
Geoffrey
G
McLennan
Richie C
RC
Pais
John
J
Freymann
Matthew S
MS
Brown
Roger M
RM
Engelmann
Peyton H
PH
Bland
Gary E
GE
Laderach
Chris
C
Piker
Junfeng
J
Guo
Zaid
Z
Towfic
David P-Y
DP
Qing
David F
DF
Yankelevitz
Denise R
DR
Aberle
Edwin J R
EJ
van Beek
Heber
H
MacMahon
Ella A
EA
Kazerooni
Barbara Y
BY
Croft
Laurence P
LP
Clarke
D003625:Data Collection / Q000379:methods*; D003628:Database Management Systems; D019992:Databases as Topic; D003936:Diagnosis, Computer-Assisted; D006801:Humans; D051188:Knowledge Bases; D008175:Lung Neoplasms / Q000000981:diagnostic imaging*; D015588:Observer Variation; D013902:Radiography, Thoracic; D011871:Radiology; D011873:Radiology Information Systems; D003074:Solitary Pulmonary Nodule / Q000000981:diagnostic imaging; D014057:Tomography, X-Ray Computed