Measuring Interobserver Disagreement in Rating Diagnostic Characteristics of Pulmonary Nodule Using the Lung Imaging Database Consortium and Image Database Resource Initiative.
The purpose of this study was to measure and analyze interobserver disagreement in rating diagnostic characteristics of pulmonary nodules on computed tomography scans using the Lung Imaging Database Consortium and Image Database Resource Initiative (LIDC/IDRI) database, and then to provide investigators with understanding the variability in rating diagnostic characteristics among radiologists.



A histogram-based accumulated nodule-level approach is proposed to measure interobserver disagreement in rating diagnostic characteristics of pulmonary nodules among radiologists. The mean rating differences of radiologists on nodule level are calculated; next, a histogram of the accumulated nodule-level disagreements is constructed; and finally, mean, variance, skewness, and kurtosis statistics based on the histogram are extracted to analyze and summary interobserver disagreement in terms of the assessment of diagnostic characteristics of radiologists. Using the developed computer scheme, the disagreement of radiologists in rating all of 1880 distinct nodules from 1018 computed tomography scans are analyzed using original ratings as well as combined ratings according to the LIDC/IDRI instruction.



The interobserver disagreement in rating diagnostic characteristics according to the defined categories of the LIDC/IDRI is substantial. The mean values of disagreement range from 0.0052 to 0.2341. The highest disagreement lies in rating subtlety characteristics, whereas internal structure receives the lowest disagreement of 0.0052. The calcification, texture, spiculation, lobulation, malignancy, sphericity, and margin receive disagreements of 0.0393, 0.1351, 0.1616, 0.1943, 0.2144, 0.2174, and 0.2228, respectively.



Disagreements exist across radiologists in rating diagnostic characteristics of pulmonary nodules, and the disagreement levels vary from each other. Agreement among radiologists is improved by combining ratings according to the LIDC/IDRI instruction. For investigators, understanding and appreciating the disagreement level of each diagnostic characteristic is required when using them in related researches.
College of Computer Science and Electronic Engineering and Collaboration and Innovation Center for Digital Chinese Medicine of 2011 Project of Colleges and Universities, Hunan University, Changsha 410082, China. Electronic address: hllin@hnu.edu.cn.
Hongli
H
Lin
College of Computer Science and Electronic Engineering and Collaboration and Innovation Center for Digital Chinese Medicine of 2011 Project of Colleges and Universities, Hunan University, Changsha 410082, China.
Changxing
C
Huang
College of Computer Science and Electronic Engineering and Collaboration and Innovation Center for Digital Chinese Medicine of 2011 Project of Colleges and Universities, Hunan University, Changsha 410082, China.
Weisheng
W
Wang
College of Computer Science and Electronic Engineering and Collaboration and Innovation Center for Digital Chinese Medicine of 2011 Project of Colleges and Universities, Hunan University, Changsha 410082, China.
Jiawei
J
Luo
Department of Computer Science, University of Regina, Regina, SK, Canada.
Xuedong
X
Yang
College of Computer Science and Electronic Engineering and Collaboration and Innovation Center for Digital Chinese Medicine of 2011 Project of Colleges and Universities, Hunan University, Changsha 410082, China.
Yuling
Y
Liu
D016208:Databases, Factual / Q000706:statistics & numerical data; D003951:Diagnostic Errors / Q000517:prevention & control*; D006801:Humans; D008175:Lung Neoplasms / Q000175:diagnosis / Q000000981:diagnostic imaging; D055613:Multiple Pulmonary Nodules / Q000175:diagnosis / Q000000981:diagnostic imaging; D015588:Observer Variation; D011857:Radiographic Image Interpretation, Computer-Assisted / Q000379:methods*; D011871:Radiology / Q000379:methods / Q000592:standards; D015203:Reproducibility of Results; D003074:Solitary Pulmonary Nodule / Q000175:diagnosis / Q000000981:diagnostic imaging; D014057:Tomography, X-Ray Computed / Q000379:methods / Q000706:statistics & numerical data
CT; LIDC/IDRI Database; interobserver disagreement; pulmonary nodule