Inter-observer Variability Analysis of Automatic Lung Delineation in Normal and Disease Patients.
Human interaction has become almost mandatory for an automated medical system wishing to be accepted by clinical regulatory agencies such as Food and Drug Administration. Since this interaction causes variability in the gathered data, the inter-observer and intra-observer variability must be analyzed in order to validate the accuracy of the system. This study focuses on the variability from different observers that interact with an automated lung delineation system that relies on human interaction in the form of delineation of the lung borders. The database consists of High Resolution Computed Tomography (HRCT): 15 normal and 81 diseased patients' images taken retrospectively at five levels per patient. Three observers manually delineated the lungs borders independently and using software called ImgTracer™ (AtheroPoint™, Roseville, CA, USA) to delineate the lung boundaries in all five levels of 3-D lung volume. The three observers consisted of Observer-1: lesser experienced novice tracer who is a resident in radiology under the guidance of radiologist, whereas Observer-2 and Observer-3 are lung image scientists trained by lung radiologist and biomedical imaging scientist and experts. The inter-observer variability can be shown by comparing each observer's tracings to the automated delineation and also by comparing each manual tracing of the observers with one another. The normality of the tracings was tested using D'Agostino-Pearson test and all observers tracings showed a normal P-value higher than 0.05. The analysis of variance (ANOVA) test between three observers and automated showed a P-value higher than 0.89 and 0.81 for the right lung (RL) and left lung (LL), respectively. The performance of the automated system was evaluated using Dice Similarity Coefficient (DSC), Jaccard Index (JI) and Hausdorff (HD) Distance measures. Although, Observer-1 has lesser experience compared to Obsever-2 and Obsever-3, the Observer Deterioration Factor (ODF) shows that Observer-1 has less than 10% difference compared to the other two, which is under acceptable range as per our analysis. To compare between observers, this study used regression plots, Bland-Altman plots, two tailed T-test, Mann-Whiney, Chi-Squared tests which showed the following P-values for RL and LL: (i) Observer-1 and Observer-3 were: 0.55, 0.48, 0.29 for RL and 0.55, 0.59, 0.29 for LL; (ii) Observer-1 and Observer-2 were: 0.57, 0.50, 0.29 for RL and 0.54, 0.59, 0.29 for LL; (iii) Observer-2 and Observer-3 were: 0.98, 0.99, 0.29 for RL and 0.99, 0.99, 0.29 for LL. Further, CC and R-squared coefficients were computed between observers which came out to be 0.9 for RL and LL. All three observers however manage to show the feature that diseased lungs are smaller than normal lungs in terms of area.
Azienda Ospedaliero Universitaria (A.O.U.) di Cagliari - Polo di Monserrato, Università di Cagliari, s.s. 554 Monserrato, Cagliari, 09045, Italy.
Luca
L
Saba
UTM Razak School of Engineering and Advanced Technology, Universiti Teknologi Malaysia, Johor Bahru, Malaysia.
Joel C M
JC
Than
Department of Engineering, UTM Razak School of Engineering and Advanced Technology, Universiti Teknologi Malaysia, Johor Bahru, Malaysia.
Norliza M
NM
Noor
Institute of Mathematical Sciences, Faculty of Science, University of Malaya, Kuala Lumpur, Malaysia.
Omar M
OM
Rijal
Department of Diagnostic Imaging, Kuala Lumpur Hospital, Kuala Lumpur, Malaysia.
Rosminah M
RM
Kassim
Institute of Respiratory Medicine, Kuala Lumpur, Malaysia.
Ashari
A
Yunus
UTM Razak School of Engineering and Advanced Technology, Universiti Teknologi Malaysia, Johor Bahru, Malaysia.
Chue R
CR
Ng
Global Biomedical Technologies, Inc., Roseville, CA, USA. jasjit.suri@atheropoint.com.|AtheroPoint™ LLC, Roseville, CA, USA. jasjit.suri@atheropoint.com.|Department of Electrical Engineering (Affl.), Idaho State University, Pocatello, ID, USA. jasjit.suri@atheropoint.com.
Jasjit S
JS
Suri
D000293:Adolescent; D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D005260:Female; D006801:Humans; D007090:Image Interpretation, Computer-Assisted; D008175:Lung Neoplasms / Q000175:diagnosis* / Q000473:pathology; D008297:Male; D008875:Middle Aged; D015588:Observer Variation; D014057:Tomography, X-Ray Computed / Q000379:methods*; D014481:United States; D055815:Young Adult
Inter-observer; Lung, CT, Automated delineation; Manual delineation; Reliability; Stability; Statistical tests; Variability