title: Multi-site quality and variability analysis of 3D FDG PET segmentations based on phantom and clinical image data.
abstract: OBJECTIVE
  Radiomics utilizes a large number of image-derived features for quantifying tumor characteristics that can in turn be correlated with response and prognosis. Unfortunately, extraction and analysis of such image-based features is subject to measurement variability and bias. The challenge for radiomics is particularly acute in Positron Emission Tomography (PET) where limited resolution, a high noise component related to the limited stochastic nature of the raw data, and the wide variety of reconstruction options confound quantitative feature metrics. Extracted feature quality is also affected by tumor segmentation methods used to define regions over which to calculate features, making it challenging to produce consistent radiomics analysis results across multiple institutions that use different segmentation algorithms in their PET image analysis. Understanding each element contributing to these inconsistencies in quantitative image feature and metric generation is paramount for ultimate utilization of these methods in multi-institutional trials and clinical oncology decision making.
  
  
  METHODS
  To assess segmentation quality and consistency at the multi-institutional level, we conducted a study of seven institutional members of the National Cancer Institute Quantitative Imaging Network. For the study, members were asked to segment a common set of phantom PET scans acquired over a range of imaging conditions as well as a second set of head and neck cancer (HNC) PET scans. Segmentations were generated at each institution using their preferred approach. In addition, participants were asked to repeat segmentations with a time interval between initial and repeat segmentation. This procedure resulted in overall 806 phantom insert and 641 lesion segmentations. Subsequently, the volume was computed from the segmentations and compared to the corresponding reference volume by means of statistical analysis.
  
  
  RESULTS
  On the two test sets (phantom and HNC PET scans), the performance of the seven segmentation approaches was as follows. On the phantom test set, the mean relative volume errors ranged from 29.9 to 87.8% of the ground truth reference volumes, and the repeat difference for each institution ranged between -36.4 to 39.9%. On the HNC test set, the mean relative volume error ranged between -50.5 to 701.5%, and the repeat difference for each institution ranged between -37.7 to 31.5%. In addition, performance measures per phantom insert/lesion size categories are given in the paper. On phantom data, regression analysis resulted in coefficient of variation (CV) components of 42.5% for scanners, 26.8% for institutional approaches, 21.1% for repeated segmentations, 14.3% for relative contrasts, 5.3% for count statistics (acquisition times), and 0.0% for repeated scans. Analysis showed that the CV components for approaches and repeated segmentations were significantly larger on the HNC test set with increases by 112.7% and 102.4%, respectively.
  
  
  CONCLUSIONS
  Analysis results underline the importance of PET scanner reconstruction harmonization and imaging protocol standardization for quantification of lesion volumes. In addition, to enable a distributed multi-site analysis of FDG PET images, harmonization of analysis approaches and operator training in combination with highly automated segmentation methods seems to be advisable. Future work will focus on quantifying the impact of segmentation variation on radiomics system performance.
authors.affiliation: Department of Electrical and Computer Engineering, The University of Iowa, Iowa City, IA, USA.|Department of Internal Medicine, The University of Iowa, Iowa City, IA, USA.
authors.forename: Reinhard R
authors.initials: RR
authors.lastname: Beichel
authors.affiliation: Department of Biostatistics, The University of Iowa, Iowa City, IA, USA.
authors.forename: Brian J
authors.initials: BJ
authors.lastname: Smith
authors.affiliation: Department of Electrical and Computer Engineering, The University of Iowa, Iowa City, IA, USA.
authors.forename: Christian
authors.initials: C
authors.lastname: Bauer
authors.affiliation: Department of Electrical and Computer Engineering, The University of Iowa, Iowa City, IA, USA.|Department of Biomedical Engineering, The University of Iowa, Iowa City, IA, USA.
authors.forename: Ethan J
authors.initials: EJ
authors.lastname: Ulrich
authors.affiliation: School of Computing Science, Simon Fraser University, Burnaby, Canada.
authors.forename: Payam
authors.initials: P
authors.lastname: Ahmadvand
authors.affiliation: H Lee Moffitt Cancer Center, Tampa, FL, USA.
authors.forename: Mikalai M
authors.initials: MM
authors.lastname: Budzevich
authors.affiliation: H Lee Moffitt Cancer Center, Tampa, FL, USA.
authors.forename: Robert J
authors.initials: RJ
authors.lastname: Gillies
authors.affiliation: Department of Computer Science and Engineering, University of South Florida, Tampa, FL, USA.
authors.forename: Dmitry
authors.initials: D
authors.lastname: Goldgof
authors.affiliation: Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY, USA.
authors.forename: Milan
authors.initials: M
authors.lastname: Grkovski
authors.affiliation: School of Computing Science, Simon Fraser University, Burnaby, Canada.
authors.forename: Ghassan
authors.initials: G
authors.lastname: Hamarneh
authors.affiliation: Department of Radiology, Columbia University Medical Center, New York, NY, USA.
authors.forename: Qiao
authors.initials: Q
authors.lastname: Huang
authors.affiliation: Department of Radiology, University of Washington Medical Center, Seattle, WA, USA.
authors.forename: Paul E
authors.initials: PE
authors.lastname: Kinahan
authors.affiliation: Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA.|Department of Radiology, University of Pittsburgh, Pittsburgh, PA, USA.
authors.forename: Charles M
authors.initials: CM
authors.lastname: Laymon
authors.affiliation: Department of Radiology, University of Pittsburgh, Pittsburgh, PA, USA.
authors.forename: James M
authors.initials: JM
authors.lastname: Mountz
authors.affiliation: Department of Radiology, University of Washington Medical Center, Seattle, WA, USA.
authors.forename: John P
authors.initials: JP
authors.lastname: Muzi
authors.affiliation: Department of Radiology, University of Washington Medical Center, Seattle, WA, USA.
authors.forename: Mark
authors.initials: M
authors.lastname: Muzi
authors.affiliation: National Center for Cancer Care and Research, Doha, Qatar.
authors.forename: Sadek
authors.initials: S
authors.lastname: Nehmeh
authors.affiliation: Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA.
authors.forename: Matthew J
authors.initials: MJ
authors.lastname: Oborski
authors.affiliation: Department of Radiology, Columbia University Medical Center, New York, NY, USA.
authors.forename: Yongqiang
authors.initials: Y
authors.lastname: Tan
authors.affiliation: Department of Radiology, Columbia University Medical Center, New York, NY, USA.
authors.forename: Binsheng
authors.initials: B
authors.lastname: Zhao
authors.affiliation: Department of Radiology, The University of Iowa, Iowa City, IA, USA.
authors.forename: John J
authors.initials: JJ
authors.lastname: Sunderland
authors.affiliation: Department of Radiation Oncology, The University of Iowa, Iowa City, IA, USA.
authors.forename: John M
authors.initials: JM
authors.lastname: Buatti
mesh_terms: D066264:Datasets as Topic; D004867:Equipment Design; D019788:Fluorodeoxyglucose F18; D006258:Head and Neck Neoplasms / Q000000981:diagnostic imaging; D006801:Humans; D021621:Imaging, Three-Dimensional / Q000295:instrumentation / Q000379:methods*; D010363:Pattern Recognition, Automated / Q000379:methods; D019047:Phantoms, Imaging; D049268:Positron-Emission Tomography / Q000295:instrumentation / Q000379:methods*; D019275:Radiopharmaceuticals; D012044:Regression Analysis; D015203:Reproducibility of Results; D012984:Software; D047368:Tumor Burden
chemical_list: D019275:Radiopharmaceuticals; D019788:Fluorodeoxyglucose F18
keywords: FDG PET; head and neck cancer; multi-site performance analysis; phantom; radiomics; segmentation