title: Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge.
abstract: Prostate MRI image segmentation has been an area of intense research due to the increased use of MRI as a modality for the clinical workup of prostate cancer. Segmentation is useful for various tasks, e.g. to accurately localize prostate boundaries for radiotherapy or to initialize multi-modal registration algorithms. In the past, it has been difficult for research groups to evaluate prostate segmentation algorithms on multi-center, multi-vendor and multi-protocol data. Especially because we are dealing with MR images, image appearance, resolution and the presence of artifacts are affected by differences in scanners and/or protocols, which in turn can have a large influence on algorithm accuracy. The Prostate MR Image Segmentation (PROMISE12) challenge was setup to allow a fair and meaningful comparison of segmentation methods on the basis of performance and robustness. In this work we will discuss the initial results of the online PROMISE12 challenge, and the results obtained in the live challenge workshop hosted by the MICCAI2012 conference. In the challenge, 100 prostate MR cases from 4 different centers were included, with differences in scanner manufacturer, field strength and protocol. A total of 11 teams from academic research groups and industry participated. Algorithms showed a wide variety in methods and implementation, including active appearance models, atlas registration and level sets. Evaluation was performed using boundary and volume based metrics which were combined into a single score relating the metrics to human expert performance. The winners of the challenge where the algorithms by teams Imorphics and ScrAutoProstate, with scores of 85.72 and 84.29 overall. Both algorithms where significantly better than all other algorithms in the challenge (p<0.05) and had an efficient implementation with a run time of 8min and 3s per case respectively. Overall, active appearance model based approaches seemed to outperform other approaches like multi-atlas registration, both on accuracy and computation time. Although average algorithm performance was good to excellent and the Imorphics algorithm outperformed the second observer on average, we showed that algorithm combination might lead to further improvement, indicating that optimal performance for prostate segmentation is not yet obtained. All results are available online at http://promise12.grand-challenge.org/.
authors.affiliation: Radboud University Nijmegen Medical Centre, The Netherlands. Electronic address: g.litjens@rad.umcn.nl.
authors.forename: Geert
authors.initials: G
authors.lastname: Litjens
authors.affiliation: Case Western Reserve University, USA.
authors.forename: Robert
authors.initials: R
authors.lastname: Toth
authors.affiliation: Radboud University Nijmegen Medical Centre, The Netherlands.
authors.forename: Wendy
authors.initials: W
authors.lastname: van de Ven
authors.affiliation: Radboud University Nijmegen Medical Centre, The Netherlands.
authors.forename: Caroline
authors.initials: C
authors.lastname: Hoeks
authors.affiliation: Radboud University Nijmegen Medical Centre, The Netherlands.
authors.forename: Sjoerd
authors.initials: S
authors.lastname: Kerkstra
authors.affiliation: Radboud University Nijmegen Medical Centre, The Netherlands.
authors.forename: Bram
authors.initials: B
authors.lastname: van Ginneken
authors.affiliation: Imorphics, England, United Kingdom.
authors.forename: Graham
authors.initials: G
authors.lastname: Vincent
authors.affiliation: Imorphics, England, United Kingdom.
authors.forename: Gwenael
authors.initials: G
authors.lastname: Guillard
authors.affiliation: Siemens Corporate Research, USA.
authors.forename: Neil
authors.initials: N
authors.lastname: Birbeck
authors.affiliation: Siemens Corporate Research, USA.
authors.forename: Jindang
authors.initials: J
authors.lastname: Zhang
authors.affiliation: Uppsala University, Sweden.
authors.forename: Robin
authors.initials: R
authors.lastname: Strand
authors.affiliation: Uppsala University, Sweden.
authors.forename: Filip
authors.initials: F
authors.lastname: Malmberg
authors.affiliation: University of Pennsylvania, USA.
authors.forename: Yangming
authors.initials: Y
authors.lastname: Ou
authors.affiliation: University of Pennsylvania, USA.
authors.forename: Christos
authors.initials: C
authors.lastname: Davatzikos
authors.affiliation: Technische Universitat Darmstadt, Germany.
authors.forename: Matthias
authors.initials: M
authors.lastname: Kirschner
authors.affiliation: Technische Universitat Darmstadt, Germany.
authors.forename: Florian
authors.initials: F
authors.lastname: Jung
authors.affiliation: Robarts Research Institute, Canada.
authors.forename: Jing
authors.initials: J
authors.lastname: Yuan
authors.affiliation: Robarts Research Institute, Canada.
authors.forename: Wu
authors.initials: W
authors.lastname: Qiu
authors.affiliation: Imperial College London, England, United Kingdom.
authors.forename: Qinquan
authors.initials: Q
authors.lastname: Gao
authors.affiliation: Imperial College London, England, United Kingdom.
authors.forename: Philip Eddie
authors.initials: PE
authors.lastname: Edwards
authors.affiliation: University of Twente, The Netherlands.
authors.forename: Bianca
authors.initials: B
authors.lastname: Maan
authors.affiliation: University of Twente, The Netherlands.
authors.forename: Ferdinand
authors.initials: F
authors.lastname: van der Heijden
authors.affiliation: Commonwealth Scientific and Industrial Research Organisation, Australia; Université de Bourgogne, France; Universitat de Girona, Spain.
authors.forename: Soumya
authors.initials: S
authors.lastname: Ghose
authors.affiliation: Commonwealth Scientific and Industrial Research Organisation, Australia; Université de Bourgogne, France; Universitat de Girona, Spain.
authors.forename: Jhimli
authors.initials: J
authors.lastname: Mitra
authors.affiliation: Commonwealth Scientific and Industrial Research Organisation, Australia.
authors.forename: Jason
authors.initials: J
authors.lastname: Dowling
authors.affiliation: University College London, England, United Kingdom.
authors.forename: Dean
authors.initials: D
authors.lastname: Barratt
authors.affiliation: Radboud University Nijmegen Medical Centre, The Netherlands.
authors.forename: Henkjan
authors.initials: H
authors.lastname: Huisman
authors.affiliation: Case Western Reserve University, USA.
authors.forename: Anant
authors.initials: A
authors.lastname: Madabhushi
mesh_terms: D000465:Algorithms; D016477:Artifacts; D006801:Humans; D007089:Image Enhancement / Q000379:methods*; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D021621:Imaging, Three-Dimensional; D008279:Magnetic Resonance Imaging / Q000592:standards*; D008297:Male; D011471:Prostatic Neoplasms / Q000532:radiotherapy*; D012015:Reference Standards; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity
keywords: Challenge; MRI; Prostate; Segmentation