title: ISLES 2015 - A public evaluation benchmark for ischemic stroke lesion segmentation from multispectral MRI.
abstract: Ischemic stroke is the most common cerebrovascular disease, and its diagnosis, treatment, and study relies on non-invasive imaging. Algorithms for stroke lesion segmentation from magnetic resonance imaging (MRI) volumes are intensely researched, but the reported results are largely incomparable due to different datasets and evaluation schemes. We approached this urgent problem of comparability with the Ischemic Stroke Lesion Segmentation (ISLES) challenge organized in conjunction with the MICCAI 2015 conference. In this paper we propose a common evaluation framework, describe the publicly available datasets, and present the results of the two sub-challenges: Sub-Acute Stroke Lesion Segmentation (SISS) and Stroke Perfusion Estimation (SPES). A total of 16 research groups participated with a wide range of state-of-the-art automatic segmentation algorithms. A thorough analysis of the obtained data enables a critical evaluation of the current state-of-the-art, recommendations for further developments, and the identification of remaining challenges. The segmentation of acute perfusion lesions addressed in SPES was found to be feasible. However, algorithms applied to sub-acute lesion segmentation in SISS still lack accuracy. Overall, no algorithmic characteristic of any method was found to perform superior to the others. Instead, the characteristics of stroke lesion appearances, their evolution, and the observed challenges should be studied in detail. The annotated ISLES image datasets continue to be publicly available through an online evaluation system to serve as an ongoing benchmarking resource (www.isles-challenge.org).
authors.affiliation: Institut for Medical Informatics, University of Lübeck, Lübeck, Germany.|Graduate School for Computing in Medicine and Live Science, University of Lübeck, Germany.
authors.forename: Oskar
authors.initials: O
authors.lastname: Maier
authors.affiliation: Institute for Advanced Study and Department of Computer Science, Technische Universität München, Munich, Germany.
authors.forename: Bjoern H
authors.initials: BH
authors.lastname: Menze
authors.affiliation: Department of Neurology, University of Lübeck, Germany.
authors.forename: Janina
authors.initials: J
authors.lastname: von der Gablentz
authors.affiliation: Institute for Surgical Technology and Biomechanics, University of Bern, Bern, Switzerland.
authors.forename: Levin
authors.initials: L
authors.lastname: Ḧani
authors.affiliation: Institut for Medical Informatics, University of Lübeck, Lübeck, Germany.
authors.forename: Mattias P
authors.initials: MP
authors.lastname: Heinrich
authors.affiliation: Department of Neurology, University of Lübeck, Germany.
authors.forename: Matthias
authors.initials: M
authors.lastname: Liebrand
authors.affiliation: Institute for Advanced Study and Department of Computer Science, Technische Universität München, Munich, Germany.
authors.forename: Stefan
authors.initials: S
authors.lastname: Winzeck
authors.affiliation: Pakistan Institute of Nuclear Science and Technology, Islamabad, Pakistan.
authors.forename: Abdul
authors.initials: A
authors.lastname: Basit
authors.affiliation: Division of Brain Sciences, Department of Medicine, Imperial College London, UK.
authors.forename: Paul
authors.initials: P
authors.lastname: Bentley
authors.affiliation: Biomedical Image Analysis Group, Department of Computing, Imperial College London, UK.|Division of Brain Sciences, Department of Medicine, Imperial College London, UK.
authors.forename: Liang
authors.initials: L
authors.lastname: Chen
authors.affiliation: ESAT/PSI, Department of Electrical Engineering, KU Leuven, Belgium.|Medical Imaging Research Center, UZ Leuven, Belgium.
authors.forename: Daan
authors.initials: D
authors.lastname: Christiaens
authors.affiliation: Université de Sherbrooke, Sherbrooke, Qc, Canada.
authors.forename: Francis
authors.initials: F
authors.lastname: Dutil
authors.affiliation: Department of Neuroradiology, University Medical Center Freiburg, Germany.
authors.forename: Karl
authors.initials: K
authors.lastname: Egger
authors.affiliation: College of Information Science and Engineering, Northeastern University, Shenyang, Liaoning, China.
authors.forename: Chaolu
authors.initials: C
authors.lastname: Feng
authors.affiliation: Biomedical Image Analysis Group, Department of Computing, Imperial College London, UK.
authors.forename: Ben
authors.initials: B
authors.lastname: Glocker
authors.affiliation: Junior Group Medical Image Computing, German Cancer Research Center, Heidelberg, Germany.
authors.forename: Michael
authors.initials: M
authors.lastname: Götz
authors.affiliation: ESAT/PSI, Department of Electrical Engineering, KU Leuven, Belgium.|Medical Imaging Research Center, UZ Leuven, Belgium.
authors.forename: Tom
authors.initials: T
authors.lastname: Haeck
authors.affiliation: HUS Medical Imaging Center, Radiology, University of Helsinki and Helsinki University Hospital, Helsinki, Finland.|Department of Neuroscience and Biomedical Engineering NBE, Aalto University School of Science, Aalto, Finland.
authors.forename: Hanna-Leena
authors.initials: HL
authors.lastname: Halme
authors.affiliation: Université de Sherbrooke, Sherbrooke, Qc, Canada.
authors.forename: Mohammad
authors.initials: M
authors.lastname: Havaei
authors.affiliation: Vision Lab, Department of Electrical and Computer Engineering, Old Dominion University, Norfolk, VA, USA.
authors.forename: Khan M
authors.initials: KM
authors.lastname: Iftekharuddin
authors.affiliation: Université de Sherbrooke, Sherbrooke, Qc, Canada.
authors.forename: Pierre-Marc
authors.initials: PM
authors.lastname: Jodoin
authors.affiliation: Biomedical Image Analysis Group, Department of Computing, Imperial College London, UK.
authors.forename: Konstantinos
authors.initials: K
authors.lastname: Kamnitsas
authors.affiliation: Department of Radiology, Medical Physics, University Medical Center Freiburg, Germany.
authors.forename: Elias
authors.initials: E
authors.lastname: Kellner
authors.affiliation: HUS Medical Imaging Center, Radiology, University of Helsinki and Helsinki University Hospital, Helsinki, Finland.
authors.forename: Antti
authors.initials: A
authors.lastname: Korvenoja
authors.affiliation: Université de Sherbrooke, Sherbrooke, Qc, Canada.
authors.forename: Hugo
authors.initials: H
authors.lastname: Larochelle
authors.affiliation: Biomedical Image Analysis Group, Department of Computing, Imperial College London, UK.
authors.forename: Christian
authors.initials: C
authors.lastname: Ledig
authors.affiliation: Graduate Institute of Biomedical Engineering, National Taiwan University of Science and Technology, Taipei City, Taiwan.
authors.forename: Jia-Hong
authors.initials: JH
authors.lastname: Lee
authors.affiliation: ESAT/PSI, Department of Electrical Engineering, KU Leuven, Belgium.|Medical Imaging Research Center, UZ Leuven, Belgium.
authors.forename: Frederik
authors.initials: F
authors.lastname: Maes
authors.affiliation: Signals and Systems, Chalmers University of Technology, Gothenburg, Sweden.|Pakistan Institute of Nuclear Science and Technology, Islamabad, Pakistan.
authors.forename: Qaiser
authors.initials: Q
authors.lastname: Mahmood
authors.affiliation: Junior Group Medical Image Computing, German Cancer Research Center, Heidelberg, Germany.
authors.forename: Klaus H
authors.initials: KH
authors.lastname: Maier-Hein
authors.affiliation: Department of Diagnostic and Interventional Neuroradiology, Inselspital Bern, Switzerland.
authors.forename: Richard
authors.initials: R
authors.lastname: McKinley
authors.affiliation: Department of Biostatistics, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA.
authors.forename: John
authors.initials: J
authors.lastname: Muschelli
authors.affiliation: Ecole Polytechnique de Montréal, Canada.
authors.forename: Chris
authors.initials: C
authors.lastname: Pal
authors.affiliation: Vision Lab, Department of Electrical and Computer Engineering, Old Dominion University, Norfolk, VA, USA.
authors.forename: Linmin
authors.initials: L
authors.lastname: Pei
authors.affiliation: ESAT/PSI, Department of Electrical Engineering, KU Leuven, Belgium.|Medical Imaging Research Center, UZ Leuven, Belgium.
authors.forename: Janaki Raman
authors.initials: JR
authors.lastname: Rangarajan
authors.affiliation: Vision Lab, Department of Electrical and Computer Engineering, Old Dominion University, Norfolk, VA, USA.
authors.forename: Syed M S
authors.initials: SMS
authors.lastname: Reza
authors.affiliation: ESAT/PSI, Department of Electrical Engineering, KU Leuven, Belgium.|Medical Imaging Research Center, UZ Leuven, Belgium.
authors.forename: David
authors.initials: D
authors.lastname: Robben
authors.affiliation: Biomedical Image Analysis Group, Department of Computing, Imperial College London, UK.
authors.forename: Daniel
authors.initials: D
authors.lastname: Rueckert
authors.affiliation: HUS Medical Imaging Center, Radiology, University of Helsinki and Helsinki University Hospital, Helsinki, Finland.
authors.forename: Eero
authors.initials: E
authors.lastname: Salli
authors.affiliation: ESAT/PSI, Department of Electrical Engineering, KU Leuven, Belgium.|Medical Imaging Research Center, UZ Leuven, Belgium.
authors.forename: Paul
authors.initials: P
authors.lastname: Suetens
authors.affiliation: Graduate Institute of Biomedical Engineering, National Taiwan University of Science and Technology, Taipei City, Taiwan.
authors.forename: Ching-Wei
authors.initials: CW
authors.lastname: Wang
authors.affiliation: Institut for Medical Informatics, University of Lübeck, Lübeck, Germany.
authors.forename: Matthias
authors.initials: M
authors.lastname: Wilms
authors.affiliation: Department of Neuroradiology, Klinikum rechts der Isar, Technische Universität München, Munich, Germany.
authors.forename: Jan S
authors.initials: JS
authors.lastname: Kirschke
authors.affiliation: Department of Neurology, University of Lübeck, Germany.|Institute of Psychology II, University of Lübeck, Germany.
authors.forename: Ulrike M
authors.initials: UM
authors.lastname: Kr Amer
authors.affiliation: Department of Neurology, University of Lübeck, Germany.
authors.forename: Thomas F
authors.initials: TF
authors.lastname: Münte
authors.affiliation: Institute of Neuroradiology, University Medical Center Lübeck.
authors.forename: Peter
authors.initials: P
authors.lastname: Schramm
authors.affiliation: Department of Diagnostic and Interventional Neuroradiology, Inselspital Bern, Switzerland.
authors.forename: Roland
authors.initials: R
authors.lastname: Wiest
authors.affiliation: Institut for Medical Informatics, University of Lübeck, Lübeck, Germany.
authors.forename: Heinz
authors.initials: H
authors.lastname: Handels
authors.affiliation: Institute for Surgical Technology and Biomechanics, University of Bern, Bern, Switzerland.
authors.forename: Mauricio
authors.initials: M
authors.lastname: Reyes
mesh_terms: D000465:Algorithms; D019985:Benchmarking; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008279:Magnetic Resonance Imaging / Q000379:methods*; D020521:Stroke / Q000000981:diagnostic imaging*
keywords: Benchmark; Challenge; Comparison; Ischemic stroke; MRI; Segmentation