title: Longitudinal multiple sclerosis lesion segmentation: Resource and challenge.
abstract: In conjunction with the ISBI 2015 conference, we organized a longitudinal lesion segmentation challenge providing training and test data to registered participants. The training data consisted of five subjects with a mean of 4.4 time-points, and test data of fourteen subjects with a mean of 4.4 time-points. All 82 data sets had the white matter lesions associated with multiple sclerosis delineated by two human expert raters. Eleven teams submitted results using state-of-the-art lesion segmentation algorithms to the challenge, with ten teams presenting their results at the conference. We present a quantitative evaluation comparing the consistency of the two raters as well as exploring the performance of the eleven submitted results in addition to three other lesion segmentation algorithms. The challenge presented three unique opportunities: (1) the sharing of a rich data set; (2) collaboration and comparison of the various avenues of research being pursued in the community; and (3) a review and refinement of the evaluation metrics currently in use. We report on the performance of the challenge participants, as well as the construction and evaluation of a consensus delineation. The image data and manual delineations will continue to be available for download, through an evaluation website2as a resource for future researchers in the area. This data resource provides a platform to compare existing methods in a fair and consistent manner to each other and multiple manual raters.
authors.affiliation: Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA; Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218, USA. Electronic address: aaron_carass@jhu.edu.
authors.forename: Aaron
authors.initials: A
authors.lastname: Carass
authors.affiliation: CNRM, The Henry M. Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD 20892, USA.
authors.forename: Snehashis
authors.initials: S
authors.lastname: Roy
authors.affiliation: Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218, USA.
authors.forename: Amod
authors.initials: A
authors.lastname: Jog
authors.affiliation: Department of Radiology, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA.
authors.forename: Jennifer L
authors.initials: JL
authors.lastname: Cuzzocreo
authors.affiliation: CNRM, The Henry M. Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD 20892, USA.
authors.forename: Elizabeth
authors.initials: E
authors.lastname: Magrath
authors.affiliation: Department of Biostatistics, The Johns Hopkins University, Baltimore, MD 21205, USA.
authors.forename: Adrian
authors.initials: A
authors.lastname: Gherman
authors.affiliation: Department of Radiology, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA.
authors.forename: Julia
authors.initials: J
authors.lastname: Button
authors.affiliation: Department of Radiology, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA.
authors.forename: James
authors.initials: J
authors.lastname: Nguyen
authors.affiliation: Translational Imaging Group, CMIC, UCL, NW1 2HE London, UK; NMR Research Unit, UCL Institute of Neurology, WC1N 3BG London, UK.
authors.forename: Ferran
authors.initials: F
authors.lastname: Prados
authors.affiliation: Translational Imaging Group, CMIC, UCL, NW1 2HE London, UK.
authors.forename: Carole H
authors.initials: CH
authors.lastname: Sudre
authors.affiliation: Translational Imaging Group, CMIC, UCL, NW1 2HE London, UK; Dementia Research Centre, UCL Institute of Neurology, WC1N 3BG London, UK.
authors.forename: Manuel
authors.initials: M
authors.lastname: Jorge Cardoso
authors.affiliation: NMR Research Unit, UCL Institute of Neurology, WC1N 3BG London, UK.
authors.forename: Niamh
authors.initials: N
authors.lastname: Cawley
authors.affiliation: NMR Research Unit, UCL Institute of Neurology, WC1N 3BG London, UK.
authors.forename: Olga
authors.initials: O
authors.lastname: Ciccarelli
authors.affiliation: NMR Research Unit, UCL Institute of Neurology, WC1N 3BG London, UK.
authors.forename: Claudia A M
authors.initials: CAM
authors.lastname: Wheeler-Kingshott
authors.affiliation: Translational Imaging Group, CMIC, UCL, NW1 2HE London, UK; Dementia Research Centre, UCL Institute of Neurology, WC1N 3BG London, UK.
authors.forename: Sébastien
authors.initials: S
authors.lastname: Ourselin
authors.affiliation: VisAGeS: INSERM U746, CNRS UMR6074, INRIA, University of Rennes I, France.
authors.forename: Laurence
authors.initials: L
authors.lastname: Catanese
authors.affiliation: VisAGeS: INSERM U746, CNRS UMR6074, INRIA, University of Rennes I, France.
authors.forename: Hrishikesh
authors.initials: H
authors.lastname: Deshpande
authors.affiliation: VisAGeS: INSERM U746, CNRS UMR6074, INRIA, University of Rennes I, France.
authors.forename: Pierre
authors.initials: P
authors.lastname: Maurel
authors.affiliation: VisAGeS: INSERM U746, CNRS UMR6074, INRIA, University of Rennes I, France.
authors.forename: Olivier
authors.initials: O
authors.lastname: Commowick
authors.affiliation: VisAGeS: INSERM U746, CNRS UMR6074, INRIA, University of Rennes I, France.
authors.forename: Christian
authors.initials: C
authors.lastname: Barillot
authors.affiliation: Computational Radiology Laboratory, Boston Childrens Hospital, Boston, MA 02115, USA; Harvard Medical School, Boston, MA 02115, USA.
authors.forename: Xavier
authors.initials: X
authors.lastname: Tomas-Fernandez
authors.affiliation: Computational Radiology Laboratory, Boston Childrens Hospital, Boston, MA 02115, USA; Harvard Medical School, Boston, MA 02115, USA.
authors.forename: Simon K
authors.initials: SK
authors.lastname: Warfield
authors.affiliation: Biomedical Imaging Lab, Department of Engineering Design, Indian Institute of Technology, Chennai 600036, India.
authors.forename: Suthirth
authors.initials: S
authors.lastname: Vaidya
authors.affiliation: Biomedical Imaging Lab, Department of Engineering Design, Indian Institute of Technology, Chennai 600036, India.
authors.forename: Abhijith
authors.initials: A
authors.lastname: Chunduru
authors.affiliation: Biomedical Imaging Lab, Department of Engineering Design, Indian Institute of Technology, Chennai 600036, India.
authors.forename: Ramanathan
authors.initials: R
authors.lastname: Muthuganapathy
authors.affiliation: Biomedical Imaging Lab, Department of Engineering Design, Indian Institute of Technology, Chennai 600036, India.
authors.forename: Ganapathy
authors.initials: G
authors.lastname: Krishnamurthi
authors.affiliation: Centre For Intelligent Machines, McGill University, Montréal, QC H3A 0E9, Canada.
authors.forename: Andrew
authors.initials: A
authors.lastname: Jesson
authors.affiliation: Centre For Intelligent Machines, McGill University, Montréal, QC H3A 0E9, Canada.
authors.forename: Tal
authors.initials: T
authors.lastname: Arbel
authors.affiliation: Institute of Medical Informatics, University of Lübeck, 23538 Lübeck, Germany.
authors.forename: Oskar
authors.initials: O
authors.lastname: Maier
authors.affiliation: Institute of Medical Informatics, University of Lübeck, 23538 Lübeck, Germany.
authors.forename: Heinz
authors.initials: H
authors.lastname: Handels
authors.affiliation: Bahçeşehir University, Faculty of Engineering and Natural Sciences, 34349 Beşiktaş, Turkey.
authors.forename: Leonardo O
authors.initials: LO
authors.lastname: Iheme
authors.affiliation: Bahçeşehir University, Faculty of Engineering and Natural Sciences, 34349 Beşiktaş, Turkey.
authors.forename: Devrim
authors.initials: D
authors.lastname: Unay
authors.affiliation: icometrix, 3012 Leuven, Belgium.
authors.forename: Saurabh
authors.initials: S
authors.lastname: Jain
authors.affiliation: icometrix, 3012 Leuven, Belgium.
authors.forename: Diana M
authors.initials: DM
authors.lastname: Sima
authors.affiliation: icometrix, 3012 Leuven, Belgium.
authors.forename: Dirk
authors.initials: D
authors.lastname: Smeets
authors.affiliation: Institute for Computing and Information Sciences, Radboud University, 6525 HP Nijmegen, Netherlands.
authors.forename: Mohsen
authors.initials: M
authors.lastname: Ghafoorian
authors.affiliation: Diagnostic Image Analysis Group, Radboud University Medical Center, 6525 GA Nijmegen, Netherlands.
authors.forename: Bram
authors.initials: B
authors.lastname: Platel
authors.affiliation: Department of Electrical Engineering, Tel-Aviv University, Tel-Aviv 69978, Israel.
authors.forename: Ariel
authors.initials: A
authors.lastname: Birenbaum
authors.affiliation: Department of Biomedical Engineering, Tel-Aviv University, Tel-Aviv 69978, Israel.
authors.forename: Hayit
authors.initials: H
authors.lastname: Greenspan
authors.affiliation: Department of Neurophysics, Max Planck Institute, 04103 Leipzig, Germany.
authors.forename: Pierre-Louis
authors.initials: PL
authors.lastname: Bazin
authors.affiliation: Department of Radiology, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA.
authors.forename: Peter A
authors.initials: PA
authors.lastname: Calabresi
authors.affiliation: Department of Biostatistics, The Johns Hopkins University, Baltimore, MD 21205, USA.
authors.forename: Ciprian M
authors.initials: CM
authors.lastname: Crainiceanu
authors.affiliation: Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA; Department of Electrical and Computer Engineering, University of Iceland, 107 Reykjavík, Iceland.
authors.forename: Lotta M
authors.initials: LM
authors.lastname: Ellingsen
authors.affiliation: Department of Radiology, The Johns Hopkins School of Medicine, Baltimore, MD 21287, USA; Translational Neuroradiology Unit, National Institute of Neurological Disorders and Stroke, Bethesda, MD 20892, USA.
authors.forename: Daniel S
authors.initials: DS
authors.lastname: Reich
authors.affiliation: Department of Electrical and Computer Engineering, The Johns Hopkins University, Baltimore, MD 21218, USA; Department of Computer Science, The Johns Hopkins University, Baltimore, MD 21218, USA.
authors.forename: Jerry L
authors.initials: JL
authors.lastname: Prince
authors.affiliation: CNRM, The Henry M. Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD 20892, USA.
authors.forename: Dzung L
authors.initials: DL
authors.lastname: Pham
mesh_terms: D000328:Adult; D000465:Algorithms; D005260:Female; D006801:Humans; D007091:Image Processing, Computer-Assisted; D021621:Imaging, Three-Dimensional; D008137:Longitudinal Studies; D008279:Magnetic Resonance Imaging; D008297:Male; D008875:Middle Aged; D009103:Multiple Sclerosis / Q000000981:diagnostic imaging*; D015588:Observer Variation; D066127:White Matter / Q000000981:diagnostic imaging
keywords: Magnetic resonance imaging; Multiple sclerosis