title: The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS).
abstract: In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients-manually annotated by up to four raters-and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.
authors.forename: Bjoern H
authors.initials: BH
authors.lastname: Menze
authors.forename: Andras
authors.initials: A
authors.lastname: Jakab
authors.forename: Stefan
authors.initials: S
authors.lastname: Bauer
authors.forename: Jayashree
authors.initials: J
authors.lastname: Kalpathy-Cramer
authors.forename: Keyvan
authors.initials: K
authors.lastname: Farahani
authors.forename: Justin
authors.initials: J
authors.lastname: Kirby
authors.forename: Yuliya
authors.initials: Y
authors.lastname: Burren
authors.forename: Nicole
authors.initials: N
authors.lastname: Porz
authors.forename: Johannes
authors.initials: J
authors.lastname: Slotboom
authors.forename: Roland
authors.initials: R
authors.lastname: Wiest
authors.forename: Levente
authors.initials: L
authors.lastname: Lanczi
authors.forename: Elizabeth
authors.initials: E
authors.lastname: Gerstner
authors.forename: Marc-André
authors.initials: MA
authors.lastname: Weber
authors.forename: Tal
authors.initials: T
authors.lastname: Arbel
authors.forename: Brian B
authors.initials: BB
authors.lastname: Avants
authors.forename: Nicholas
authors.initials: N
authors.lastname: Ayache
authors.forename: Patricia
authors.initials: P
authors.lastname: Buendia
authors.forename: D Louis
authors.initials: DL
authors.lastname: Collins
authors.forename: Nicolas
authors.initials: N
authors.lastname: Cordier
authors.forename: Jason J
authors.initials: JJ
authors.lastname: Corso
authors.forename: Antonio
authors.initials: A
authors.lastname: Criminisi
authors.forename: Tilak
authors.initials: T
authors.lastname: Das
authors.forename: Hervé
authors.initials: H
authors.lastname: Delingette
authors.forename: Çağatay
authors.initials: Ç
authors.lastname: Demiralp
authors.forename: Christopher R
authors.initials: CR
authors.lastname: Durst
authors.forename: Michel
authors.initials: M
authors.lastname: Dojat
authors.forename: Senan
authors.initials: S
authors.lastname: Doyle
authors.forename: Joana
authors.initials: J
authors.lastname: Festa
authors.forename: Florence
authors.initials: F
authors.lastname: Forbes
authors.forename: Ezequiel
authors.initials: E
authors.lastname: Geremia
authors.forename: Ben
authors.initials: B
authors.lastname: Glocker
authors.forename: Polina
authors.initials: P
authors.lastname: Golland
authors.forename: Xiaotao
authors.initials: X
authors.lastname: Guo
authors.forename: Andac
authors.initials: A
authors.lastname: Hamamci
authors.forename: Khan M
authors.initials: KM
authors.lastname: Iftekharuddin
authors.forename: Raj
authors.initials: R
authors.lastname: Jena
authors.forename: Nigel M
authors.initials: NM
authors.lastname: John
authors.forename: Ender
authors.initials: E
authors.lastname: Konukoglu
authors.forename: Danial
authors.initials: D
authors.lastname: Lashkari
authors.forename: José Antonió
authors.initials: JA
authors.lastname: Mariz
authors.forename: Raphael
authors.initials: R
authors.lastname: Meier
authors.forename: Sérgio
authors.initials: S
authors.lastname: Pereira
authors.forename: Doina
authors.initials: D
authors.lastname: Precup
authors.forename: Stephen J
authors.initials: SJ
authors.lastname: Price
authors.forename: Tammy Riklin
authors.initials: TR
authors.lastname: Raviv
authors.forename: Syed M S
authors.initials: SM
authors.lastname: Reza
authors.forename: Michael
authors.initials: M
authors.lastname: Ryan
authors.forename: Duygu
authors.initials: D
authors.lastname: Sarikaya
authors.forename: Lawrence
authors.initials: L
authors.lastname: Schwartz
authors.forename: Hoo-Chang
authors.initials: HC
authors.lastname: Shin
authors.forename: Jamie
authors.initials: J
authors.lastname: Shotton
authors.forename: Carlos A
authors.initials: CA
authors.lastname: Silva
authors.forename: Nuno
authors.initials: N
authors.lastname: Sousa
authors.forename: Nagesh K
authors.initials: NK
authors.lastname: Subbanna
authors.forename: Gabor
authors.initials: G
authors.lastname: Szekely
authors.forename: Thomas J
authors.initials: TJ
authors.lastname: Taylor
authors.forename: Owen M
authors.initials: OM
authors.lastname: Thomas
authors.forename: Nicholas J
authors.initials: NJ
authors.lastname: Tustison
authors.forename: Gozde
authors.initials: G
authors.lastname: Unal
authors.forename: Flor
authors.initials: F
authors.lastname: Vasseur
authors.forename: Max
authors.initials: M
authors.lastname: Wintermark
authors.forename: Dong Hye
authors.initials: DH
authors.lastname: Ye
authors.forename: Liang
authors.initials: L
authors.lastname: Zhao
authors.forename: Binsheng
authors.initials: B
authors.lastname: Zhao
authors.forename: Darko
authors.initials: D
authors.lastname: Zikic
authors.forename: Marcel
authors.initials: M
authors.lastname: Prastawa
authors.forename: Mauricio
authors.initials: M
authors.lastname: Reyes
authors.forename: Koen
authors.initials: K
authors.lastname: Van Leemput
mesh_terms: D000465:Algorithms; D019985:Benchmarking; D005910:Glioma / Q000473:pathology; D006801:Humans; D008279:Magnetic Resonance Imaging / Q000379:methods / Q000592:standards; D059906:Neuroimaging / Q000379:methods / Q000592:standards