title: Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: the CADDementia challenge.
abstract: Algorithms for computer-aided diagnosis of dementia based on structural MRI have demonstrated high performance in the literature, but are difficult to compare as different data sets and methodology were used for evaluation. In addition, it is unclear how the algorithms would perform on previously unseen data, and thus, how they would perform in clinical practice when there is no real opportunity to adapt the algorithm to the data at hand. To address these comparability, generalizability and clinical applicability issues, we organized a grand challenge that aimed to objectively compare algorithms based on a clinically representative multi-center data set. Using clinical practice as the starting point, the goal was to reproduce the clinical diagnosis. Therefore, we evaluated algorithms for multi-class classification of three diagnostic groups: patients with probable Alzheimer's disease, patients with mild cognitive impairment and healthy controls. The diagnosis based on clinical criteria was used as reference standard, as it was the best available reference despite its known limitations. For evaluation, a previously unseen test set was used consisting of 354 T1-weighted MRI scans with the diagnoses blinded. Fifteen research teams participated with a total of 29 algorithms. The algorithms were trained on a small training set (n=30) and optionally on data from other sources (e.g., the Alzheimer's Disease Neuroimaging Initiative, the Australian Imaging Biomarkers and Lifestyle flagship study of aging). The best performing algorithm yielded an accuracy of 63.0% and an area under the receiver-operating-characteristic curve (AUC) of 78.8%. In general, the best performances were achieved using feature extraction based on voxel-based morphometry or a combination of features that included volume, cortical thickness, shape and intensity. The challenge is open for new submissions via the web-based framework: http://caddementia.grand-challenge.org.
authors: Biomedical Imaging Group Rotterdam, Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands; Biomedical Imaging Group Rotterdam, Department of Radiology, Erasmus MC, Rotterdam, The Netherlands. Electronic address: caddementia@bigr.nl.
authors: Esther E
authors: EE
authors: Bron
authors: Department of Radiology, Erasmus MC, Rotterdam, The Netherlands.
authors: Marion
authors: M
authors: Smits
authors: Alzheimer Center, Department of Neurology, VU University Medical Center, Neuroscience Campus Amsterdam, The Netherlands; Department of Epidemiology & Biostatistics, VU University Medical Center, Neuroscience Campus Amsterdam, The Netherlands.
authors: Wiesje M
authors: WM
authors: van der Flier
authors: Department of Radiology & Nuclear Medicine, VU University Medical Center, Neuroscience Campus Amsterdam, The Netherlands.
authors: Hugo
authors: H
authors: Vrenken
authors: Department of Radiology & Nuclear Medicine, VU University Medical Center, Neuroscience Campus Amsterdam, The Netherlands.
authors: Frederik
authors: F
authors: Barkhof
authors: Alzheimer Center, Department of Neurology, VU University Medical Center, Neuroscience Campus Amsterdam, The Netherlands.
authors: Philip
authors: P
authors: Scheltens
authors: Department of Neurology, Erasmus MC, Rotterdam, The Netherlands; Department of Radiology, Erasmus MC, Rotterdam, The Netherlands.
authors: Janne M
authors: JM
authors: Papma
authors: Department of Radiology, Erasmus MC, Rotterdam, The Netherlands.
authors: Rebecca M E
authors: RM
authors: Steketee
authors: Department of Radiology, Erasmus MC, Rotterdam, The Netherlands; Department of Neurology, Erasmus MC, Rotterdam, The Netherlands.
authors: Carolina
authors: C
authors: Méndez Orellana
authors: Department of Radiology, Erasmus MC, Rotterdam, The Netherlands.
authors: Rozanna
authors: R
authors: Meijboom
authors: Department of Neurology, Hospital de São João, Porto, Portugal.
authors: Madalena
authors: M
authors: Pinto
authors: Department of Neurology, Hospital de São João, Porto, Portugal.
authors: Joana R
authors: JR
authors: Meireles
authors: Department of Neurology, Hospital de São João, Porto, Portugal; Department of Clinical Neurosciences and Mental Health, Faculty of Medicine, University of Porto, Porto, Portugal.
authors: Carolina
authors: C
authors: Garrett
authors: Department of Medical Imaging, Faculty of Medicine, University of Porto, Porto, Portugal.
authors: António J
authors: AJ
authors: Bastos-Leite
authors: Department of Psychiatry & Psychotherapy, University Medical Centre Freiburg, Germany; Department of Neurology, University Medical Centre Freiburg, Germany; Department of Computer Science, University of Freiburg, Germany.
authors: Ahmed
authors: A
authors: Abdulkadir
authors: BIOSS Centre for Biological Signaling Studies, University of Freiburg, Germany; Department of Computer Science, University of Freiburg, Germany.
authors: Olaf
authors: O
authors: Ronneberger
authors: National Institute of Nuclear Physics, Branch of Bari, Italy; Department of Physics, University of Bari, Italy.
authors: Nicola
authors: N
authors: Amoroso
authors: National Institute of Nuclear Physics, Branch of Bari, Italy; Department of Physics, University of Bari, Italy.
authors: Roberto
authors: R
authors: Bellotti
authors: Signal Processing and Recognition Group, Universidad Nacional de Colombia, Colombia.
authors: David
authors: D
authors: Cárdenas-Peña
authors: Signal Processing and Recognition Group, Universidad Nacional de Colombia, Colombia.
authors: Andrés M
authors: AM
authors: Álvarez-Meza
authors: Vision Lab, Old Dominion University, Norfolk, VA 23529, USA.
authors: Chester V
authors: CV
authors: Dolph
authors: Vision Lab, Old Dominion University, Norfolk, VA 23529, USA.
authors: Khan M
authors: KM
authors: Iftekharuddin
authors: Center of Functionally Integrative Neuroscience and MINDLab, Aarhus University, Aarhus, Denmark.
authors: Simon F
authors: SF
authors: Eskildsen
authors: Laboratoire Bordelais de Recherche en Informatique, Unit Mixte de Recherche CNRS (UMR 5800), PICTURA Research Group, Bordeaux, France.
authors: Pierrick
authors: P
authors: Coupé
authors: McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, Montreal, Canada.
authors: Vladimir S
authors: VS
authors: Fonov
authors: Structural Brain Mapping Group, Department of Neurology, Jena University Hospital, Germany; Structural Brain Mapping Group, Department of Psychiatry, Jena University Hospital, Germany.
authors: Katja
authors: K
authors: Franke
authors: Structural Brain Mapping Group, Department of Neurology, Jena University Hospital, Germany; Structural Brain Mapping Group, Department of Psychiatry, Jena University Hospital, Germany.
authors: Christian
authors: C
authors: Gaser
authors: Biomedical Image Analysis (BioMedIA) Group, Department of Computing, Imperial College London, UK.
authors: Christian
authors: C
authors: Ledig
authors: Biomedical Image Analysis (BioMedIA) Group, Department of Computing, Imperial College London, UK.
authors: Ricardo
authors: R
authors: Guerrero
authors: Biomedical Image Analysis (BioMedIA) Group, Department of Computing, Imperial College London, UK.
authors: Tong
authors: T
authors: Tong
authors: Biomedical Image Analysis (BioMedIA) Group, Department of Computing, Imperial College London, UK.
authors: Katherine R
authors: KR
authors: Gray
authors: Department of Signal Processing, Tampere University of Technology, Finland.
authors: Elaheh
authors: E
authors: Moradi
authors: Department of Signal Processing, Tampere University of Technology, Finland.
authors: Jussi
authors: J
authors: Tohka
authors: Inserm U1127, CNRS UMR 7225, Sorbonne Universités, UPMC Univ Paris 06 UMR S 1127, Institut du Cerveau et de la Moelle épinière, ICM, Inria Paris-Rocquencourt, F-75013 Paris, France; Centre d'Acquisition et de Traitement des Images (CATI), Paris, France.
authors: Alexandre
authors: A
authors: Routier
authors: Inserm U1127, CNRS UMR 7225, Sorbonne Universités, UPMC Univ Paris 06 UMR S 1127, Institut du Cerveau et de la Moelle épinière, ICM, Inria Paris-Rocquencourt, F-75013 Paris, France; Centre d'Acquisition et de Traitement des Images (CATI), Paris, France.
authors: Stanley
authors: S
authors: Durrleman
authors: Bioinformatics Laboratory, Department of Medical and Surgical Sciences, Magna Graecia University, Catanzaro, Italy.
authors: Alessia
authors: A
authors: Sarica
authors: School of Systems Engineering, University of Reading, Reading RG6 6AY, UK.
authors: Giuseppe
authors: G
authors: Di Fatta
authors: National Institute of Nuclear Physics, Branch of Genoa, Italy.
authors: Francesco
authors: F
authors: Sensi
authors: National Institute of Nuclear Physics, Branch of Genoa, Italy.
authors: Andrea
authors: A
authors: Chincarini
authors: Centre for Integrative Neuroscience and Neurodynamics, University of Reading, RG6 6AH, UK; School of Systems Engineering, University of Reading, Reading RG6 6AY, UK.
authors: Garry M
authors: GM
authors: Smith
authors: Centre for Integrative Neuroscience and Neurodynamics, University of Reading, RG6 6AH, UK; School of Systems Engineering, University of Reading, Reading RG6 6AY, UK.
authors: Zhivko V
authors: ZV
authors: Stoyanov
authors: Department of Computer Science, University of Copenhagen, Denmark.
authors: Lauge
authors: L
authors: Sørensen
authors: Department of Computer Science, University of Copenhagen, Denmark.
authors: Mads
authors: M
authors: Nielsen
authors: National Institute of Nuclear Physics, Branch of Bari, Italy.
authors: Sabina
authors: S
authors: Tangaro
authors: National Institute of Nuclear Physics, Branch of Bari, Italy.
authors: Paolo
authors: P
authors: Inglese
authors: Computer Science and Artificial Intelligence Lab, MA Institute of Technology, Cambridge, USA; Massachusetts General Hospital, Harvard Medical School, Cambridge, USA.
authors: Christian
authors: C
authors: Wachinger
authors: Computer Science and Artificial Intelligence Lab, MA Institute of Technology, Cambridge, USA; Massachusetts General Hospital, Harvard Medical School, Cambridge, USA.
authors: Martin
authors: M
authors: Reuter
authors: Department of Neurology, Erasmus MC, Rotterdam, The Netherlands.
authors: John C
authors: JC
authors: van Swieten
authors: Biomedical Imaging Group Rotterdam, Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands; Biomedical Imaging Group Rotterdam, Department of Radiology, Erasmus MC, Rotterdam, The Netherlands; Imaging Physics, Applied Sciences, Delft University of Technology, The Netherlands.
authors: Wiro J
authors: WJ
authors: Niessen
authors: Biomedical Imaging Group Rotterdam, Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands; Biomedical Imaging Group Rotterdam, Department of Radiology, Erasmus MC, Rotterdam, The Netherlands.
authors: Stefan
authors: S
authors: Klein
mesh_terms: D000368:Aged; D000369:Aged, 80 and over; D000465:Algorithms; D000544:Alzheimer Disease / Q000145:classification / Q000175:diagnosis*; D060825:Cognitive Dysfunction / Q000145:classification / Q000175:diagnosis*; D003936:Diagnosis, Computer-Assisted / Q000379:methods* / Q000592:standards; D005260:Female; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods* / Q000592:standards; D008279:Magnetic Resonance Imaging / Q000379:methods* / Q000592:standards; D008297:Male; D008875:Middle Aged; D012680:Sensitivity and Specificity
keywords: Alzheimer's disease; Challenge; Classification; Computer-aided diagnosis; Mild cognitive impairment; Structural MRI
references: Alzheimers Dement. 2014 Mar;10(2):e47-92
references: 24818261
references: Alzheimer Dis Assoc Disord. 1993 Fall;7(3):152-64
references: 8217089
references: Neuroimage. 2013 Sep;78:270-83
references: 23583748
references: Neuroinformatics. 2015 Jan;13(1):65-81
references: 25129841
references: Neuroimage. 2011 May 15;56(2):766-81
references: 20542124
references: J Neurol. 2013 Apr;260(4):1104-15
references: 23224109
references: J Am Geriatr Soc. 1999 May;47(5):564-9
references: 10323650
references: IEEE Trans Med Imaging. 2010 Jun;29(6):1310-20
references: 20378467
references: Neuroinformatics. 2015 Jan;13(1):31-46
references: 25048627
references: Brain. 2008 Mar;131(Pt 3):681-9
references: 18202106
references: Radiology. 2013 Apr;267(1):221-30
references: 23238159
references: Neurology. 2006 Oct 10;67(7):1215-20
references: 17030755
references: Acad Radiol. 2012 May;19(5):509-11
references: 22463960
references: Neuroimage. 2008 Jun;41(2):277-85
references: 18400519
references: Alzheimers Dement. 2011 May;7(3):257-62
references: 21514247
references: Neurology. 2001 Jul 24;57(2):226-34
references: 11468306
references: IEEE Trans Med Imaging. 2012 Dec;31(12):2290-306
references: 23008245
references: Hum Brain Mapp. 2014 Jun;35(6):2836-51
references: 24115179
references: Med Image Comput Comput Assist Interv. 2014;17(Pt 3):41-8
references: 25320780
references: PLoS One. 2012;7(2):e31112
references: 22348041
references: Arch Neurol. 2012 Jul;69(7):856-67
references: 22409939
references: Neuroimage. 2008 Feb 1;39(3):1186-97
references: 18054253
references: Neuroimage. 2012 Aug 15;62(2):774-81
references: 22248573
references: Alzheimers Dement. 2011 May;7(3):263-9
references: 21514250
references: Neuroimage. 2012 Jun;61(2):457-63
references: 22094642
references: Front Neurol. 2014 May 12;5:71
references: 24860545
references: J Alzheimers Dis. 2014;41(3):685-708
references: 24718104
references: Neuroradiology. 2009 Feb;51(2):73-83
references: 18846369
references: Neurobiol Aging. 2013 Dec;34(12 ):2815-26
references: 23890839
references: Int Psychogeriatr. 2009 Aug;21(4):672-87
references: 19470201
references: Neurobiol Aging. 2015 Jan;36 Suppl 1:S23-31
references: 25260851
references: J Alzheimers Dis. 2012;32(4):969-79
references: 22890102
references: Alzheimers Dement. 2013 Jan;9(1):63-75.e2
references: 23305823
references: J Intern Med. 2004 Sep;256(3):183-94
references: 15324362
references: Ann Neurol. 2004 Mar;55(3):306-19
references: 14991808
references: Med Image Anal. 2013 Oct;17(7):790-804
references: 23725639
references: Neuroimage. 2008 Feb 15;39(4):1731-43
references: 18053747
references: J Alzheimers Dis. 2014;41(1):313-27
references: 24614907
references: Alzheimers Dement. 2011 May;7(3):270-9
references: 21514249
references: IEEE Trans Med Imaging. 2008 Apr;27(4):509-20
references: 18390347
references: Brain. 2010 Nov;133(11):3336-48
references: 20935035
references: Alzheimers Dement. 2011 Mar;7(2):171-4
references: 21414554
references: J Magn Reson Imaging. 2008 Apr;27(4):685-91
references: 18302232
references: Neuroimage. 2014 Nov 1;101:35-49
references: 24973601
references: Neuroimage. 2013 Jan 15;65:167-75
references: 23041336
references: Neuroimage. 2008 Jul 15;41(4):1220-7
references: 18474436
references: Neuroimage. 2009 Feb 15;44(4):1415-22
references: 19027862
references: Neurobiol Aging. 2008 Apr;29(4):514-23
references: 17174012
references: Ann N Y Acad Sci. 2009 Oct;1180:36-46
references: 19906259
references: Lancet Neurol. 2013 Feb;12(2):207-16
references: 23332364
references: PLoS One. 2011;6(10 ):e25446
references: 22022397
references: Neurology. 1984 Jul;34(7):939-44
references: 6610841