Assessing atrophy measurement techniques in dementia: Results from the MIRIAD atrophy challenge.
Structural MRI is widely used for investigating brain atrophy in many neurodegenerative disorders, with several research groups developing and publishing techniques to provide quantitative assessments of this longitudinal change. Often techniques are compared through computation of required sample size estimates for future clinical trials. However interpretation of such comparisons is rendered complex because, despite using the same publicly available cohorts, the various techniques have been assessed with different data exclusions and different statistical analysis models. We created the MIRIAD atrophy challenge in order to test various capabilities of atrophy measurement techniques. The data consisted of 69 subjects (46 Alzheimer's disease, 23 control) who were scanned multiple (up to twelve) times at nine visits over a follow-up period of one to two years, resulting in 708 total image sets. Nine participating groups from 6 countries completed the challenge by providing volumetric measurements of key structures (whole brain, lateral ventricle, left and right hippocampi) for each dataset and atrophy measurements of these structures for each time point pair (both forward and backward) of a given subject. From these results, we formally compared techniques using exactly the same dataset. First, we assessed the repeatability of each technique using rates obtained from short intervals where no measurable atrophy is expected. For those measures that provided direct measures of atrophy between pairs of images, we also assessed symmetry and transitivity. Then, we performed a statistical analysis in a consistent manner using linear mixed effect models. The models, one for repeated measures of volume made at multiple time-points and a second for repeated "direct" measures of change in brain volume, appropriately allowed for the correlation between measures made on the same subject and were shown to fit the data well. From these models, we obtained estimates of the distribution of atrophy rates in the Alzheimer's disease (AD) and control groups and of required sample sizes to detect a 25% treatment effect, in relation to healthy ageing, with 95% significance and 80% power over follow-up periods of 6, 12, and 24months. Uncertainty in these estimates, and head-to-head comparisons between techniques, were carried out using the bootstrap. The lateral ventricles provided the most stable measurements, followed by the brain. The hippocampi had much more variability across participants, likely because of differences in segmentation protocol and less distinct boundaries. Most methods showed no indication of bias based on the short-term interval results, and direct measures provided good consistency in terms of symmetry and transitivity. The resulting annualized rates of change derived from the model ranged from, for whole brain: -1.4% to -2.2% (AD) and -0.35% to -0.67% (control), for ventricles: 4.6% to 10.2% (AD) and 1.2% to 3.4% (control), and for hippocampi: -1.5% to -7.0% (AD) and -0.4% to -1.4% (control). There were large and statistically significant differences in the sample size requirements between many of the techniques. The lowest sample sizes for each of these structures, for a trial with a 12month follow-up period, were 242 (95% CI: 154 to 422) for whole brain, 168 (95% CI: 112 to 282) for ventricles, 190 (95% CI: 146 to 268) for left hippocampi, and 158 (95% CI: 116 to 228) for right hippocampi. This analysis represents one of the most extensive statistical comparisons of a large number of different atrophy measurement techniques from around the globe. The challenge data will remain online and publicly available so that other groups can assess their methods.
Dementia Research Centre, UCL Institute of Neurology, London, UK; Translational Imaging Group, Centre for Medical Image Computing, UCL, London, UK. Electronic address: d.cash@ucl.ac.uk.
David M
DM
Cash
Dementia Research Centre, UCL Institute of Neurology, London, UK; Department of Medical Statistics, London School of Hygiene and Tropical Medicine, London, UK.
Chris
C
Frost
Electrical & Electronics Engineering, Bahcesehir University, Istanbul, Turkey.
Leonardo O
LO
Iheme
Biomedical Engineering, Bahcesehir University, Istanbul, Turkey.
Devrim
D
Ünay
Department of Neurology, Bayindir Hospital Icerenkoy, Istanbul, Turkey.
Melek
M
Kandemir
The Australian eHealth Research Centre, CSIRO Digital Productivity Flagship, Herston, QLD, Australia.
Jurgen
J
Fripp
The Australian eHealth Research Centre, CSIRO Digital Productivity Flagship, Herston, QLD, Australia.
Olivier
O
Salvado
The Australian eHealth Research Centre, CSIRO Digital Productivity Flagship, Herston, QLD, Australia.
Pierrick
P
Bourgeat
Martinos Center for Biomedical Imaging, Dept. of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA; Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, Cambridge, MA, USA.
Martin
M
Reuter
Martinos Center for Biomedical Imaging, Dept. of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA; Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, Cambridge, MA, USA.
Bruce
B
Fischl
Asclepios Research Project, INRIA Sophia Antipolis, Sophia Antipolis, France.
Marco
M
Lorenzi
IRCCS San Giovanni di Dio Fatebenefratelli, Brescia, Italy; Memory Clinic and Laboratoire de Neuroimagerie du Vieillissement (LANVIE), University Hospitals and University of Geneva, Geneva, Switzerland.
Giovanni B
GB
Frisoni
Asclepios Research Project, INRIA Sophia Antipolis, Sophia Antipolis, France.
Xavier
X
Pennec
Brain Image Analysis, LLC, Coralville, IA, USA.
Ronald K
RK
Pierson
Department of Radiology, Mayo Clinic, Rochester, MN, USA.
Jeffrey L
JL
Gunter
Department of Radiology, Mayo Clinic, Rochester, MN, USA.
Matthew L
ML
Senjem
Department of Radiology, Mayo Clinic, Rochester, MN, USA.
Clifford R
CR
Jack
McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, Montreal, QC, Canada.
Nicolas
N
Guizard
McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, Montreal, QC, Canada.
Vladimir S
VS
Fonov
McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, Montreal, QC, Canada.
D Louis
DL
Collins
Translational Imaging Group, Centre for Medical Image Computing, UCL, London, UK.
Marc
M
Modat
Translational Imaging Group, Centre for Medical Image Computing, UCL, London, UK.
M Jorge
MJ
Cardoso
Dementia Research Centre, UCL Institute of Neurology, London, UK.
Kelvin K
KK
Leung
Almaden Research Center, IBM Research, Almaden, CA, USA; Penn Image Computing and Science Laboratory, Department of Radiology, University of Pennsylvania, USA.
Hongzhi
H
Wang
Penn Image Computing and Science Laboratory, Department of Radiology, University of Pennsylvania, USA.
Sandhitsu R
SR
Das
Penn Image Computing and Science Laboratory, Department of Radiology, University of Pennsylvania, USA.
Paul A
PA
Yushkevich
Dementia Research Centre, UCL Institute of Neurology, London, UK.
Ian B
IB
Malone
Dementia Research Centre, UCL Institute of Neurology, London, UK.
Nick C
NC
Fox
Dementia Research Centre, UCL Institute of Neurology, London, UK.
Jonathan M
JM
Schott
Dementia Research Centre, UCL Institute of Neurology, London, UK; Translational Imaging Group, Centre for Medical Image Computing, UCL, London, UK.
Sebastien
S
Ourselin
D000368:Aged; D000544:Alzheimer Disease / Q000473:pathology*; D001284:Atrophy; D001921:Brain / Q000473:pathology*; D003627:Data Interpretation, Statistical; D005260:Female; D006624:Hippocampus / Q000473:pathology; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008279:Magnetic Resonance Imaging / Q000379:methods*; D008297:Male; D008875:Middle Aged; D015203:Reproducibility of Results