title: Hippocampal volume change measurement: quantitative assessment of the reproducibility of expert manual outlining and the automated methods FreeSurfer and FIRST.
abstract: BACKGROUND
  To measure hippocampal volume change in Alzheimer's disease (AD) or mild cognitive impairment (MCI), expert manual delineation is often used because of its supposed accuracy. It has been suggested that expert outlining yields poorer reproducibility as compared to automated methods, but this has not been investigated.
  
  
  OBJECTIVE
  To determine the reproducibilities of expert manual outlining and two common automated methods for measuring hippocampal atrophy rates in healthy aging, MCI and AD.
  
  
  METHODS
  From the Alzheimer's Disease Neuroimaging Initiative (ADNI), 80 subjects were selected: 20 patients with AD, 40 patients with mild cognitive impairment (MCI) and 20 healthy controls (HCs). Left and right hippocampal volume change between baseline and month-12 visit was assessed by using expert manual delineation, and by the automated software packages FreeSurfer (longitudinal processing stream) and FIRST. To assess reproducibility of the measured hippocampal volume change, both back-to-back (BTB) MPRAGE scans available for each visit were analyzed. Hippocampal volume change was expressed in μL, and as a percentage of baseline volume. Reproducibility of the 1-year hippocampal volume change was estimated from the BTB measurements by using linear mixed model to calculate the limits of agreement (LoA) of each method, reflecting its measurement uncertainty. Using the delta method, approximate p-values were calculated for the pairwise comparisons between methods. Statistical analyses were performed both with inclusion and exclusion of visibly incorrect segmentations.
  
  
  RESULTS
  Visibly incorrect automated segmentation in either one or both scans of a longitudinal scan pair occurred in 7.5% of the hippocampi for FreeSurfer and in 6.9% of the hippocampi for FIRST. After excluding these failed cases, reproducibility analysis for 1-year percentage volume change yielded LoA of ±7.2% for FreeSurfer, ±9.7% for expert manual delineation, and ±10.0% for FIRST. Methods ranked the same for reproducibility of 1-year μL volume change, with LoA of ±218 μL for FreeSurfer, ±319 μL for expert manual delineation, and ±333 μL for FIRST. Approximate p-values indicated that reproducibility was better for FreeSurfer than for manual or FIRST, and that manual and FIRST did not differ. Inclusion of failed automated segmentations led to worsening of reproducibility of both automated methods for 1-year raw and percentage volume change.
  
  
  CONCLUSIONS
  Quantitative reproducibility values of 1-year microliter and percentage hippocampal volume change were roughly similar between expert manual outlining, FIRST and FreeSurfer, but FreeSurfer reproducibility was statistically significantly superior to both manual outlining and FIRST after exclusion of failed segmentations.
authors: Image Analysis Center, VU University Medical Center, Amsterdam, The Netherlands; Department of Radiology, VU University Medical Center, Amsterdam, The Netherlands.
authors: Emma R
authors: ER
authors: Mulder
authors: Image Analysis Center, VU University Medical Center, Amsterdam, The Netherlands; Department of Radiology, VU University Medical Center, Amsterdam, The Netherlands.
authors: Remko A
authors: RA
authors: de Jong
authors: Department of Epidemiology and Biostatistics, VU University Medical Center, Amsterdam, The Netherlands.
authors: Dirk L
authors: DL
authors: Knol
authors: Image Analysis Center, VU University Medical Center, Amsterdam, The Netherlands; Department of Information and Communication Technology, VU University Medical Center, Amsterdam, The Netherlands.
authors: Ronald A
authors: RA
authors: van Schijndel
authors: Department of Physics and Medical Technology, VU University Medical Center, Amsterdam, The Netherlands.
authors: Keith S
authors: KS
authors: Cover
authors: Department of Neurology, VU University Medical Center, Amsterdam, The Netherlands.
authors: Pieter J
authors: PJ
authors: Visser
authors: Image Analysis Center, VU University Medical Center, Amsterdam, The Netherlands; Department of Radiology, VU University Medical Center, Amsterdam, The Netherlands.
authors: Frederik
authors: F
authors: Barkhof
authors: Department of Radiology, VU University Medical Center, Amsterdam, The Netherlands; Department of Physics and Medical Technology, VU University Medical Center, Amsterdam, The Netherlands. Electronic address: H.Vrenken@vumc.nl.
authors: Hugo
authors: H
authors: Vrenken
mesh_terms: D000368:Aged; D000465:Algorithms; D000544:Alzheimer Disease / Q000473:pathology*; D001185:Artificial Intelligence; D001284:Atrophy; D060825:Cognitive Dysfunction / Q000473:pathology*; D005260:Female; D006624:Hippocampus / Q000473:pathology*; D006801:Humans; D007089:Image Enhancement / Q000379:methods; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D021621:Imaging, Three-Dimensional / Q000379:methods*; D008279:Magnetic Resonance Imaging / Q000379:methods*; D008297:Male; D015588:Observer Variation; D009929:Organ Size; D010363:Pattern Recognition, Automated / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D012984:Software; D012986:Software Validation
keywords: Alzheimer's disease; Automatic segmentation; Hippocampus; Magnetic resonance imaging; Manual segmentation; Mild cognitive impairment