title: Data-analytical stability of cluster-wise and peak-wise inference in fMRI data analysis.
abstract: BACKGROUND
  Carp (2012) demonstrated the large variability that is present in the method sections of fMRI studies. This methodological variability between studies limits reproducible research.
  
  
  METHODS
  Evaluation protocols for methods used in fMRI should include data-analytical stability measures quantifying the variability in results following choices in the methods. Data-analytical stability can be seen as a proxy for reproducibility. To illustrate how one can perform such evaluations, we study two competing approaches for topological feature based inference (random field theory and permutation based testing) and two competing methods for smoothing (Gaussian smoothing and adaptive smoothing). We compare these approaches from the perspective of data-analytical stability in real data, and additionally consider validity and reliability in simulations.
  
  
  RESULTS
  There is clear evidence that choices in the methods impact the validity, reliability and stability of the results. For the particular comparison studied, we find that permutation based methods render the most valid results. For stability and reliability, the performance of different smoothing and inference types depends on the setting. However, while being more reliable, adaptive smoothing can evoke less stable results when using larger kernel width, especially with cluster size based permutation inference.
  
  
  METHODS
  While existing evaluation methods focus on validity and reliability, we show that data-analytical stability enables to further distinguish between performance of different methods.
  
  
  CONCLUSIONS
  Data-analytical stability is an important additional criterion that can easily be incorporated in evaluation protocols.
authors.affiliation: Department of Data Analysis, Ghent University, H. Dunantlaan 1, B-9000 Ghent, Belgium. Electronic address: sanne.roels@ugent.be.
authors.forename: S P
authors.initials: SP
authors.lastname: Roels
authors.affiliation: Department of Data Analysis, Ghent University, H. Dunantlaan 1, B-9000 Ghent, Belgium.
authors.forename: H
authors.initials: H
authors.lastname: Bossier
authors.affiliation: Department of Data Analysis, Ghent University, H. Dunantlaan 1, B-9000 Ghent, Belgium.
authors.forename: T
authors.initials: T
authors.lastname: Loeys
authors.affiliation: Department of Data Analysis, Ghent University, H. Dunantlaan 1, B-9000 Ghent, Belgium.
authors.forename: B
authors.initials: B
authors.lastname: Moerkerke
mesh_terms: D001921:Brain / Q000502:physiology; D001931:Brain Mapping / Q000379:methods*; D003198:Computer Simulation; D007091:Image Processing, Computer-Assisted / Q000379:methods; D008279:Magnetic Resonance Imaging / Q000379:methods*; D008959:Models, Neurological; D016011:Normal Distribution; D015203:Reproducibility of Results
keywords: Reliability; Reproducibility; Stability; fMRI