title: Combating the scientific decline effect with confidence (intervals).
abstract: A symptom of the need for greater reproducibility in scientific practice is the decline effect-the fact that the size of many experimental effects decline with subsequent study or fail to replicate entirely. A simple way to combat this problem is for scientists to more routinely use confidence intervals (CIs) in their work. CIs provide frequentist bounds on the true size of an effect and can reveal when a statistically significant effect is possibly too small to be reliable or when a large effect might have been missed due to insufficient statistical power. CIs are often lacking in psychophysiological reports, likely due to the large number of dependent variables, which complicates deriving and visualizing CIs. In this article, I explain the value of CIs and show how to compute them for analyses involving multiple variables in various ways that adjust the intervals for the greater uncertainty induced by multiple statistical comparisons. The methods are illustrated using a basic visual oddball ERP dataset and freely available MATLAB software.
authors.affiliation: Department of Psychology, University of Toronto, Toronto, Ontario, Canada.|Feinstein Institute for Medical Research, Manhasset, New York, USA.
authors.forename: David M
authors.initials: DM
authors.lastname: Groppe
mesh_terms: D002540:Cerebral Cortex / Q000502:physiology; D003627:Data Interpretation, Statistical; D004569:Electroencephalography / Q000379:methods*; D005071:Evoked Potentials; D006801:Humans; D011603:Psychophysiology / Q000379:methods*; D015203:Reproducibility of Results
keywords: Analysis/statistical methods; EEG; EMG; ERPs; MEG; Methods