Accuracy of effect size estimates from published psychological experiments involving multiple trials.
The reporting of exaggerated effect size estimates may occur either through researchers accepting statistically significant results when power is inadequate and/or from repeated measures approaches aggregating, averaging multiple items, or multiple trials. Monte-Carlo simulations with input of a small, medium, or large effect size were conducted on multiple items or trials that were either averaged or aggregated to create a single dependent measure. Alpha was set at the .05 level, and the trials were assessed over item or trial correlations ranging from 0 to 1. Simulations showed a large increase in observed effect size averages and the power to accept these estimates as statistically significant increased over numbers of trials or items. Overestimation effects were mitigated as correlations between trials increased but still remained substantial in some cases. The implications of these findings for meta-analyses and different research scenarios are discussed.
a King's College London.
Andrew
A
Brand
M T
MT
Bradley
Lisa A
LA
Best
George
G
Stoica
D003627:Data Interpretation, Statistical; D006801:Humans; D015201:Meta-Analysis as Topic; D009010:Monte Carlo Method; D011589:Psychology, Experimental / Q000379:methods*; D012107:Research Design; D018401:Sample Size
Alpha levels; Monte Carlo; correlation; effect size; multiple assessments; power