title: Multiple trials may yield exaggerated effect size estimates.
abstract: Published psychological research attempting to support the existence of small and medium effect sizes may not have enough participants to do so accurately, and thus, repeated trials or the use of multiple items may be used in an attempt to obtain significance. Through a series of Monte-Carlo simulations, this article describes the results of multiple trials or items on effect size estimates when the averages and aggregates of a dependent measure are analyzed. The simulations revealed a large increase in observed effect size estimates when the numbers of trials or items in an experiment were increased. Overestimation effects are mitigated by correlations between trials or items, but remain substantial in some cases. Some concepts, such as a P300 wave or a test score, are best defined as a composite of measures. Troubles may arise in more exploratory research where the interrelations among trials or items may not be well described.
authors.affiliation: Department of Psychology, Kings College, London.
authors.forename: Andrew
authors.initials: A
authors.lastname: Brand
authors.forename: Michael T
authors.initials: MT
authors.lastname: Bradley
authors.forename: Lisa A
authors.initials: LA
authors.lastname: Best
authors.forename: George
authors.initials: G
authors.lastname: Stoica
mesh_terms: D015982:Bias; D003625:Data Collection / Q000706:statistics & numerical data*; D018913:Event-Related Potentials, P300 / Q000502:physiology; D006801:Humans; D009010:Monte Carlo Method; D011581:Psychological Tests / Q000706:statistics & numerical data; D011584:Psychology / Q000706:statistics & numerical data*; D011594:Psychometrics / Q000706:statistics & numerical data; D012044:Regression Analysis; D015203:Reproducibility of Results; D013223:Statistics as Topic