Searching for reliable relationships with statistics packages: an empirical example of the potential problems.
Many social scientists appear to possess an overconfidence in the reliability of research results from a single, small-sample, inferential study. In this article, the authors speculate that "user-friendly" statistics packages have the potential to exacerbate statistical misinterpretation by providing researchers with a tool to explore data easily and identify what is interpreted as "reliable" relationships. This article contains an empirical demonstration of the potential problems that arise when a large number of statistical tests are interpreted. Results show that statistically significant results may be unreliable. Also, a zero relationship can erroneously appear as a medium to large effect size relationship when a small sample is used (e.g., n = 30). The authors suggest the need for multiple replications as the criterion of a reliable finding.
Riniolo
T C
TC
Department of Psychology, Adams State College, Alamosa, CO 81102, USA. tcrinol@adams.edu
Schmidt
L A
LA
D000328:Adult; D015982:Bias; D005260:Female; D006801:Humans; D008297:Male; D008432:Mathematical Computing; D011594:Psychometrics; D015203:Reproducibility of Results; D012106:Research / Q000706:statistics & numerical data; D012984:Software