The impact of data-analysis methods on cumulative research knowledge: statistical significance testing, confidence intervals, and meta-analysis.
The methods of data analysis used in research have a major effect on the development of cumulative knowledge. Traditional methods based on statistical significance testing have systematically retarded the growth of cumulative research by making it virtually impossible to discern the real meaning of research literatures. Meta-analysis makes it possible to demonstrate graphically the high price the research enterprise has paid for its reliance on significance testing. But in addition to these demonstrations, reform will require that researchers come to understand that the benefits they see as flowing from the use of significance tests are illusory. In research practice and in training of researchers, we must use and teach appropriate data analysis methods: point estimates of effect sizes and confidence intervals within individual studies, and meta-analysis in the integration of multiple studies to produce final conclusions. These reforms are essential to the progress of cumulative research knowledge.
Schmidt
F
F
University of Iowa, Iowa City 52242, USA.
Hunter
J E
JE
D000704:Analysis of Variance; D003627:Data Interpretation, Statistical; D005069:Evaluation Studies as Topic; D006302:Health Services Research / Q000379:methods*; D015201:Meta-Analysis as Topic; D015233:Models, Statistical; D012107:Research Design