Meta-assessment of bias in science.
Numerous biases are believed to affect the scientific literature, but their actual prevalence across disciplines is unknown. To gain a comprehensive picture of the potential imprint of bias in science, we probed for the most commonly postulated bias-related patterns and risk factors, in a large random sample of meta-analyses taken from all disciplines. The magnitude of these biases varied widely across fields and was overall relatively small. However, we consistently observed a significant risk of small, early, and highly cited studies to overestimate effects and of studies not published in peer-reviewed journals to underestimate them. We also found at least partial confirmation of previous evidence suggesting that US studies and early studies might report more extreme effects, although these effects were smaller and more heterogeneously distributed across meta-analyses and disciplines. Authors publishing at high rates and receiving many citations were, overall, not at greater risk of bias. However, effect sizes were likely to be overestimated by early-career researchers, those working in small or long-distance collaborations, and those responsible for scientific misconduct, supporting hypotheses that connect bias to situational factors, lack of mutual control, and individual integrity. Some of these patterns and risk factors might have modestly increased in intensity over time, particularly in the social sciences. Our findings suggest that, besides one being routinely cautious that published small, highly-cited, and earlier studies may yield inflated results, the feasibility and costs of interventions to attenuate biases in the literature might need to be discussed on a discipline-specific and topic-specific basis.
Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Palo Alto, CA 94304; email@danielefanelli.com.
Daniele
D
Fanelli
Centre for Science and Technology Studies, Leiden University, 2333 AL Leiden, The Netherlands.
Rodrigo
R
Costas
Meta-Research Innovation Center at Stanford (METRICS), Stanford University, Palo Alto, CA 94304.|Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305.|Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA 94305.|Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA 94305.
John P A
JP
Ioannidis
D005260:Female; D006801:Humans; D008297:Male; D011643:Publishing; D012107:Research Design; D012108:Research Personnel; D015871:Scientific Misconduct; D015983:Selection Bias
bias; integrity; meta-analysis; meta-research; misconduct