title: False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant.
abstract: In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.
authors: Wharton School, University of Pennsylvania, Philadelphia, PA 19104, USA. jsimmo@wharton.upenn.edu
authors: Joseph P
authors: JP
authors: Simmons
authors: Leif D
authors: LD
authors: Nelson
authors: Uri
authors: U
authors: Simonsohn
mesh_terms: D000328:Adult; D003198:Computer Simulation; D003625:Data Collection / Q000592:standards*; D003627:Data Interpretation, Statistical; D006801:Humans; D017712:Peer Review, Research / Q000592:standards; D017410:Practice Guidelines as Topic; D011642:Publications / Q000592:standards*; D012107:Research Design / Q000592:standards*; D012108:Research Personnel / Q000523:psychology / Q000592:standards*; D013223:Statistics as Topic; D055815:Young Adult