title: Thinking About Data, Research Methods, and Statistical Analyses: Commentary on Sijtsma's (2014) "Playing with Data".
abstract: We comment on Sijtsma's (2014) thought-provoking essay on how to minimize questionable research practices (QRPs) in psychology. We agree with Sijtsma that proactive measures to decrease the risk of QRPs will ultimately be more productive than efforts to target individual researchers and their work. In particular, we concur that encouraging researchers to make their data and research materials public is the best institutional antidote against QRPs, although we are concerned that Sijtsma's proposal to delegate more responsibility to statistical and methodological consultants could inadvertently reinforce the dichotomy between the substantive and statistical aspects of research. We also discuss sources of false-positive findings and replication failures in psychological research, and outline potential remedies for these problems. We conclude that replicability is the best metric of the minimization of QRPs and their adverse effects on psychological research.
authors.affiliation: Department of Psychology, Emory University, 475 PAIS Building, 36 Eagle Row, Atlanta, GA, 30322, USA. psyiw@emory.edu.
authors.forename: Irwin D
authors.initials: ID
authors.lastname: Waldman
authors.affiliation: Department of Psychology, Emory University, 475 PAIS Building, 36 Eagle Row, Atlanta, GA, 30322, USA. slilien@emory.edu.
authors.forename: Scott O
authors.initials: SO
authors.lastname: Lilienfeld
mesh_terms: D006801:Humans; D011594:Psychometrics; D012107:Research Design
keywords: minimizing false positives; questionable research practices; replicability
references.citation: PLoS One. 2011;6(11):e26828
references.pmid: 22073203
references.citation: Nat Genet. 2013 Oct;45(10):1150-9
references.pmid: 23974872
references.citation: Perspect Psychol Sci. 2012 Nov;7(6):528-30
references.pmid: 26168108
references.citation: Perspect Psychol Sci. 2012 Nov;7(6):600-4
references.pmid: 26168118
references.citation: Nat Rev Neurosci. 2013 May;14 (5):365-76
references.pmid: 23571845
references.citation: Perspect Psychol Sci. 2012 Nov;7(6):562-71
references.pmid: 26168113
references.citation: Psychol Sci. 2014 Jan;25(1):7-29
references.pmid: 24220629
references.citation: Psychol Methods. 2000 Jun;5(2):241-301
references.pmid: 10937333
references.citation: Proc Natl Acad Sci U S A. 2005 Nov 15;102(46):16569-72
references.pmid: 16275915
references.citation: PLoS Med. 2005 Aug;2(8):e124
references.pmid: 16060722
references.citation: Psychol Sci. 2011 Nov;22(11):1359-66
references.pmid: 22006061
references.citation: Perspect Psychol Sci. 2013 Sep;8(5):549-53
references.pmid: 26173211
references.citation: Psychol Sci. 2012 May 1;23(5):524-32
references.pmid: 22508865
references.citation: Science. 1974 Sep 27;185(4157):1124-31
references.pmid: 17835457
references.citation: Perspect Psychol Sci. 2012 Nov;7(6):615-31
references.pmid: 26168121
references.citation: Psychol Bull. 1960 Sep;57:416-28
references.pmid: 13744252
references.citation: J Exp Psychol Gen. 2014 Apr;143(2):534-47
references.pmid: 23855496
references.citation: Perspect Psychol Sci. 2014 Jan;9(1):76-80
references.pmid: 26173243
references.citation: Psychol Sci Public Interest. 2010 Dec;11(3):89-121
references.pmid: 26168416
references.citation: J Abnorm Soc Psychol. 1962 Sep;65:145-53
references.pmid: 13880271
references.citation: Trends Cogn Sci. 2014 May;18(5):235-41
references.pmid: 24656991
references.citation: Br J Psychiatry. 2013 Jan;202(1):1-4
references.pmid: 23284144
references.citation: Psychol Sci. 2004 Feb;15(2):119-26
references.pmid: 14738519
references.citation: Perspect Psychol Sci. 2012 Jan;7(1):67-71
references.pmid: 26168425