title: Menage a quoi? Optimal number of peer reviewers.
abstract: Peer review represents the primary mechanism used by funding agencies to allocate financial support and by journals to select manuscripts for publication, yet recent Cochrane reviews determined literature on peer review best practice is sparse. Key to improving the process are reduction of inherent vulnerability to high degree of randomness and, from an economic perspective, limiting both the substantial indirect costs related to reviewer time invested and direct administrative costs to funding agencies, publishers and research institutions. Use of additional reviewers per application may increase reliability and decision consistency, but adds to overall cost and burden. The optimal number of reviewers per application, while not known, is thought to vary with accuracy of judges or evaluation methods. Here I use bootstrapping of replicated peer review data from a Post-doctoral Fellowships competition to show that five reviewers per application represents a practical optimum which avoids large random effects evident when fewer reviewers are used, a point where additional reviewers at increasing cost provides only diminishing incremental gains in chance-corrected consistency of decision outcomes. Random effects were most evident in the relative mid-range of competitiveness. Results support aggressive high- and low-end stratification or triaging of applications for subsequent stages of review, with the proportion and set of mid-range submissions to be retained for further consideration being dependent on overall success rate.
authors.affiliation: Research Knowledge Translation and Ethics Portfolio, Canadian Institutes of Health Research, Ottawa, Ontario, Canada.
authors.forename: Richard R
authors.initials: RR
authors.lastname: Snell
mesh_terms: D035843:Biomedical Research / Q000191:economics; D005257:Fellowships and Scholarships; D010380:Peer Review
references.citation: Science. 1981 Nov 20;214(4523):881-6
references.pmid: 7302566
references.citation: J Clin Epidemiol. 2006 Aug;59(8):842-8
references.pmid: 16828678
references.citation: Lancet. 1998 Jul 25;352(9124):301-5
references.pmid: 9690424
references.citation: J Clin Epidemiol. 1997 Nov;50(11):1189-95
references.pmid: 9393374
references.citation: Am Stat. 2009 May 1;63(2):155-162
references.pmid: 22544972
references.citation: Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000016
references.pmid: 17443635
references.citation: J Epidemiol Community Health. 2007 Jan;61(1):9-12
references.pmid: 17183008
references.citation: J Clin Epidemiol. 2012 Jan;65(1):47-52
references.pmid: 21831594
references.citation: Am Psychol. 2008 Apr;63(3):160-8
references.pmid: 18377106
references.citation: Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000003
references.pmid: 17443627
references.citation: Biometrics. 1977 Mar;33(1):159-74
references.pmid: 843571
references.citation: PLoS One. 2008 Jul 23;3(7):e2761
references.pmid: 18648494
references.citation: PLoS One. 2010 Apr 08;5(4):e10072
references.pmid: 20386704
references.citation: BMC Med. 2010 Oct 20;8:62
references.pmid: 20961441
references.citation: EMBO Rep. 2014 Feb;15(2):131-3
references.pmid: 24397931
references.citation: Soc Sci Med. 2011 Apr;72(7):1056-63
references.pmid: 21388730
references.citation: Front Comput Neurosci. 2011 Dec 14;5:56
references.pmid: 22174702
references.citation: PLoS One. 2008;3(10):e3480
references.pmid: 18941530
references.citation: PLoS One. 2012;7(9):e46054
references.pmid: 23029386
references.citation: BMJ. 2011 Sep 27;343:d4797
references.pmid: 21951756
references.citation: PLoS One. 2011 Apr 12;6(4):e18680
references.pmid: 21533268