title: Editorial peer reviewers' recommendations at a general medical journal: are they reliable and do editors care?
abstract: BACKGROUND
  Editorial peer review is universally used but little studied. We examined the relationship between external reviewers' recommendations and the editorial outcome of manuscripts undergoing external peer-review at the Journal of General Internal Medicine (JGIM).
  
  
  RESULTS
  We examined reviewer recommendations and editors' decisions at JGIM between 2004 and 2008. For manuscripts undergoing peer review, we calculated chance-corrected agreement among reviewers on recommendations to reject versus accept or revise. Using mixed effects logistic regression models, we estimated intra-class correlation coefficients (ICC) at the reviewer and manuscript level. Finally, we examined the probability of rejection in relation to reviewer agreement and disagreement. The 2264 manuscripts sent for external review during the study period received 5881 reviews provided by 2916 reviewers; 28% of reviews recommended rejection. Chance corrected agreement (kappa statistic) on rejection among reviewers was 0.11 (p<.01). In mixed effects models adjusting for study year and manuscript type, the reviewer-level ICC was 0.23 (95% confidence interval [CI], 0.19-0.29) and the manuscript-level ICC was 0.17 (95% CI, 0.12-0.22). The editors' overall rejection rate was 48%: 88% when all reviewers for a manuscript agreed on rejection (7% of manuscripts) and 20% when all reviewers agreed that the manuscript should not be rejected (48% of manuscripts) (p<0.01).
  
  
  CONCLUSIONS
  Reviewers at JGIM agreed on recommendations to reject vs. accept/revise at levels barely beyond chance, yet editors placed considerable weight on reviewers' recommendations. Efforts are needed to improve the reliability of the peer-review process while helping editors understand the limitations of reviewers' recommendations.
authors.affiliation: Department of Medicine, University of California Davis, Sacramento, California, United States of America. rlkravitz@ucdavis.edu
authors.forename: Richard L
authors.initials: RL
authors.lastname: Kravitz
authors.forename: Peter
authors.initials: P
authors.lastname: Franks
authors.forename: Mitchell D
authors.initials: MD
authors.lastname: Feldman
authors.forename: Martha
authors.initials: M
authors.lastname: Gerrity
authors.forename: Cindy
authors.initials: C
authors.lastname: Byrne
authors.forename: William M
authors.initials: WM
authors.lastname: Tierney
mesh_terms: D035081:Editorial Policies; D010380:Peer Review / Q000592:standards*; D017712:Peer Review, Research / Q000592:standards; D010506:Periodicals as Topic / Q000592:standards*; D011643:Publishing / Q000592:standards
references.citation: JAMA. 2006 Apr 19;295(15):1801-8
references.pmid: 16622142
references.citation: Epidemiology. 2007 Mar;18(2):262-5
references.pmid: 17301708
references.citation: Cochrane Database Syst Rev. 2007 Apr 18;(2):MR000016
references.pmid: 17443635
references.citation: Indian Pediatr. 2006 Jun;43(6):479-89
references.pmid: 16820657
references.citation: Radiology. 1991 Mar;178(3):637-42
references.pmid: 1994394
references.citation: Am J Ophthalmol. 2007 Apr;143(4):551-60
references.pmid: 17276380
references.citation: Clin Rehabil. 2004 Mar;18(2):117-24
references.pmid: 15053119
references.citation: Annu Rev Med. 1999;50:99-110
references.pmid: 10073266
references.citation: Acad Med. 1992 Dec;67(12):856-9
references.pmid: 1457023
references.citation: Ann Intern Med. 1986 Aug;105(2):309-12
references.pmid: 3729227
references.citation: Adv Health Sci Educ Theory Pract. 2003;8(1):75-96
references.pmid: 12652170
references.citation: JAMA. 2006 Jan 18;295(3):314-7
references.pmid: 16418467
references.citation: Croat Med J. 1998 Mar;39(1):3-9
references.pmid: 9475799
references.citation: BMC Health Serv Res. 2002 Mar 26;2(1):8
references.pmid: 11914164
references.citation: J Gen Intern Med. 1993 May;8(5):255-8
references.pmid: 8505684
references.citation: Am J Psychiatry. 1993 Jun;150(6):947-52
references.pmid: 8494074
references.citation: Med Sci Monit. 2009 Feb;15(2):SR1-4
references.pmid: 19179983
references.citation: West J Med. 1990 Nov;153(5):520-2
references.pmid: 2260288
references.citation: Fam Med. 2005 May;37(5):360-3
references.pmid: 15883903
references.citation: Arch Pediatr Adolesc Med. 1996 Apr;150(4):380-3
references.pmid: 8634732
references.citation: Ann Intern Med. 1994 Jul 1;121(1):11-21
references.pmid: 8198342
references.citation: Brain. 2000 Sep;123 ( Pt 9):1964-9
references.pmid: 10960059
references.citation: J Gen Intern Med. 2009 Dec;24(12):1349-55
references.pmid: 19936844
references.citation: Br J Psychiatry. 1998 Aug;173:110-3; discussion 114-5
references.pmid: 9850221