title: Substantial agreement of referee recommendations at a general medical journal--a peer review evaluation at Deutsches Ärzteblatt International.
abstract: BACKGROUND
  Peer review is the mainstay of editorial decision making for medical journals. There is a dearth of evaluations of journal peer review with regard to reliability and validity, particularly in the light of the wide variety of medical journals. Studies carried out so far indicate low agreement among reviewers. We present an analysis of the peer review process at a general medical journal, Deutsches Ärzteblatt International.
  
  
  RESULTS
  554 reviewer recommendations on 206 manuscripts submitted between 7/2008 and 12/2009 were analyzed: 7% recommended acceptance, 74% revision and 19% rejection. Concerning acceptance (with or without revision) versus rejection, there was a substantial agreement among reviewers (74.3% of pairs of recommendations) that was not reflected by Fleiss' or Cohen's kappa (<0.2). The agreement rate amounted to 84% for acceptance, but was only 31% for rejection. An alternative kappa-statistic, however, Gwet's kappa (AC1), indicated substantial agreement (0.63). Concordance between reviewer recommendation and editorial decision was almost perfect when reviewer recommendations were unanimous. The correlation of reviewer recommendations and citations as counted by Web of Science was low (partial correlation adjusted for year of publication: -0.03, n.s.).
  
  
  CONCLUSIONS
  Although our figures are similar to those reported in the literature our conclusion differs from the widely held view that reviewer agreement is low: Based on overall agreement we consider the concordance among reviewers sufficient for the purposes of editorial decision making. We believe that various measures, such as positive and negative agreement or alternative Kappa values are superior to the application of Cohen's or Fleiss' Kappa in the analysis of nominal or ordinal level data regarding reviewer agreement. Also, reviewer recommendations seem to be a poor proxy for citations because, for example, manuscripts will be changed considerably during the revision process.
authors: Deutsches Ärzteblatt International, Editorial Offices, Cologne, Germany. baethge@aerzteblatt.de
authors: Christopher
authors: C
authors: Baethge
authors: Jeremy
authors: J
authors: Franklin
authors: Stephan
authors: S
authors: Mertens
mesh_terms: D032921:Consensus; D035081:Editorial Policies; D018156:Journalism, Medical; D010380:Peer Review; D013223:Statistics as Topic
references: Croat Med J. 2002 Jun;43(3):286-9
references: 12035133
references: Dtsch Arztebl Int. 2012 Apr;109(15):267-9
references: 22567061
references: PLoS One. 2010 Apr 08;5(4):e10072
references: 20386704
references: JAMA. 1990 Mar 9;263(10):1323-9
references: 2406470
references: JAMA. 1990 Mar 9;263(10):1321-2
references: 2406469
references: Dtsch Arztebl Int. 2012 Dec;109(51-52):900-2
references: 23372614
references: PLoS One. 2010 Dec 14;5(12):e14331
references: 21179459
references: Brain. 2000 Sep;123 ( Pt 9):1964-9
references: 10960059
references: Biometrics. 1977 Mar;33(1):159-74
references: 843571
references: J Clin Epidemiol. 1990;43(6):543-9
references: 2348207
references: J Clin Epidemiol. 1990;43(6):551-8
references: 2189948
references: JAMA. 2009 Sep 9;302(10):1092-6
references: 19738094
references: Br J Math Stat Psychol. 2008 May;61(Pt 1):29-48
references: 18482474
references: Cardiovasc Res. 2002 Dec;56(3):339-46
references: 12445872
references: JAMA. 1990 Mar 9;263(10):1438-41
references: 2304222