title: Spine journals: is reviewer agreement on publication recommendations greater than would be expected by chance?
abstract: BACKGROUND
  It is commonly believed that the peer-review process is reliable and consistent. It appears, however, that depending on the journal and the editorial leadership, agreement by reviewers on whether to publish submitted articles varies widely; from substantial to slightly greater than one would expect with random assignments of acceptance or rejection.
  
  
  OBJECTIVE
  The purpose was to assess peer-review agreement in major spine journals.
  
  
  METHODS
  This study is for the assessment of reviewer agreement.
  
  
  METHODS
  The study consisted of consecutive reviews of 200 submitted articles.
  
  
  METHODS
  Agreement via Kappa statistics.
  
  
  METHODS
  Group A consisted of 200 consecutive article reviews for which the senior author was involved in the review or editorial process over the past 8 years for two major spine journals. Reviewers' recommendations were placed into one of two groups: accept/minimal revisions or major revision/reject. Standard Kappa statistics were used to assess reviewer agreement. Group B consisted of a similar set, but with wholly randomly generated recommendations. Again, Kappa statistics were used.
  
  
  RESULTS
  Kappa for Group A was 0.155 with a range of 0.017 to 0.294 at 95% confidence interval and agreement at 0.6; suggesting "slight" reviewer agreement. Kappa for Group B behaved as expected, with "poor" agreement.
  
  
  CONCLUSIONS
  Agreement regarding peer-review recommendations for publication in spine journals appears to be better than would be expected in the random situation; but still only "slight." This suggests that review methodology varies considerably among reviewers and that further study should be undertaken to determine "ideal" agreement levels and ways to increase review consistency/quality commensurate with the editorial missions of the journals.
authors.affiliation: Weill Cornell Medical College, New York, NY, USA. bkweiner@tmhs.org
authors.forename: Bradley K
authors.initials: BK
authors.lastname: Weiner
authors.forename: Jacob P
authors.initials: JP
authors.lastname: Weiner
authors.forename: Harvey E
authors.initials: HE
authors.lastname: Smith
mesh_terms: D035081:Editorial Policies; D006801:Humans; D015588:Observer Variation; D009985:Orthopedics / Q000592:standards* / Q000639:trends; D017712:Peer Review, Research; D010506:Periodicals as Topic / Q000592:standards* / Q000639:trends; D017594:Publication Bias / Q000706:statistics & numerical data*; D011643:Publishing / Q000592:standards / Q000706:statistics & numerical data*; D015203:Reproducibility of Results