title: Longitudinal trends in the performance of scientific peer reviewers.
abstract: OBJECTIVE
  We characterize changes in review quality by individual peer reviewers over time.
  
  
  METHODS
  Editors at a specialty journal in the top 11% of Institute of Scientific Information journals rated the quality of every review, using a validated 5-point quality score. Linear mixed-effect models were used to analyze rating changes over time, calculating within-reviewer trends plus predicted slope of change in score for each reviewer. Reviewers at this journal have been shown comparable to those at other journals.
  
  
  RESULTS
  Reviews (14,808) were performed by 1,499 reviewers and rated by 84 editors during the 14-year study. Ninety-two percent of reviewers demonstrated very slow but steady deterioration in their scores (mean -0.04 points [-0.8%] per year). Rate of deterioration was unrelated to duration of reviewing but moderately correlated with mean reviewer quality score (R=0.52). The mean score of each reviewer's first 4 reviews predicted subsequent performance with a sensitivity of 75% and specificity of 47%. Scores of the group stayed constant over time despite deterioration because newly recruited reviewers initially had higher mean quality scores than their predecessors.
  
  
  CONCLUSIONS
  This study, one of few tracking expert performance longitudinally, demonstrates that most journal peer reviewers received lower quality scores for article assessment over the years. This could be due to deteriorating performance (caused by either cognitive changes or competing priorities) or, to a partial degree, escalating expectations; other explanations were ruled out. This makes monitoring reviewer quality even more crucial to maintain the mission of scientific journals.
authors.affiliation: Department of Emergency Medicine, University of California, San Francisco, USA.
authors.forename: Michael
authors.initials: M
authors.lastname: Callaham
authors.forename: Charles
authors.initials: C
authors.lastname: McCulloch
mesh_terms: D004635:Emergency Medicine; D055811:Journal Impact Factor; D008137:Longitudinal Studies; D015588:Observer Variation; D010380:Peer Review / Q000592:standards / Q000639:trends; D010506:Periodicals as Topic / Q000592:standards / Q000639:trends