title: Peering at peer review revealed high degree of chance associated with funding of grant applications.
abstract: OBJECTIVE
  There is a persistent degree of uncertainty and dissatisfaction with the peer review process underlining the need to validate the current grant awarding procedures. This study compared the CLassic Structured Scientific In-depth two reviewer critique (CLASSIC) with an all panel members' independent ranking method (RANKING). Eleven reviewers, reviewed 32 applications for a pilot project competition at a major university medical center.
  
  
  RESULTS
  The degree of agreement between the two methods was poor (kappa = 0.36). The top rated project in each stream would have failed the funding cutoff with a frequency of 9 and 35%, depending on which pair of reviewers had been selected. Four of the top 10 projects identified by RANKING had a greater than 50% of not being funded by the CLASSIC ranking. Ten reviewers provided optimal consistency for the RANKING method.
  
  
  CONCLUSIONS
  This study found that there is a considerable amount of chance associated with funding decisions under the traditional method of assigning the grant to two main reviewers. We recommend using the all reviewer ranking procedure to arrive at decisions about grant applications as this removes the impact of extreme reviews.
authors.affiliation: Division of Clinical Epidemiology R4.29, McGill University Health Center, RVH Site, 687 Pine Avenue West, Montreal, H3A 1A1, Canada. nancy.mayo@mcgill.ca
authors.forename: Nancy E
authors.initials: NE
authors.lastname: Mayo
authors.forename: James
authors.initials: J
authors.lastname: Brophy
authors.forename: Mark S
authors.initials: MS
authors.lastname: Goldberg
authors.forename: Marina B
authors.initials: MB
authors.lastname: Klein
authors.forename: Sydney
authors.initials: S
authors.lastname: Miller
authors.forename: Robert W
authors.initials: RW
authors.lastname: Platt
authors.forename: Judith
authors.initials: J
authors.lastname: Ritchie
mesh_terms: D002170:Canada; D006801:Humans; D015588:Observer Variation; D017712:Peer Review, Research / Q000379:methods / Q000592:standards*; D012109:Research Support as Topic