Peering at peer review revealed high degree of chance associated with funding of grant applications.
OBJECTIVE
There is a persistent degree of uncertainty and dissatisfaction with the peer review process underlining the need to validate the current grant awarding procedures. This study compared the CLassic Structured Scientific In-depth two reviewer critique (CLASSIC) with an all panel members' independent ranking method (RANKING). Eleven reviewers, reviewed 32 applications for a pilot project competition at a major university medical center.


RESULTS
The degree of agreement between the two methods was poor (kappa = 0.36). The top rated project in each stream would have failed the funding cutoff with a frequency of 9 and 35%, depending on which pair of reviewers had been selected. Four of the top 10 projects identified by RANKING had a greater than 50% of not being funded by the CLASSIC ranking. Ten reviewers provided optimal consistency for the RANKING method.


CONCLUSIONS
This study found that there is a considerable amount of chance associated with funding decisions under the traditional method of assigning the grant to two main reviewers. We recommend using the all reviewer ranking procedure to arrive at decisions about grant applications as this removes the impact of extreme reviews.
Mayo
Nancy E
NE
Division of Clinical Epidemiology R4.29, McGill University Health Center, RVH Site, 687 Pine Avenue West, Montreal, H3A 1A1, Canada. nancy.mayo@mcgill.ca
Brophy
James
J
Goldberg
Mark S
MS
Klein
Marina B
MB
Miller
Sydney
S
Platt
Robert W
RW
Ritchie
Judith
J
D002170:Canada; D006801:Humans; D015588:Observer Variation; D017712:Peer Review, Research / Q000379:methods / Q000592:standards*; D012109:Research Support as Topic