The use of dedicated methodology and statistical reviewers for peer review: a content analysis of comments to authors made by methodology and regular reviewers.
OBJECTIVE
In 1997, Annals of Emergency Medicine initiated a protocol by which every original research article, in addition to each regular review, was concurrently evaluated by 1 of 2 methodology and statistical reviewers. We characterized and contrasted comments made by the methodology and regular peer reviewers.


METHODS
After pilot testing, interrater reliability assessment, and revision, we finalized a 99-item taxonomy of reviewer comments organized in 8 categories. Two authors, uninvolved in the writing of reviews, classified each comment from a random sample of methodology reviews from 1999. For 30 of these reviews (15 for each methodology reviewer), the 2 authors also scored all (range 2 to 5) regular reviews.


RESULTS
Sixty-five reviews by methodologist A, 60 by methodologist B, and 68 by regular reviewers were analyzed. Comments by methodologist A most frequently concerned the presentation of results (33% of all comments) and methods (17%). Methodologist B commented most frequently on presentation of results (28%) and statistical methods (16%). Regular reviewers most frequently made non-methodology/statistical comments (45%) and comments on presentation of results (18%). Of note, comments made by methodology and regular reviewers about methods issues were often contradictory.


CONCLUSIONS
The distributions of comments made by the 2 methodology and statistical reviewers were similar, although reviewer A emphasized presentation and reviewer B stressed statistical issues. The regular reviewers (most of whom were unaware that a dedicated methodology and statistical reviewer would be reviewing the article) paid much less attention to methodology issues. The 2 dedicated methodology and statistical reviewers created reviews that were similarly focused and emphasized methodology issues that were distinct from the issues raised by regular reviewers.
UCLA Emergency Medicine Center, UCLA School of Medicine, Los Angeles, CA 90048, USA. fday@ucla.edu
Frank C
FC
Day
David L
DL
Schriger
Christopher
C
Todd
Robert L
RL
Wears
D017712:Peer Review, Research / Q000379:methods*; D010506:Periodicals as Topic; D015203:Reproducibility of Results