Reviewer bias against the unconventional? A randomized double-blind study of peer review.
OBJECTIVE
To test the hypothesis that there is a reviewer bias against publication of a test of an unconventional drug.


METHODS
Randomized, controlled, double-blind study of peer review.


METHODS
Convenience sample of 291 medical doctors from a wide variety of specialties drawn from a list of conference participants.


METHODS
Reviewers were randomly assigned to receive one of two versions of a manuscript. Version M related to an in-vitro experiment on a mainstream drug (Metoprolol). The otherwise identical version V used a highly unconventional drug (beef spleen cell extract) for the same experiment. Reviewers were asked to complete a standardised evaluation sheet including visual analogue scales (VASs) on a set of predefined quality criteria. All participants were debriefed after completion of the study.


RESULTS
The response rate was 61%. There were no significant differences in VAS ratings between the two versions of the manuscript. Ratings covered the entire range of the VASs.


CONCLUSIONS
In the present setting, there was no evidence for a reviewer-bias against testing an unconventional drug. The low inter-rater reliability, however, suggested inadequate validity of peer review.
Ernst
E
E
University of Exeter, Department of Complementary Medicine, UK.
Resch
K L
KL
D000818:Animals; D015982:Bias; D002417:Cattle; D000529:Complementary Therapies; D004311:Double-Blind Method; D006801:Humans; D015588:Observer Variation; D017712:Peer Review, Research; D015203:Reproducibility of Results; D018709:Statistics, Nonparametric; D014020:Tissue Extracts / Q000627:therapeutic use*
D014020:Tissue Extracts