How reliable is peer review of scientific abstracts? Looking back at the 1991 Annual Meeting of the Society of General Internal Medicine.
OBJECTIVE
To evaluate the interrater reproducibility of scientific abstract review.


METHODS
Retrospective analysis.


METHODS
Review for the 1991 Society of General Internal Medicine (SGIM) annual meeting.


METHODS
426 abstracts in seven topic categories evaluated by 55 reviewers.


METHODS
Reviewers rated abstracts from 1 (poor) to 5 (excellent), globally and on three specific dimensions: interest to the SGIM audience, quality of methods, and quality of presentation. Each abstract was reviewed by five to seven reviewers. Each reviewer's ratings of the three dimensions were added to compute that reviewer's summary score for a given abstract. The mean of all reviewers' summary scores for an abstract, the final score, was used by SGIM to select abstracts for the meeting.


RESULTS
Final scores ranged from 4.6 to 13.6 (mean = 9.9). Although 222 abstracts (52%) were accepted for publication, the 95% confidence interval around the final score of 300 (70.4%) of the 426 abstracts overlapped with the threshold for acceptance of an abstract. Thus, these abstracts were potentially misclassified. Only 36% of the variance in summary scores was associated with an abstract's identity, 12% with the reviewer's identity, and the remainder with idiosyncratic reviews of abstracts. Global ratings were more reproducible than summary scores.


CONCLUSIONS
Reviewers disagreed substantially when evaluating the same abstracts. Future meeting organizers may wish to rank abstracts using global ratings, and to experiment with structured review criteria and other ways to improve raters' agreement.
Rubin
H R
HR
Division of Internal Medicine, Johns Hopkins University, Baltimore, Maryland 21205.
Redelmeier
D A
DA
Wu
A W
AW
Steinberg
E P
EP
D000043:Abstracting and Indexing / Q000592:standards*; D000704:Analysis of Variance; D003657:Decision Making; D005069:Evaluation Studies as Topic; D006801:Humans; D007255:Information Services / Q000592:standards*; D007600:Judgment; D015588:Observer Variation; D010380:Peer Review / Q000592:standards*; D011643:Publishing / Q000592:standards*; D012044:Regression Analysis; D015203:Reproducibility of Results; D012189:Retrospective Studies