Quality assessment of reviewers' reports using a simple instrument.
OBJECTIVE
To validate and test a simple instrument for assessing the quality of a review.


METHODS
In this prospective observational study, the quality of 247 reviews of 119 original articles submitted to the Dutch Journal of Medicine was assessed using a 5-point scale that has been used for years by Obstetrics & Gynecology. Each review was assessed by three editors of the journal. Intraobserver variability, calculated as an intraclass correlation coefficient, was assessed by having the same editors rate 76 reviews for a second time. Validation of the scale was done in two ways. First, editors of three other medical journals were asked to rate the 247 reviews using the same 5-point scale. Second, all reviews were sent to the authors of the article with a questionnaire consisting of 12 yes-or-no questions and one question asking for an overall score for the review.


RESULTS
The interobserver intraclass correlation coefficient for the three editors was 0.62 (95% confidence interval 0.50-0.71) for the first assessment of 247 reviews. For the second assessment of 76 reviews, the interobserver intraclass correlation coefficient was 0.62 (0.45-0.74). The intraobserver intraclass correlation coefficient for each of the internal editors ranged from 0.66 to 0.88. The interobserver intraclass correlation coefficient for the external editors was 0.60 (0.51-0.68). The interobserver intraclass correlation coefficient for all six editors was 0.62 (0.55-0.68). The authors' response rate to the questionnaires was 83%. A significant correlation was found between the mean total editorial quality assessment and the overall score of the authors (intraclass correlation coefficient 0.28, 0.14-0.41).


CONCLUSIONS
This 5-point scale proved to be a simple, reliable, and valid instrument enabling editors to assess the quality of reviews. A significant correlation was found between mean editorial quality assessment and the quality as determined by authors.


METHODS
III.
Landkroon
Annemieke P
AP
Nederlands Tijdschrift voor Geneeskunde (Dutch Journal of Medicine), the Netherlands. oplandkroon@gmail.com
Euser
Anne Margriet
AM
Veeken
Hans
H
Hart
Wimpeter
W
Overbeke
A John P M
AJ
D006801:Humans; D015588:Observer Variation; D017712:Peer Review, Research / Q000379:methods; D011446:Prospective Studies; D011643:Publishing / Q000592:standards* / Q000706:statistics & numerical data; D011786:Quality Control; D011795:Surveys and Questionnaires