Agreement among reviewers of review articles.
OBJECTIVE
To assess the consistency of an index of the scientific quality of research overviews.


METHODS
Agreement was measured among nine judges, each of whom assessed the scientific quality of 36 published review articles. ITEM SELECTION: An iterative process was used to select ten criteria relative to five key tasks entailed in conducting a research overview.


METHODS
The review articles were drawn from three sampling frames: articles highly rated by criteria external to the study; meta-analyses; and a broad spectrum of medical journals. JUDGES: Three categories of judges were used: research assistants; clinicians with research training; and experts in research methodology; with three judges in each category.


RESULTS
The level of agreement within the three groups of judges was similar for their overall assessment of scientific quality and for six of the nine other items. With four exceptions, agreement among judges within each group and across groups, as measured by the intraclass correlation coefficient (ICC), was greater than 0.5, and 60% (24/40) of the ICCs were greater than 0.7.


CONCLUSIONS
It was possible to achieve reasonable to excellent agreement for all of the items in the index, including the overall assessment of scientific quality. The implications of these results for practising clinicians and the peer review system are discussed.
Oxman
A D
AD
Department of Clinical Epidemiology & Biostatistics, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada.
Guyatt
G H
GH
Singer
J
J
Goldsmith
C H
CH
Hutchison
B G
BG
Milner
R A
RA
Streiner
D L
DL
D007255:Information Services / Q000592:standards; D015201:Meta-Analysis as Topic; D010380:Peer Review / Q000379:methods / Q000592:standards; D010506:Periodicals as Topic / Q000592:standards; D011786:Quality Control; D015203:Reproducibility of Results; D012106:Research / Q000592:standards; D012196:Review Literature as Topic