title: Agreement among reviewers of review articles.
abstract: OBJECTIVE
  To assess the consistency of an index of the scientific quality of research overviews.
  
  
  METHODS
  Agreement was measured among nine judges, each of whom assessed the scientific quality of 36 published review articles. ITEM SELECTION: An iterative process was used to select ten criteria relative to five key tasks entailed in conducting a research overview.
  
  
  METHODS
  The review articles were drawn from three sampling frames: articles highly rated by criteria external to the study; meta-analyses; and a broad spectrum of medical journals. JUDGES: Three categories of judges were used: research assistants; clinicians with research training; and experts in research methodology; with three judges in each category.
  
  
  RESULTS
  The level of agreement within the three groups of judges was similar for their overall assessment of scientific quality and for six of the nine other items. With four exceptions, agreement among judges within each group and across groups, as measured by the intraclass correlation coefficient (ICC), was greater than 0.5, and 60% (24/40) of the ICCs were greater than 0.7.
  
  
  CONCLUSIONS
  It was possible to achieve reasonable to excellent agreement for all of the items in the index, including the overall assessment of scientific quality. The implications of these results for practising clinicians and the peer review system are discussed.
authors.affiliation: Department of Clinical Epidemiology & Biostatistics, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada.
authors.forename: A D
authors.initials: AD
authors.lastname: Oxman
authors.forename: G H
authors.initials: GH
authors.lastname: Guyatt
authors.forename: J
authors.initials: J
authors.lastname: Singer
authors.forename: C H
authors.initials: CH
authors.lastname: Goldsmith
authors.forename: B G
authors.initials: BG
authors.lastname: Hutchison
authors.forename: R A
authors.initials: RA
authors.lastname: Milner
authors.forename: D L
authors.initials: DL
authors.lastname: Streiner
mesh_terms: D007255:Information Services / Q000592:standards; D015201:Meta-Analysis as Topic; D010380:Peer Review / Q000379:methods / Q000592:standards; D010506:Periodicals as Topic / Q000592:standards; D011786:Quality Control; D015203:Reproducibility of Results; D012106:Research / Q000592:standards; D012196:Review Literature as Topic