title: Impact of choice of quality appraisal tool for systematic reviews in overviews.
abstract: OBJECTIVE
  The question whether the choice of a critical appraisal tool has an impact on the result of the evidence synthesis in systematic reviews has been neglected by research. This is also true for psychometric properties of critical appraisal tools. The objective of the study is to exemplify that in the context of overviews (reviews of reviews).
  
  
  METHODS
  Based on a published overview investigating the hospital volume-outcome relationship in surgery, 32 therein included systematic reviews were independently evaluated with four critical appraisal tools by two reviewers. We rated the relationship on a five-point rating scale using qualitative evidence synthesis. Measures of reliability and correlation coefficients were calculated.
  
  
  RESULTS
  The result of the evidence synthesis was not dependent on the choice of a critical appraisal tool. Inter-rater reliability differed depending on the tool, Cohens Kappa ranging from 0.47 to 0.76. There was a high heterogeneity between the two pairs of reviewers.
  
  
  CONCLUSIONS
  The choice of a critical appraisal tool has no impact on the result of the evidence synthesis, despite differences in the covered components by each CAT. Further studies should concentrate on investigating psychometric properties and the impact of choice of CATs on the evidence synthesis in other contexts. The high heterogeneity between the two pairs of reviewers, all of them experienced in appraising systematic reviews, indicates a degree of interpretability in the items.
authors.affiliation: Institute for Research in Operative Medicine, Witten/Herdecke University, D- 51109 Cologne, Germany.
authors.forename: Dawid
authors.initials: D
authors.lastname: Pieper
authors.forename: Tim
authors.initials: T
authors.lastname: Mathes
authors.forename: Michaela
authors.initials: M
authors.lastname: Eikermann
mesh_terms: D005069:Evaluation Studies as Topic; D015588:Observer Variation; D012196:Review Literature as Topic; D011795:Surveys and Questionnaires
keywords: Evidence‐based Medicine; observer variation; psychometrics; research design; review