title: Testing a tool for the classification of study designs in systematic reviews of interventions and exposures showed moderate reliability and low accuracy.
abstract: OBJECTIVE
  To develop and test a study design classification tool.
  
  
  METHODS
  We contacted relevant organizations and individuals to identify tools used to classify study designs and ranked these using predefined criteria. The highest ranked tool was a design algorithm developed, but no longer advocated, by the Cochrane Non-Randomized Studies Methods Group; this was modified to include additional study designs and decision points. We developed a reference classification for 30 studies; 6 testers applied the tool to these studies. Interrater reliability (Fleiss' κ) and accuracy against the reference classification were assessed. The tool was further revised and retested.
  
  
  RESULTS
  Initial reliability was fair among the testers (κ=0.26) and the reference standard raters κ=0.33). Testing after revisions showed improved reliability (κ=0.45, moderate agreement) with improved, but still low, accuracy. The most common disagreements were whether the study design was experimental (5 of 15 studies), and whether there was a comparison of any kind (4 of 15 studies). Agreement was higher among testers who had completed graduate level training versus those who had not.
  
  
  CONCLUSIONS
  The moderate reliability and low accuracy may be because of lack of clarity and comprehensiveness of the tool, inadequate reporting of the studies, and variability in tester characteristics. The results may not be generalizable to all published studies, as the test studies were selected because they had posed challenges for previous reviewers with respect to their design classification. Application of such a tool should be accompanied by training, pilot testing, and context-specific decision rules.
authors.affiliation: Department of Pediatrics, Alberta Research Center for Health Evidence and the University of Alberta Evidence-based Practice Center, University of Alberta, 11402 University Avenue, Edmonton, Alberta, Canada. hartling@ualberta.ca
authors.forename: Lisa
authors.initials: L
authors.lastname: Hartling
authors.forename: Kenneth
authors.initials: K
authors.lastname: Bond
authors.forename: P Lina
authors.initials: PL
authors.lastname: Santaguida
authors.forename: Meera
authors.initials: M
authors.lastname: Viswanathan
authors.forename: Donna M
authors.initials: DM
authors.lastname: Dryden
mesh_terms: D032921:Consensus; D019317:Evidence-Based Medicine / Q000145:classification*; D006801:Humans; D015588:Observer Variation; D017712:Peer Review, Research / Q000592:standards*; D011786:Quality Control; D012015:Reference Standards; D015203:Reproducibility of Results; D012107:Research Design / Q000592:standards*; D012196:Review Literature as Topic