title: Appraising the quality of randomized controlled trials: inter-rater reliability for the OTseeker evidence database.
abstract: OBJECTIVE
  'OTseeker' is an online database of randomized controlled trials (RCTs) and systematic reviews relevant to occupational therapy. RCTs are critically appraised and rated for quality using the 'PEDro' scale. We aimed to investigate the inter-rater reliability of the PEDro scale before and after revising rating guidelines.
  
  
  METHODS
  In study 1, five raters scored 100 RCTs using the original PEDro scale guidelines. In study 2, two raters scored 40 different RCTs using revised guidelines. All RCTs were randomly selected from the OTseeker database. Reliability was calculated using Kappa and intraclass correlation coefficients [ICC (model 2,1)].
  
  
  RESULTS
  Inter-rater reliability was 'good to excellent' in the first study (Kappas >or= 0.53; ICCs >or= 0.71). After revising the rating guidelines, the reliability levels were equivalent or higher to those previously obtained (Kappas >or= 0.53; ICCs >or= 0.89), except for the item, 'groups similar at baseline', which still had moderate reliability (Kappa = 0.53). In study 2, two PEDro scale items, which had their definitions revised, 'less than 15% dropout' and 'point measures and variability', showed higher reliability. In both studies, the PEDro items with the lowest reliability were 'groups similar at baseline' (Kappas = 0.53), 'less than 15% dropout' (Kappas <or= 0.68) and 'point measures and variability data' (Kappas <or= 0.68).
  
  
  CONCLUSIONS
  The PEDro scale is a reliable instrument for rating the quality of RCTs. Revised rating guidelines are provided for scale items that are difficult to rate, and helped to improve inter-rater reliability.
authors.affiliation: School of Population Health, University of Queensland, QLD, Australia. L.Tooth@sph.uq.edu.au
authors.forename: Leigh
authors.initials: L
authors.lastname: Tooth
authors.forename: Sally
authors.initials: S
authors.lastname: Bennett
authors.forename: Annie
authors.initials: A
authors.lastname: McCluskey
authors.forename: Tammy
authors.initials: T
authors.lastname: Hoffmann
authors.forename: Kryss
authors.initials: K
authors.lastname: McKenna
authors.forename: Meryl
authors.initials: M
authors.lastname: Lovarini
mesh_terms: D001315:Australia; D006801:Humans; D020407:Internet; D009313:National Health Programs; D009788:Occupational Therapy; D016032:Randomized Controlled Trials as Topic / Q000145:classification* / Q000592:standards