title: Interrater reliability in assessing quality of diagnostic accuracy studies using the QUADAS tool. A preliminary assessment.
abstract: OBJECTIVE
  Quality Assessment of Diagnostic Accuracy Studies (QUADAS) is a new tool to measure the methodological quality of diagnostic accuracy studies in systematic reviews. We used data from a systematic review of magnetic resonance spectroscopy (MRS) in the characterization of suspected brain tumors to provide a preliminary evaluation of the inter-rater reliability of QUADAS.
  
  
  METHODS
  A structured literature search identified 19 diagnostic accuracy studies. These publications were distributed randomly to primary and secondary reviewers for dual independent assessment. Reviewers recorded methodological quality by using QUADAS on a custom-designed spreadsheet. We calculated correlation, percentage of agreement, and kappa statistic to assess inter-rater reliability.
  
  
  RESULTS
  Most studies in our review were judged to have used an accurate reference standard. Conversely, the MRS literature frequently failed to specify the length of time between index and reference tests or that the clinicians were unaware of the index test findings when reporting the reference standard. There was good correlation (rho = 0.78) between reviewers in assessment of the overall number of quality criteria met. However, mean agreement for individual QUADAS questions was only fair (kappa = 0.22) and ranged from no agreement beyond chance (kappa < 0) to moderate agreement (kappa = 0.58).
  
  
  CONCLUSIONS
  Inter-rater reliability in our study was relatively low. Nevertheless, we believe that QUADAS potentially is a useful tool for highlighting the strengths and weaknesses of existing diagnostic accuracy studies. Low reliability suggests that different reviewers will reach different conclusions if QUADAS is used to exclude "low-quality" articles from meta-analyses. We discuss methods for improving the validity and reliability of QUADAS.
authors.affiliation: Department of Radiology, Box 359960, 325 Ninth Avenue, Seattle, WA 98104-2499, USA. willh@u.washington.edu
authors.forename: William
authors.initials: W
authors.lastname: Hollingworth
authors.forename: L Santiago
authors.initials: LS
authors.lastname: Medina
authors.forename: Robert E
authors.initials: RE
authors.lastname: Lenkinski
authors.forename: Dean K
authors.initials: DK
authors.lastname: Shibata
authors.forename: Byron
authors.initials: B
authors.lastname: Bernal
authors.forename: David
authors.initials: D
authors.lastname: Zurakowski
authors.forename: Bryan
authors.initials: B
authors.lastname: Comstock
authors.forename: Jeffrey G
authors.initials: JG
authors.lastname: Jarvik
mesh_terms: D001932:Brain Neoplasms / Q000175:diagnosis*; D032921:Consensus; D003954:Diagnostic Services / Q000592:standards*; D019317:Evidence-Based Medicine; D006801:Humans; D009682:Magnetic Resonance Spectroscopy / Q000592:standards*; D015588:Observer Variation; D017712:Peer Review, Research; D011786:Quality Control; D015203:Reproducibility of Results; D012196:Review Literature as Topic; D011795:Surveys and Questionnaires / Q000592:standards*