title: Empirical evidence of design-related bias in studies of diagnostic tests.
abstract: BACKGROUND
  The literature contains a large number of potential biases in the evaluation of diagnostic tests. Strict application of appropriate methodological criteria would invalidate the clinical application of most study results.
  
  
  OBJECTIVE
  To empirically determine the quantitative effect of study design shortcomings on estimates of diagnostic accuracy.
  
  
  METHODS
  Observational study of the methodological features of 184 original studies evaluating 218 diagnostic tests. Meta-analyses on diagnostic tests were identified through a systematic search of the literature using MEDLINE, EMBASE, and DARE databases and the Cochrane Library (1996-1997). Associations between study characteristics and estimates of diagnostic accuracy were evaluated with a regression model.
  
  
  METHODS
  Relative diagnostic odds ratio (RDOR), which compared the diagnostic odds ratios of studies of a given test that lacked a particular methodological feature with those without the corresponding shortcomings in design.
  
  
  RESULTS
  Fifteen (6.8%) of 218 evaluations met all 8 criteria; 64 (30%) met 6 or more. Studies evaluating tests in a diseased population and a separate control group overestimated the diagnostic performance compared with studies that used a clinical population (RDOR, 3.0; 95% confidence interval [CI], 2.0-4.5). Studies in which different reference tests were used for positive and negative results of the test under study overestimated the diagnostic performance compared with studies using a single reference test for all patients (RDOR, 2.2; 95% CI, 1.5-3.3). Diagnostic performance was also overestimated when the reference test was interpreted with knowledge of the test result (RDOR, 1.3; 95% CI, 1.0-1.9), when no criteria for the test were described (RDOR, 1.7; 95% CI, 1.1-2.5), and when no description of the population under study was provided (RDOR, 1.4; 95% CI, 1.1-1.7).
  
  
  CONCLUSIONS
  These data provide empirical evidence that diagnostic studies with methodological shortcomings may overestimate the accuracy of a diagnostic test, particularly those including nonrepresentative patients or applying different reference standards.
authors.affiliation: Department of Clinical Epidemiology and Biostatistics, Academic Medical Center, University of Amsterdam, The Netherlands. j.g.lijmer@amc.uva.nl
authors.forename: J G
authors.initials: JG
authors.lastname: Lijmer
authors.forename: B W
authors.initials: BW
authors.lastname: Mol
authors.forename: S
authors.initials: S
authors.lastname: Heisterkamp
authors.forename: G J
authors.initials: GJ
authors.lastname: Bonsel
authors.forename: M H
authors.initials: MH
authors.lastname: Prins
authors.forename: J H
authors.initials: JH
authors.lastname: van der Meulen
authors.forename: P M
authors.initials: PM
authors.lastname: Bossuyt
mesh_terms: D015982:Bias; D003955:Diagnostic Tests, Routine; D005069:Evaluation Studies as Topic; D016017:Odds Ratio; D011786:Quality Control; D012015:Reference Standards; D012044:Regression Analysis; D012107:Research Design / Q000592:standards; D012680:Sensitivity and Specificity