title: Design Characteristics Influence Performance of Clinical Prediction Rules in Validation: A Meta-Epidemiological Study.
abstract: BACKGROUND
  Many new clinical prediction rules are derived and validated. But the design and reporting quality of clinical prediction research has been less than optimal. We aimed to assess whether design characteristics of validation studies were associated with the overestimation of clinical prediction rules' performance. We also aimed to evaluate whether validation studies clearly reported important methodological characteristics.
  
  
  METHODS
  Electronic databases were searched for systematic reviews of clinical prediction rule studies published between 2006 and 2010. Data were extracted from the eligible validation studies included in the systematic reviews. A meta-analytic meta-epidemiological approach was used to assess the influence of design characteristics on predictive performance. From each validation study, it was assessed whether 7 design and 7 reporting characteristics were properly described.
  
  
  RESULTS
  A total of 287 validation studies of clinical prediction rule were collected from 15 systematic reviews (31 meta-analyses). Validation studies using case-control design produced a summary diagnostic odds ratio (DOR) 2.2 times (95% CI: 1.2-4.3) larger than validation studies using cohort design and unclear design. When differential verification was used, the summary DOR was overestimated by twofold (95% CI: 1.2 -3.1) compared to complete, partial and unclear verification. The summary RDOR of validation studies with inadequate sample size was 1.9 (95% CI: 1.2 -3.1) compared to studies with adequate sample size. Study site, reliability, and clinical prediction rule was adequately described in 10.1%, 9.4%, and 7.0% of validation studies respectively.
  
  
  CONCLUSIONS
  Validation studies with design shortcomings may overestimate the performance of clinical prediction rules. The quality of reporting among studies validating clinical prediction rules needs to be improved.
authors.affiliation: Evidence-Based Health Care Programme, Department of Continuing Education, Kellogg College, University of Oxford, Oxford, United Kingdom.
authors.forename: Jong-Wook
authors.initials: JW
authors.lastname: Ban
authors.affiliation: CASPe, CIBER-ESP, Clinical Epidemiology Unit, Hospital Universitario Donostia, San Sebastian, Spain.
authors.forename: José Ignacio
authors.initials: JI
authors.lastname: Emparanza
authors.affiliation: CASPe, CIBER-ESP, Clinical Epidemiology Unit, Hospital Universitario Donostia, San Sebastian, Spain.
authors.forename: Iratxe
authors.initials: I
authors.lastname: Urreta
authors.affiliation: School of Health Sciences, City University London, London, United Kingdom.
authors.forename: Amanda
authors.initials: A
authors.lastname: Burls
mesh_terms: D035843:Biomedical Research / Q000379:methods / Q000592:standards / Q000706:statistics & numerical data; D003661:Decision Support Techniques; D016021:Epidemiologic Studies; D006801:Humans; D016017:Odds Ratio; D015203:Reproducibility of Results; D012107:Research Design / Q000592:standards*; D058028:Research Report / Q000592:standards*; D018401:Sample Size; D054928:Validation Studies as Topic