title: Multivariate meta-analysis of individual participant data helped externally validate the performance and implementation of a prediction model.
abstract: OBJECTIVE
  Our aim was to improve meta-analysis methods for summarizing a prediction model's performance when individual participant data are available from multiple studies for external validation.
  
  
  METHODS
  We suggest multivariate meta-analysis for jointly synthesizing calibration and discrimination performance, while accounting for their correlation. The approach estimates a prediction model's average performance, the heterogeneity in performance across populations, and the probability of "good" performance in new populations. This allows different implementation strategies (e.g., recalibration) to be compared. Application is made to a diagnostic model for deep vein thrombosis (DVT) and a prognostic model for breast cancer mortality.
  
  
  RESULTS
  In both examples, multivariate meta-analysis reveals that calibration performance is excellent on average but highly heterogeneous across populations unless the model's intercept (baseline hazard) is recalibrated. For the cancer model, the probability of "good" performance (defined by C statistic ≥0.7 and calibration slope between 0.9 and 1.1) in a new population was 0.67 with recalibration but 0.22 without recalibration. For the DVT model, even with recalibration, there was only a 0.03 probability of "good" performance.
  
  
  CONCLUSIONS
  Multivariate meta-analysis can be used to externally validate a prediction model's calibration and discrimination performance across multiple populations and to evaluate different implementation strategies.
authors.affiliation: Public Health, Epidemiology and Biostatistics, School of Health and Population Sciences, Public Health Building, University of Birmingham, Edgbaston, Birmingham B15 2TT, UK.
authors.forename: Kym I E
authors.initials: KI
authors.lastname: Snell
authors.affiliation: School of Mathematics, Watson Building, University of Birmingham, Edgbaston, Birmingham B15 2TT, UK.
authors.forename: Harry
authors.initials: H
authors.lastname: Hua
authors.affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Str. 6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands; Dutch Cochrane Centre, University Medical Center Utrecht, Str. 6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands.
authors.forename: Thomas P A
authors.initials: TP
authors.lastname: Debray
authors.affiliation: Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire ST5 5BG, UK.
authors.forename: Joie
authors.initials: J
authors.lastname: Ensor
authors.affiliation: Department of Medical Oncology, Erasmus MC Cancer Institute, Erasmus University Medical Center, PO Box 2040, 3000 CA Rotterdam, The Netherlands.
authors.forename: Maxime P
authors.initials: MP
authors.lastname: Look
authors.affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Str. 6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands; Dutch Cochrane Centre, University Medical Center Utrecht, Str. 6.131, PO Box 85500, 3508 GA Utrecht, The Netherlands.
authors.forename: Karel G M
authors.initials: KG
authors.lastname: Moons
authors.affiliation: Research Institute for Primary Care and Health Sciences, Keele University, Staffordshire ST5 5BG, UK. Electronic address: r.riley@keele.ac.uk.
authors.forename: Richard D
authors.initials: RD
authors.lastname: Riley
mesh_terms: D005260:Female; D006801:Humans; D008297:Male; D015201:Meta-Analysis as Topic; D015233:Models, Statistical; D015999:Multivariate Analysis; D011379:Prognosis; D054928:Validation Studies as Topic; D020246:Venous Thrombosis
keywords: Calibration; Discrimination; External validation; Heterogeneity; Individual participant data (IPD); Model comparison; Multivariate meta-analysis; Prognostic model; Risk prediction