title: Does ignoring clustering in multicenter data influence the performance of prediction models? A simulation study.
abstract: Clinical risk prediction models are increasingly being developed and validated on multicenter datasets. In this article, we present a comprehensive framework for the evaluation of the predictive performance of prediction models at the center level and the population level, considering population-averaged predictions, center-specific predictions, and predictions assuming an average random center effect. We demonstrated in a simulation study that calibration slopes do not only deviate from one because of over- or underfitting of patterns in the development dataset, but also as a result of the choice of the model (standard versus mixed effects logistic regression), the type of predictions (marginal versus conditional versus assuming an average random effect), and the level of model validation (center versus population). In particular, when data is heavily clustered (ICC 20%), center-specific predictions offer the best predictive performance at the population level and the center level. We recommend that models should reflect the data structure, while the level of model validation should reflect the research question.
authors.affiliation: 1 KU Leuven Department of Electrical Engineering (ESAT), STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, Leuven, Belgium.|2 KU Leuven iMinds Department Medical Information Technologies, Leuven, Belgium.
authors.forename: L
authors.initials: L
authors.lastname: Wynants
authors.affiliation: 3 Center for Medical Decision Sciences, Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.
authors.forename: Y
authors.initials: Y
authors.lastname: Vergouwe
authors.affiliation: 1 KU Leuven Department of Electrical Engineering (ESAT), STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, Leuven, Belgium.|2 KU Leuven iMinds Department Medical Information Technologies, Leuven, Belgium.
authors.forename: S
authors.initials: S
authors.lastname: Van Huffel
authors.affiliation: 4 KU Leuven Department of Development and Regeneration, Leuven, Belgium.
authors.forename: D
authors.initials: D
authors.lastname: Timmerman
authors.affiliation: 3 Center for Medical Decision Sciences, Department of Public Health, Erasmus Medical Center, Rotterdam, The Netherlands.|4 KU Leuven Department of Development and Regeneration, Leuven, Belgium.
authors.forename: B
authors.initials: B
authors.lastname: Van Calster
mesh_terms: D000465:Algorithms; D035843:Biomedical Research / Q000706:statistics & numerical data; D016000:Cluster Analysis; D016015:Logistic Models; D015337:Multicenter Studies as Topic
keywords: Mixed model; bias; calibration; clinical prediction model; discrimination; logistic regression; predictive performance