title: A simulation study of sample size demonstrated the importance of the number of events per variable to develop prediction models in clustered data.
abstract: OBJECTIVE
  This study aims to investigate the influence of the amount of clustering [intraclass correlation (ICC) = 0%, 5%, or 20%], the number of events per variable (EPV) or candidate predictor (EPV = 5, 10, 20, or 50), and backward variable selection on the performance of prediction models.
  
  
  METHODS
  Researchers frequently combine data from several centers to develop clinical prediction models. In our simulation study, we developed models from clustered training data using multilevel logistic regression and validated them in external data.
  
  
  RESULTS
  The amount of clustering was not meaningfully associated with the models' predictive performance. The median calibration slope of models built in samples with EPV = 5 and strong clustering (ICC = 20%) was 0.71. With EPV = 5 and ICC = 0%, it was 0.72. A higher EPV related to an increased performance: the calibration slope was 0.85 at EPV = 10 and ICC = 20% and 0.96 at EPV = 50 and ICC = 20%. Variable selection sometimes led to a substantial relative bias in the estimated predictor effects (up to 118% at EPV = 5), but this had little influence on the model's performance in our simulations.
  
  
  CONCLUSIONS
  We recommend at least 10 EPV to fit prediction models in clustered data using logistic regression. Up to 50 EPV may be needed when variable selection is performed.
authors.affiliation: KU Leuven Department of Electrical Engineering-ESAT, STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, Kasteelpark Arenberg 10, Box 2446, Leuven 3001, Belgium; KU Leuven iMinds Medical IT Department, Kasteelpark Arenberg 10, Box 2446, Leuven 3001, Belgium. Electronic address: Laure.wynants@esat.kuleuven.be.
authors.forename: L
authors.initials: L
authors.lastname: Wynants
authors.affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands; Pharmerit B.V., Marten Meesweg 107, Rotterdam 3068 AV, The Netherlands.
authors.forename: W
authors.initials: W
authors.lastname: Bouwmeester
authors.affiliation: Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands.
authors.forename: K G M
authors.initials: KG
authors.lastname: Moons
authors.affiliation: Department of Methodology and Statistics, Utrecht University, Padualaan 14, 3584 CH Utrecht, The Netherlands.
authors.forename: M
authors.initials: M
authors.lastname: Moerbeek
authors.affiliation: KU Leuven Department of Development and Regeneration, Herestraat 49 Box 7003, Leuven 3000, Belgium; Department of Obstetrics and Gynaecology, University Hospitals Leuven, Herestraat 49, 3000 Leuven, Belgium.
authors.forename: D
authors.initials: D
authors.lastname: Timmerman
authors.affiliation: KU Leuven Department of Electrical Engineering-ESAT, STADIUS Center for Dynamical Systems, Signal Processing and Data Analytics, Kasteelpark Arenberg 10, Box 2446, Leuven 3001, Belgium; KU Leuven iMinds Medical IT Department, Kasteelpark Arenberg 10, Box 2446, Leuven 3001, Belgium.
authors.forename: S
authors.initials: S
authors.lastname: Van Huffel
authors.affiliation: KU Leuven Department of Development and Regeneration, Herestraat 49 Box 7003, Leuven 3000, Belgium; Center for Medical Decision Sciences, Department of Public Health, Erasmus Medical Center, Wytemaweg 80, 3015 CN Rotterdam, The Netherlands.
authors.forename: B
authors.initials: B
authors.lastname: Van Calster
authors.affiliation: Center for Medical Decision Sciences, Department of Public Health, Erasmus Medical Center, Wytemaweg 80, 3015 CN Rotterdam, The Netherlands.
authors.forename: Y
authors.initials: Y
authors.lastname: Vergouwe
mesh_terms: D015982:Bias; D016000:Cluster Analysis; D003198:Computer Simulation; D003625:Data Collection / Q000706:statistics & numerical data*; D003661:Decision Support Techniques; D005260:Female; D006801:Humans; D016015:Logistic Models; D015233:Models, Statistical; D010051:Ovarian Neoplasms / Q000175:diagnosis* / Q000453:epidemiology*; D012044:Regression Analysis; D018401:Sample Size; D013223:Statistics as Topic
keywords: Clustered data; Events per variable; Logistic model; Multicenter study; Prediction model; Simulation study