title: Internal validation of risk models in clustered data: a comparison of bootstrap schemes.
abstract: Internal validity of a risk model can be studied efficiently with bootstrapping to assess possible optimism in model performance. Assumptions of the regular bootstrap are violated when the development data are clustered. We compared alternative resampling schemes in clustered data for the estimation of optimism in model performance. A simulation study was conducted to compare regular resampling on only the patient level with resampling on only the cluster level and with resampling sequentially on both the cluster and patient levels (2-step approach). Optimism for the concordance index and calibration slope was estimated. Resampling of only patients or only clusters showed accurate estimates of optimism in model performance. The 2-step approach overestimated the optimism in model performance. If the number of centers or intraclass correlation coefficient was high, resampling of clusters showed more accurate estimates than resampling of patients. The 3 bootstrap schemes also were applied to empirical data that were clustered. The results presented in this paper support the use of resampling on only the clusters for estimation of optimism in model performance when data are clustered.
authors: Julius Center for Health Sciences and Primary Care, UMC Utrecht, P.O. Box 85500, 3508 GA Utrecht, The Netherlands.
authors: W
authors: W
authors: Bouwmeester
authors: K G M
authors: KG
authors: Moons
authors: T H
authors: TH
authors: Kappen
authors: W A
authors: WA
authors: van Klei
authors: J W R
authors: JW
authors: Twisk
authors: M J C
authors: MJ
authors: Eijkemans
authors: Y
authors: Y
authors: Vergouwe
mesh_terms: D003198:Computer Simulation; D006801:Humans; D015233:Models, Statistical; D020250:Postoperative Nausea and Vomiting; D012044:Regression Analysis; D018570:Risk Assessment; D054928:Validation Studies as Topic
keywords: bootstrapping; clustered data; internal validation; model performance; multilevel analysis; risk models