title: The prediction accuracy of dynamic mixed-effects models in clustered data.
abstract: BACKGROUND
  Clinical prediction models often fail to generalize in the context of clustered data, because most models fail to account for heterogeneity in outcome values and covariate effects across clusters. Furthermore, standard approaches for modeling clustered data, including generalized linear mixed-effects models, would not be expected to provide accurate predictions in novel clusters, because such predictions are typically based on the hypothetical mean cluster. We hypothesized that dynamic mixed-effects models, which incorporate data from previous predictions to refine the model for future predictions, would allow for cluster-specific predictions in novel clusters as the model is updated over time, thus improving overall model generalizability.
  
  
  RESULTS
  We quantified the potential gains in prediction accuracy from using a dynamic modeling strategy in a simulation study. Furthermore, because clinical prediction models in the context of clustered data often involve outcomes that are dependent on patient volume, we examined whether using dynamic mixed-effects models would be robust to misspecification of the volume-outcome relationship. Our results indicated that dynamic mixed-effects models led to substantial improvements in prediction accuracy in clustered populations over a broad range of conditions, and were uniformly superior to static models. In addition, dynamic mixed-effects models were particularly robust to misspecification of the volume-outcome relationship and to variation in the frequency of model updating. The extent of the improvement in prediction accuracy that was observed with dynamic mixed-effects models depended on the relative impact of fixed and random effects on the outcome as well as the degree of misspecification of model fixed effects.
  
  
  CONCLUSIONS
  Dynamic mixed-effects models led to substantial improvements in prediction model accuracy across a broad range of simulated conditions. Therefore, dynamic mixed-effects models could be a useful alternative to standard static models for improving the generalizability of clinical prediction models in the setting of clustered data, and, thus, well worth the logistical challenges that may accompany their implementation in practice.
authors.affiliation: Center for Clinical Epidemiology and Biostatistics, Department of Biostatistics and Epidemiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA USA ; Center for Therapeutic Effectiveness Research, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA USA.
authors.forename: Brian S
authors.identifier: 0000-0002-4348-8208
authors.initials: BS
authors.lastname: Finkelman
authors.affiliation: Center for Clinical Epidemiology and Biostatistics, Department of Biostatistics and Epidemiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA USA.
authors.forename: Benjamin
authors.initials: B
authors.lastname: French
authors.affiliation: Center for Clinical Epidemiology and Biostatistics, Department of Biostatistics and Epidemiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA USA ; Center for Therapeutic Effectiveness Research, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA USA ; Department of Medicine, Cardiovascular Division, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA USA.
authors.forename: Stephen E
authors.initials: SE
authors.lastname: Kimmel
keywords: Bayesian statistics; Clustered data; Dynamic modeling; Generalizability; Mixed-effects models; Prediction
references.citation: N Engl J Med. 2002 Apr 11;346(15):1128-37
references.pmid: 11948273
references.citation: Biom J. 2009 Feb;51(1):19-30
references.pmid: 19197956
references.citation: Ann Intern Med. 2005 Jul 19;143(2):100-7
references.pmid: 16027451
references.citation: Arch Surg. 2003 Jul;138(7):721-5; discussion 726
references.pmid: 12860752
references.citation: J Thorac Dis. 2015 Apr;7(Suppl 2):S152-5
references.pmid: 25984361
references.citation: Ann Surg Oncol. 2014 Aug;21(8):2659-66
references.pmid: 24748161
references.citation: Stat Methods Med Res. 2016 Feb;25(1):255-71
references.pmid: 22767865
references.citation: J Clin Epidemiol. 2006 Oct;59(10):1102-9
references.pmid: 16980151
references.citation: Circ Cardiovasc Qual Outcomes. 2013 Nov;6(6):649-58
references.pmid: 24150044
references.citation: Eur J Cardiothorac Surg. 2006 Apr;29(4):441-6; discussion 446
references.pmid: 16473519
references.citation: N Engl J Med. 2006 Jul 6;355(1):41-50
references.pmid: 16822995
references.citation: Health Serv Res. 2010 Oct;45(5 Pt 1):1148-67
references.pmid: 20579125
references.citation: J R Stat Soc Ser A Stat Soc. 2009 Oct 1;172(4):729-748
references.pmid: 20069131
references.citation: Am J Epidemiol. 2012 Feb 1;175(3):210-7
references.pmid: 22262640
references.citation: Biometrics. 2011 Mar;67(1):270-9
references.pmid: 20528860
references.citation: Eur J Radiol. 2014 Sep;83(9):1639-44
references.pmid: 24972452
references.citation: Am J Respir Crit Care Med. 2014 Sep 15;190(6):665-74
references.pmid: 25117723
references.citation: J Clin Epidemiol. 2006 Oct;59(10):1092-101
references.pmid: 16980150
references.citation: Am J Cardiol. 2015 Sep 1;116(5):791-800
references.pmid: 26100585
references.citation: Biometrics. 2012 Mar;68(1):23-30
references.pmid: 21838812
references.citation: Am J Manag Care. 2011 Nov;17(11):729-36
references.pmid: 22084892
references.citation: Liver Transpl. 2013 Sep;19(9):957-64
references.pmid: 23784730
references.citation: Arch Intern Med. 2008 Jul 14;168(13):1371-86
references.pmid: 18625917
references.citation: Stat Med. 2012 Feb 20;31(4):366-82
references.pmid: 22086835