title: Factors affecting the performance of the models in the Mortality Probability Model II system and strategies of customization: a simulation study.
abstract: OBJECTIVE
  To examine the impact of hospital mortality and intensive care unit (ICU) size on the performance of the Mortality Probability Model II system for use in quality assessment, and to examine the ability of model customization to produce accurate estimates of hospital mortality to characterize patients by severity of illness for clinical trials.
  
  
  METHODS
  Prospective evaluation of model performance, using retrospective data.
  
  
  METHODS
  Data for the simulation were assembled from six adult medical and surgical ICUs in Massachusetts and New York.
  
  
  METHODS
  Consecutive admissions (n = 4,224) to the Massachusetts and New York ICUs were studied. The mortality rate in the database was 18.7%.
  
  
  METHODS
  A computer simulation of several different hospital mortality rates and ICU sample sizes, using a multicenter database of consecutive ICU admissions, was utilized. We simulated 20 different mortality rates by randomly changing the outcomes at hospital discharge from "survived" to "deceased" and from "deceased" to "survived". Four sample size simulations used 75%, 50%, 25%, and 10% of the database. Ten replications of each mortality rate and samples size were constructed, and model calibration and discrimination were assessed for each replication. Model coefficients were customized, using logistic regression.
  
  
  RESULTS
  Vital status at hospital discharge was the outcome measure among the ICU patient population. Model performance was assessed using the Hosmer-Lemeshow C statistic for calibration, and the area under the receiver operating characteristic curve for discrimination. Goodness-of-fit tests and receiver operating characteristic curve areas demonstrated that the models were sensitive to differences in hospital mortality, indicating that they are useful quality assurance tools. Goodness-of-fit tests were more sensitive than the receiver operating characteristic curve areas. The further the hospital mortality rate diverged from the original rate, the worse the performance of the model. Sample size had an impact on these results. The smaller the sample size, the less likely the model was to perform poorly. Model coefficients were successfully customized to demonstrate that improved model performance can be achieved when necessary for clinical trial stratification.
  
  
  CONCLUSIONS
  Mortality Probability Model II models can be used to assess quality of care in ICUs, but the size of the sample should be considered when assessing calibration and discrimination.
authors.affiliation: School of Public Health, University of Massachusetts, Amherst, USA.
authors.forename: B P
authors.initials: BP
authors.lastname: Zhu
authors.forename: S
authors.initials: S
authors.lastname: Lemeshow
authors.forename: D W
authors.initials: DW
authors.lastname: Hosmer
authors.forename: J
authors.initials: J
authors.lastname: Klar
authors.forename: J
authors.initials: J
authors.lastname: Avrunin
authors.forename: D
authors.initials: D
authors.lastname: Teres
mesh_terms: D000328:Adult; D003198:Computer Simulation; D006302:Health Services Research / Q000379:methods; D017052:Hospital Mortality; D006801:Humans; D007362:Intensive Care Units / Q000592:standards / Q000706:statistics & numerical data*; D016015:Logistic Models; D015233:Models, Statistical; D017063:Outcome Assessment, Health Care; D011785:Quality Assurance, Health Care; D012372:ROC Curve; D018401:Sample Size; D012720:Severity of Illness Index; D014481:United States