title: Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models.
abstract: Risk prediction models allow for the incorporation of individualized risk and clinical effectiveness information to identify patients for whom therapy is most appropriate and cost-effective. This approach has the potential to identify inefficient (or harmful) care in subgroups at different risks, even when the overall results appear favorable. Here, we explore the value of personalized risk information and the factors that influence it.
  
  
  
  Using an expected value of individualized care (EVIC) framework, which monetizes the value of customizing care, we developed a general approach to calculate individualized incremental cost effectiveness ratios (ICERs) as a function of individual outcome risk. For a case study (tPA v. streptokinase to treat possible myocardial infarction), we used a simulation to explore how an EVIC is influenced by population outcome prevalence, model discrimination (c-statistic) and calibration, and willingness-to-pay (WTP) thresholds.
  
  
  
  In our simulations, for well-calibrated models, which do not over- or underestimate predicted v. observed event risk, the EVIC ranged from $0 to $700 per person, with better discrimination (higher c-statistic values) yielding progressively higher EVIC values. For miscalibrated models, the EVIC ranged from -$600 to $600 in different simulated scenarios. The EVIC values decreased as discrimination improved from a c-statistic of 0.5 to 0.6, before becoming positive as the c-statistic reached values of ~0.8.
  
  
  
  Individualizing treatment decisions using risk may produce substantial value but also has the potential for net harm. Good model calibration ensures a non-negative EVIC. Improvements in discrimination generally increase the EVIC; however, when models are miscalibrated, greater discriminating power can paradoxically reduce the EVIC under some circumstances.
authors.affiliation: Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA (NO, JTC, PJN, JBW, DMK).
authors.forename: Natalia
authors.initials: N
authors.lastname: Olchanski
authors.affiliation: Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA (NO, JTC, PJN, JBW, DMK).
authors.forename: Joshua T
authors.initials: JT
authors.lastname: Cohen
authors.affiliation: Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA (NO, JTC, PJN, JBW, DMK).
authors.forename: Peter J
authors.initials: PJ
authors.lastname: Neumann
authors.affiliation: Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA (NO, JTC, PJN, JBW, DMK).|Division of Clinical Decision Making, Tufts Medical Center, Boston, MA (JBW).
authors.forename: John B
authors.initials: JB
authors.lastname: Wong
authors.affiliation: Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, MA (NO, JTC, PJN, JBW, DMK).
authors.forename: David M
authors.initials: DM
authors.lastname: Kent
mesh_terms: D003198:Computer Simulation; D017281:Cost of Illness; D003362:Cost-Benefit Analysis / Q000379:methods*; D003657:Decision Making; D003661:Decision Support Techniques; D005343:Fibrinolytic Agents / Q000191:economics / Q000627:therapeutic use; D006801:Humans; D009203:Myocardial Infarction / Q000188:drug therapy / Q000191:economics; D019057:Quality-Adjusted Life Years; D016032:Randomized Controlled Trials as Topic; D018570:Risk Assessment / Q000379:methods*; D013300:Streptokinase; D010959:Tissue Plasminogen Activator; D016896:Treatment Outcome
chemical_list: D005343:Fibrinolytic Agents; D013300:Streptokinase; D010959:Tissue Plasminogen Activator
keywords: group decision making; outcomes research; quality of care; translating research into practice