title: Simple dichotomous updating methods improved the validity of polytomous prediction models.
abstract: OBJECTIVE
  Prediction models may perform poorly in a new setting. We aimed to determine which model updating methods should be applied for models predicting polytomous outcomes, which often suffer from one or more categories with low prevalence.
  
  
  METHODS
  We used case studies on testicular and ovarian tumors. The original regression models were based on 502 and 2,037 patients and validated on 273 and 1,107 patients, respectively. The polytomous models combined dichotomous models for category A vs. B + C and B vs. C (sequential dichotomous modeling). Simple recalibration, revision, and redevelopment methods were considered. To assess discrimination (using dichotomous and polytomous c-statistics) and calibration (by comparing observed and expected prevalences) of these methods, the validation data were divided into updating and test parts. Five hundred such divisions were randomly generated, and the average test set results reported.
  
  
  RESULTS
  None of the updating methods could improve discrimination of the original models, but recalibration, revision, and redevelopment strongly improved calibration. Redevelopment was unstable with respect to overfitting and performance.
  
  
  CONCLUSIONS
  Simple dichotomous updating methods behaved well when applied to polytomous models. Our results suggest that recalibration is preferred, but larger validation sets may make revision or redevelopment a sensible alternative.
authors.affiliation: Department of Electrical Engineering ESAT-SCD, KU Leuven-University of Leuven, Leuven, Belgium.
authors.forename: Kirsten
authors.initials: K
authors.lastname: Van Hoorde
authors.forename: Yvonne
authors.initials: Y
authors.lastname: Vergouwe
authors.forename: Dirk
authors.initials: D
authors.lastname: Timmerman
authors.forename: Sabine
authors.initials: S
authors.lastname: Van Huffel
authors.forename: Ewout W
authors.initials: EW
authors.lastname: Steyerberg
authors.forename: Ben
authors.initials: B
authors.lastname: Van Calster
mesh_terms: D000328:Adult; D002138:Calibration; D005260:Female; D006801:Humans; D008297:Male; D008875:Middle Aged; D015233:Models, Statistical; D018365:Neoplasm, Residual; D010051:Ovarian Neoplasms / Q000473:pathology*; D015203:Reproducibility of Results; D013736:Testicular Neoplasms / Q000188:drug therapy / Q000473:pathology*; D016896:Treatment Outcome
keywords: Calibration; Discrimination; Model updating; Polytomous outcomes; Prediction models; Sequential dichotomous modeling