title: Pitfalls in Prediction Modeling for Normal Tissue Toxicity in Radiation Therapy: An Illustration With the Individual Radiation Sensitivity and Mammary Carcinoma Risk Factor Investigation Cohorts.
abstract: To identify the main causes underlying the failure of prediction models for radiation therapy toxicity to replicate.
  
  
  
  Data were used from two German cohorts, Individual Radiation Sensitivity (ISE) (n=418) and Mammary Carcinoma Risk Factor Investigation (MARIE) (n=409), of breast cancer patients with similar characteristics and radiation therapy treatments. The toxicity endpoint chosen was telangiectasia. The LASSO (least absolute shrinkage and selection operator) logistic regression method was used to build a predictive model for a dichotomized endpoint (Radiation Therapy Oncology Group/European Organization for the Research and Treatment of Cancer score 0, 1, or ≥2). Internal areas under the receiver operating characteristic curve (inAUCs) were calculated by a naïve approach whereby the training data (ISE) were also used for calculating the AUC. Cross-validation was also applied to calculate the AUC within the same cohort, a second type of inAUC. Internal AUCs from cross-validation were calculated within ISE and MARIE separately. Models trained on one dataset (ISE) were applied to a test dataset (MARIE) and AUCs calculated (exAUCs).
  
  
  
  Internal AUCs from the naïve approach were generally larger than inAUCs from cross-validation owing to overfitting the training data. Internal AUCs from cross-validation were also generally larger than the exAUCs, reflecting heterogeneity in the predictors between cohorts. The best models with largest inAUCs from cross-validation within both cohorts had a number of common predictors: hypertension, normalized total boost, and presence of estrogen receptors. Surprisingly, the effect (coefficient in the prediction model) of hypertension on telangiectasia incidence was positive in ISE and negative in MARIE. Other predictors were also not common between the 2 cohorts, illustrating that overcoming overfitting does not solve the problem of replication failure of prediction models completely.
  
  
  
  Overfitting and cohort heterogeneity are the 2 main causes of replication failure of prediction models across cohorts. Cross-validation and similar techniques (eg, bootstrapping) cope with overfitting, but the development of validated predictive models for radiation therapy toxicity requires strategies that deal with cohort heterogeneity.
authors.affiliation: Department of Basic Medical Sciences, Faculty of Health Sciences, Ghent University, Ghent, Belgium; Department of Mathematical Modeling, Statistics, and Bioinformatics, Faculty of Bioscience Engineering, Ghent University, Ghent, Belgium. Electronic address: chamberlain.mbah@ugent.be.
authors.forename: Chamberlain
authors.initials: C
authors.lastname: Mbah
authors.affiliation: Department of Basic Medical Sciences, Faculty of Health Sciences, Ghent University, Ghent, Belgium.
authors.forename: Hubert
authors.initials: H
authors.lastname: Thierens
authors.affiliation: Department of Mathematical Modeling, Statistics, and Bioinformatics, Faculty of Bioscience Engineering, Ghent University, Ghent, Belgium; National Institute for Applied Statistics Research Australia, University of Wollongong, Wollongong, New South Wales, Australia.
authors.forename: Olivier
authors.initials: O
authors.lastname: Thas
authors.affiliation: Department of Data Analysis, Faculty of Psychology and Educational Sciences, Ghent University, Ghent, Belgium.
authors.forename: Jan
authors.initials: J
authors.lastname: De Neve
authors.affiliation: Division of Cancer Epidemiology, German Cancer Research Center, Heidelberg, Germany.
authors.forename: Jenny
authors.initials: J
authors.lastname: Chang-Claude
authors.affiliation: Division of Cancer Epidemiology, German Cancer Research Center, Heidelberg, Germany.
authors.forename: Petra
authors.initials: P
authors.lastname: Seibold
authors.affiliation: Division of Cancer Epidemiology, German Cancer Research Center, Heidelberg, Germany.
authors.forename: Akke
authors.initials: A
authors.lastname: Botma
authors.affiliation: Translational Radiobiology Group, Institute of Cancer Sciences, Radiotherapy Related Research, Christie Hospital NHS Trust, University of Manchester, Manchester, United Kingdom.
authors.forename: Catharine
authors.initials: C
authors.lastname: West
authors.affiliation: Department of Basic Medical Sciences, Faculty of Health Sciences, Ghent University, Ghent, Belgium.
authors.forename: Kim
authors.initials: K
authors.lastname: De Ruyck
mesh_terms: D000328:Adult; D000368:Aged; D016477:Artifacts; D001943:Breast Neoplasms / Q000453:epidemiology* / Q000532:radiotherapy*; D015331:Cohort Studies; D003198:Computer Simulation; D004307:Dose-Response Relationship, Radiation; D005260:Female; D005858:Germany / Q000453:epidemiology; D006801:Humans; D008875:Middle Aged; D015233:Models, Statistical; D017063:Outcome Assessment, Health Care / Q000379:methods; D015995:Prevalence; D016016:Proportional Hazards Models; D011832:Radiation Injuries / Q000453:epidemiology*; D011879:Radiotherapy Dosage; D015203:Reproducibility of Results; D018570:Risk Assessment / Q000379:methods; D012680:Sensitivity and Specificity; D013684:Telangiectasis / Q000175:diagnosis / Q000453:epidemiology*