title: Double-blind evaluation and benchmarking of survival models in a multi-centre study.
abstract: Accurate modelling of time-to-event data is of particular importance for both exploratory and predictive analysis in cancer, and can have a direct impact on clinical care. This study presents a detailed double-blind evaluation of the accuracy in out-of-sample prediction of mortality from two generic non-linear models, using artificial neural networks benchmarked against a partial logistic spline, log-normal and COX regression models. A data set containing 2880 samples was shared over the Internet using a purpose-built secure environment called GEOCONDA (www.geoconda.com). The evaluation was carried out in three parts. The first was a comparison between the predicted survival estimates for each of the four survival groups defined by the TNM staging system, against the empirical estimates derived by the Kaplan-Meier method. The second approach focused on the accurate prediction of survival over time, quantified with the time dependent C index (C(td)). Finally, calibration plots were obtained over the range of follow-up and tested using a generalization of the Hosmer-Lemeshow test. All models showed satisfactory performance, with values of C(td) of about 0.7. None of the models showed a systematic tendency towards over/under estimation of the observed survival at tau=3 and 5 years. At tau=10 years, all models underestimated the observed survival, except for COX regression which returned an overestimate. The study presents a robust and unbiased benchmarking methodology using a bespoke web facility. It was concluded that powerful, recent flexible modelling algorithms show a comparative predictive performance to that of more established methods from the medical and biological literature, for the reference data set.
authors.affiliation: Department of Clinical Engineering, Royal Liverpool University Hospital, Liverpool, UK. afgt@liv.ac.uk
authors.forename: A
authors.initials: A
authors.lastname: Taktak
authors.forename: L
authors.initials: L
authors.lastname: Antolini
authors.forename: M
authors.initials: M
authors.lastname: Aung
authors.forename: P
authors.initials: P
authors.lastname: Boracchi
authors.forename: I
authors.initials: I
authors.lastname: Campbell
authors.forename: B
authors.initials: B
authors.lastname: Damato
authors.forename: E
authors.initials: E
authors.lastname: Ifeachor
authors.forename: N
authors.initials: N
authors.lastname: Lama
authors.forename: P
authors.initials: P
authors.lastname: Lisboa
authors.forename: C
authors.initials: C
authors.lastname: Setzkorn
authors.forename: V
authors.initials: V
authors.lastname: Stalbovskaya
authors.forename: E
authors.initials: E
authors.lastname: Biganzoli
mesh_terms: D019985:Benchmarking; D003198:Computer Simulation; D016208:Databases, Factual; D004311:Double-Blind Method; D005260:Female; D006801:Humans; D053208:Kaplan-Meier Estimate; D016014:Linear Models; D008297:Male; D008545:Melanoma / Q000401:mortality; D008875:Middle Aged; D016571:Neural Networks, Computer; D017711:Nonlinear Dynamics; D016016:Proportional Hazards Models; D016019:Survival Analysis; D006113:United Kingdom / Q000453:epidemiology; D014604:Uveal Neoplasms / Q000401:mortality