title: Exploiting the noise: improving biomarkers with ensembles of data analysis methodologies.
abstract: BACKGROUND
  The advent of personalized medicine requires robust, reproducible biomarkers that indicate which treatment will maximize therapeutic benefit while minimizing side effects and costs. Numerous molecular signatures have been developed over the past decade to fill this need, but their validation and up-take into clinical settings has been poor. Here, we investigate the technical reasons underlying reported failures in biomarker validation for non-small cell lung cancer (NSCLC).
  
  
  METHODS
  We evaluated two published prognostic multi-gene biomarkers for NSCLC in an independent 442-patient dataset. We then systematically assessed how technical factors influenced validation success.
  
  
  RESULTS
  Both biomarkers validated successfully (biomarker #1: hazard ratio (HR) 1.63, 95% confidence interval (CI) 1.21 to 2.19, P = 0.001; biomarker #2: HR 1.42, 95% CI 1.03 to 1.96, P = 0.030). Further, despite being underpowered for stage-specific analyses, both biomarkers successfully stratified stage II patients and biomarker #1 also stratified stage IB patients. We then systematically evaluated reasons for reported validation failures and find they can be directly attributed to technical challenges in data analysis. By examining 24 separate pre-processing techniques we show that minor alterations in pre-processing can change a successful prognostic biomarker (HR 1.85, 95% CI 1.37 to 2.50, P < 0.001) into one indistinguishable from random chance (HR 1.15, 95% CI 0.86 to 1.54, P = 0.348). Finally, we develop a new method, based on ensembles of analysis methodologies, to exploit this technical variability to improve biomarker robustness and to provide an independent confidence metric.
  
  
  CONCLUSIONS
  Biomarkers comprise a fundamental component of personalized medicine. We first validated two NSCLC prognostic biomarkers in an independent patient cohort. Power analyses demonstrate that even this large, 442-patient cohort is under-powered for stage-specific analyses. We then use these results to discover an unexpected sensitivity of validation to subtle data analysis decisions. Finally, we develop a novel algorithmic approach to exploit this sensitivity to improve biomarker robustness.
authors.affiliation: Informatics and Biocomputing Platform, Ontario Institute for Cancer Research, Toronto, ON, M5G 0A3, Canada ; Department of Radiation Oncology (Maastro), GROW-School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, The Netherlands.
authors.forename: Maud Hw
authors.initials: MH
authors.lastname: Starmans
authors.affiliation: Ontario Cancer Institute and the Campbell Family Institute for Cancer Research, University Health Network, Toronto, ON, M5G 2M9, Canada.
authors.forename: Melania
authors.initials: M
authors.lastname: Pintilie
authors.affiliation: Ludwig Institute for Cancer Research, Austin Health, Melbourne, Australia.
authors.forename: Thomas
authors.initials: T
authors.lastname: John
authors.affiliation: Ontario Cancer Institute and the Campbell Family Institute for Cancer Research, University Health Network, Toronto, ON, M5G 2M9, Canada.
authors.forename: Sandy D
authors.initials: SD
authors.lastname: Der
authors.affiliation: Department of Medical Oncology and Hematology, Princess Margaret Hospital, University Health Network, Toronto, ON, M5G 2M9, Canada.
authors.forename: Frances A
authors.initials: FA
authors.lastname: Shepherd
authors.affiliation: Ontario Cancer Institute and the Campbell Family Institute for Cancer Research, University Health Network, Toronto, ON, M5G 2M9, Canada ; Department of Medical Biophysics, University of Toronto, Toronto, ON, M5G 2M9, Canada ; Techna Institute, University Health Network, Toronto, ON, M5G 2M9, Canada ; Department of Computer Science, University of Toronto, Toronto, ON, M5G 2M9, Canada.
authors.forename: Igor
authors.initials: I
authors.lastname: Jurisica
authors.affiliation: Department of Radiation Oncology (Maastro), GROW-School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, The Netherlands.
authors.forename: Philippe
authors.initials: P
authors.lastname: Lambin
authors.affiliation: Ontario Cancer Institute and the Campbell Family Institute for Cancer Research, University Health Network, Toronto, ON, M5G 2M9, Canada.
authors.forename: Ming-Sound
authors.initials: MS
authors.lastname: Tsao
authors.affiliation: Informatics and Biocomputing Platform, Ontario Institute for Cancer Research, Toronto, ON, M5G 0A3, Canada ; Department of Medical Biophysics, University of Toronto, Toronto, ON, M5G 2M9, Canada.
authors.forename: Paul C
authors.initials: PC
authors.lastname: Boutros