title: Measuring the effect of inter-study variability on estimating prediction error.
abstract: BACKGROUND
  The biomarker discovery field is replete with molecular signatures that have not translated into the clinic despite ostensibly promising performance in predicting disease phenotypes. One widely cited reason is lack of classification consistency, largely due to failure to maintain performance from study to study. This failure is widely attributed to variability in data collected for the same phenotype among disparate studies, due to technical factors unrelated to phenotypes (e.g., laboratory settings resulting in "batch-effects") and non-phenotype-associated biological variation in the underlying populations. These sources of variability persist in new data collection technologies.
  
  
  METHODS
  Here we quantify the impact of these combined "study-effects" on a disease signature's predictive performance by comparing two types of validation methods: ordinary randomized cross-validation (RCV), which extracts random subsets of samples for testing, and inter-study validation (ISV), which excludes an entire study for testing. Whereas RCV hardwires an assumption of training and testing on identically distributed data, this key property is lost in ISV, yielding systematic decreases in performance estimates relative to RCV. Measuring the RCV-ISV difference as a function of number of studies quantifies influence of study-effects on performance.
  
  
  RESULTS
  As a case study, we gathered publicly available gene expression data from 1,470 microarray samples of 6 lung phenotypes from 26 independent experimental studies and 769 RNA-seq samples of 2 lung phenotypes from 4 independent studies. We find that the RCV-ISV performance discrepancy is greater in phenotypes with few studies, and that the ISV performance converges toward RCV performance as data from additional studies are incorporated into classification.
  
  
  CONCLUSIONS
  We show that by examining how fast ISV performance approaches RCV as the number of studies is increased, one can estimate when "sufficient" diversity has been achieved for learning a molecular signature likely to translate without significant loss of accuracy to new clinical settings.
authors.affiliation: Institute for Systems Biology, Seattle, Washington, United States of America; Department of Chemical and Biomolecular Engineering, University of Illinois, Urbana, Illinois, United States of America.
authors.forename: Shuyi
authors.initials: S
authors.lastname: Ma
authors.affiliation: Institute for Systems Biology, Seattle, Washington, United States of America; Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk, Republic of Korea.
authors.forename: Jaeyun
authors.initials: J
authors.lastname: Sung
authors.affiliation: Institute for Systems Biology, Seattle, Washington, United States of America; Center for Biophysics and Computational Biology, University of Illinois, Urbana, Illinois, United States of America.
authors.forename: Andrew T
authors.initials: AT
authors.lastname: Magis
authors.affiliation: Institute for Systems Biology, Seattle, Washington, United States of America; Sage Bionetworks, Seattle, Washington, United States of America.
authors.forename: Yuliang
authors.initials: Y
authors.lastname: Wang
authors.affiliation: Institute for Computational Medicine & Department of Applied Mathematics and Statistics, John Hopkins University, Baltimore, Maryland, United States of America.
authors.forename: Donald
authors.initials: D
authors.lastname: Geman
authors.affiliation: Institute for Systems Biology, Seattle, Washington, United States of America; Department of Chemical and Biomolecular Engineering, University of Illinois, Urbana, Illinois, United States of America; Center for Biophysics and Computational Biology, University of Illinois, Urbana, Illinois, United States of America.
authors.forename: Nathan D
authors.initials: ND
authors.lastname: Price
mesh_terms: D000230:Adenocarcinoma / Q000235:genetics / Q000473:pathology; D000077192:Adenocarcinoma of Lung; D014408:Biomarkers, Tumor / Q000096:biosynthesis*; D002294:Carcinoma, Squamous Cell / Q000235:genetics / Q000473:pathology; D020869:Gene Expression Profiling / Q000379:methods*; D015972:Gene Expression Regulation, Neoplastic; D006801:Humans; D008175:Lung Neoplasms / Q000235:genetics / Q000473:pathology; D020411:Oligonucleotide Array Sequence Analysis / Q000379:methods*; D010641:Phenotype; D029424:Pulmonary Disease, Chronic Obstructive / Q000235:genetics / Q000473:pathology; D017423:Sequence Analysis, RNA; D060388:Support Vector Machine
chemical_list: D014408:Biomarkers, Tumor