title: A simple but highly effective approach to evaluate the prognostic performance of gene expression signatures.
abstract: BACKGROUND
  Highly parallel analysis of gene expression has recently been used to identify gene sets or 'signatures' to improve patient diagnosis and risk stratification. Once a signature is generated, traditional statistical testing is used to evaluate its prognostic performance. However, due to the dimensionality of microarrays, this can lead to false interpretation of these signatures.
  
  
  RESULTS
  A method was developed to test batches of a user-specified number of randomly chosen signatures in patient microarray datasets. The percentage of random generated signatures yielding prognostic value was assessed using ROC analysis by calculating the area under the curve (AUC) in six public available cancer patient microarray datasets. We found that a signature consisting of randomly selected genes has an average 10% chance of reaching significance when assessed in a single dataset, but can range from 1% to ∼40% depending on the dataset in question. Increasing the number of validation datasets markedly reduces this number.
  
  
  CONCLUSIONS
  We have shown that the use of an arbitrary cut-off value for evaluation of signature significance is not suitable for this type of research, but should be defined for each dataset separately. Our method can be used to establish and evaluate signature performance of any derived gene signature in a dataset by comparing its performance to thousands of randomly generated signatures. It will be of most interest for cases where few data are available and testing in multiple datasets is limited.
authors.affiliation: Department of Radiation Oncology (Maastro), GROW-School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, The Netherlands. maud.starmans@maastrichtuniversity.nl
authors.forename: Maud H W
authors.initials: MH
authors.lastname: Starmans
authors.forename: Glenn
authors.initials: G
authors.lastname: Fung
authors.forename: Harald
authors.initials: H
authors.lastname: Steck
authors.forename: Bradly G
authors.initials: BG
authors.lastname: Wouters
authors.forename: Philippe
authors.initials: P
authors.lastname: Lambin
mesh_terms: D019540:Area Under Curve; D001943:Breast Neoplasms / Q000378:metabolism; D019295:Computational Biology / Q000379:methods*; D003627:Data Interpretation, Statistical; D020869:Gene Expression Profiling; D015972:Gene Expression Regulation, Neoplastic; D006801:Humans; D007680:Kidney Neoplasms / Q000378:metabolism; D008175:Lung Neoplasms / Q000378:metabolism; D015233:Models, Statistical; D009369:Neoplasms / Q000235:genetics* / Q000378:metabolism*; D020411:Oligonucleotide Array Sequence Analysis; D011379:Prognosis; D012372:ROC Curve; D015203:Reproducibility of Results