title: Genomic data sampling and its effect on classification performance assessment.
abstract: BACKGROUND
  Supervised classification is fundamental in bioinformatics. Machine learning models, such as neural networks, have been applied to discover genes and expression patterns. This process is achieved by implementing training and test phases. In the training phase, a set of cases and their respective labels are used to build a classifier. During testing, the classifier is used to predict new cases. One approach to assessing its predictive quality is to estimate its accuracy during the test phase. Key limitations appear when dealing with small-data samples. This paper investigates the effect of data sampling techniques on the assessment of neural network classifiers.
  
  
  RESULTS
  Three data sampling techniques were studied: Cross-validation, leave-one-out, and bootstrap. These methods are designed to reduce the bias and variance of small-sample estimations. Two prediction problems based on small-sample sets were considered: Classification of microarray data originating from a leukemia study and from small, round blue-cell tumours. A third problem, the prediction of splice-junctions, was analysed to perform comparisons. Different accuracy estimations were produced for each problem. The variations are accentuated in the small-data samples. The quality of the estimates depends on the number of train-test experiments and the amount of data used for training the networks.
  
  
  CONCLUSIONS
  The predictive quality assessment of biomolecular data classifiers depends on the data size, sampling techniques and the number of train-test experiments. Conservative and optimistic accuracy estimations can be obtained by applying different methods. Guidelines are suggested to select a sampling technique according to the complexity of the prediction problem under consideration.
authors.affiliation: School of Computing and Mathematics, University of Ulster, Jordanstown, Nothern Ireland, UK. fj.azuaje@ulster.ac.uk
authors.forename: Francisco
authors.initials: F
authors.lastname: Azuaje
mesh_terms: D019295:Computational Biology / Q000379:methods / Q000706:statistics & numerical data; D003625:Data Collection / Q000379:methods / Q000706:statistics & numerical data; D019277:Entropy; D005091:Exons / Q000235:genetics; D020869:Gene Expression Profiling / Q000379:methods / Q000706:statistics & numerical data; D015972:Gene Expression Regulation, Neoplastic / Q000235:genetics; D023281:Genomics / Q000379:methods* / Q000706:statistics & numerical data*; D006801:Humans; D007438:Introns / Q000235:genetics; D009369:Neoplasms / Q000145:classification* / Q000235:genetics*; D016571:Neural Networks, Computer; D011237:Predictive Value of Tests; D022821:RNA Splice Sites / Q000235:genetics; D012494:Sampling Studies
chemical_list: D022821:RNA Splice Sites