title: A divide-and-conquer strategy to solve the out-of-memory problem of processing thousands of Affymetrix microarrays.
abstract: Out-of-memory problem was frequently encountered when processing thousands of CEL files using Bioconductor. We propose a divide-and-conquer strategy combined with randomised resampling to solve this problem. The CAMDA 2007 META-analysis data set which contains 5896 CEL files was used to test the approach on a typical commodity computer cluster by running established pre-processing algorithms for Affymetrix arrays in the Bioconductor package. The results were validated against a golden standard obtained by using a supercomputer. In addition to the performance improvement, the general divide-and-conquer strategy can be applied to any other normalisation algorithms without modifying the underlying implementation.
authors.affiliation: Computational Biology and Bioinformatics Program, Northwestemrn University, Evanston, IL 60208, USA. ChiaJuLee2008@u.northwestern.edu
authors.forename: Chia-Ju
authors.initials: CJ
authors.lastname: Lee
authors.forename: Dong
authors.initials: D
authors.lastname: Fu
authors.forename: Pan
authors.initials: P
authors.lastname: Du
authors.forename: Hongmei
authors.initials: H
authors.lastname: Jiang
authors.forename: Simon M
authors.initials: SM
authors.lastname: Lin
authors.forename: Warren
authors.initials: W
authors.lastname: Kibbe
mesh_terms: D056808:Biostatistics / Q000379:methods*; D019295:Computational Biology / Q000379:methods*; D016248:Computer Storage Devices; D003201:Computers; D004867:Equipment Design; D006801:Humans; D020411:Oligonucleotide Array Sequence Analysis / Q000379:methods*; D011340:Problem Solving