title: Optimising parallel R correlation matrix calculations on gene expression data using MapReduce.
abstract: BACKGROUND
  High-throughput molecular profiling data has been used to improve clinical decision making by stratifying subjects based on their molecular profiles. Unsupervised clustering algorithms can be used for stratification purposes. However, the current speed of the clustering algorithms cannot meet the requirement of large-scale molecular data due to poor performance of the correlation matrix calculation. With high-throughput sequencing technologies promising to produce even larger datasets per subject, we expect the performance of the state-of-the-art statistical algorithms to be further impacted unless efforts towards optimisation are carried out. MapReduce is a widely used high performance parallel framework that can solve the problem.
  
  
  RESULTS
  In this paper, we evaluate the current parallel modes for correlation calculation methods and introduce an efficient data distribution and parallel calculation algorithm based on MapReduce to optimise the correlation calculation. We studied the performance of our algorithm using two gene expression benchmarks. In the micro-benchmark, our implementation using MapReduce, based on the R package RHIPE, demonstrates a 3.26-5.83 fold increase compared to the default Snowfall and 1.56-1.64 fold increase compared to the basic RHIPE in the Euclidean, Pearson and Spearman correlations. Though vanilla R and the optimised Snowfall outperforms our optimised RHIPE in the micro-benchmark, they do not scale well with the macro-benchmark. In the macro-benchmark the optimised RHIPE performs 2.03-16.56 times faster than vanilla R. Benefiting from the 3.30-5.13 times faster data preparation, the optimised RHIPE performs 1.22-1.71 times faster than the optimised Snowfall. Both the optimised RHIPE and the optimised Snowfall successfully performs the Kendall correlation with TCGA dataset within 7 hours. Both of them conduct more than 30 times faster than the estimated vanilla R.
  
  
  CONCLUSIONS
  The performance evaluation found that the new MapReduce algorithm and its implementation in RHIPE outperforms vanilla R and the conventional parallel algorithms implemented in R Snowfall. We propose that MapReduce framework holds great promise for large molecular data analysis, in particular for high-dimensional genomic data such as that demonstrated in the performance evaluation described in this paper. We aim to use this new algorithm as a basis for optimising high-throughput molecular data correlation calculation for Big Data.
authors.affiliation: Data Science Institute, Imperial College London, London, UK. s.wang11@imperial.ac.uk.
authors.forename: Shicai
authors.initials: S
authors.lastname: Wang
authors.affiliation: Data Science Institute, Imperial College London, London, UK. i.pandis@imperial.ac.uk.
authors.forename: Ioannis
authors.initials: I
authors.lastname: Pandis
authors.affiliation: Data Science Institute, Imperial College London, London, UK. david.johnson@imperial.ac.uk.
authors.forename: David
authors.initials: D
authors.lastname: Johnson
authors.affiliation: Data Science Institute, Imperial College London, London, UK. i.emam@imperial.ac.uk.
authors.forename: Ibrahim
authors.initials: I
authors.lastname: Emam
authors.affiliation: Data Science Institute, Imperial College London, London, UK. f.guitton@imperial.ac.uk.
authors.forename: Florian
authors.initials: F
authors.lastname: Guitton
authors.affiliation: Data Science Institute, Imperial College London, London, UK. axelfrancois.oehmichen11@imperial.ac.uk.
authors.forename: Axel
authors.initials: A
authors.lastname: Oehmichen
authors.affiliation: Data Science Institute, Imperial College London, London, UK. y.guo@imperial.ac.uk.|School of Computer Science, Shanghai University, Shanghai, China. y.guo@imperial.ac.uk.
authors.forename: Yike
authors.initials: Y
authors.lastname: Guo
mesh_terms: D000465:Algorithms; D016000:Cluster Analysis; D020869:Gene Expression Profiling / Q000379:methods*; D059014:High-Throughput Nucleotide Sequencing; D006801:Humans; D012984:Software