title: Optimizing R with SparkR on a commodity cluster for biomedical research.
abstract: OBJECTIVE
  Medical researchers are challenged today by the enormous amount of data collected in healthcare. Analysis methods such as genome-wide association studies (GWAS) are often computationally intensive and thus require enormous resources to be performed in a reasonable amount of time. While dedicated clusters and public clouds may deliver the desired performance, their use requires upfront financial efforts or anonymous data, which is often not possible for preliminary or occasional tasks. We explored the possibilities to build a private, flexible cluster for processing scripts in R based on commodity, non-dedicated hardware of our department.
  
  
  METHODS
  For this, a GWAS-calculation in R on a single desktop computer, a Message Passing Interface (MPI)-cluster, and a SparkR-cluster were compared with regards to the performance, scalability, quality, and simplicity.
  
  
  RESULTS
  The original script had a projected runtime of three years on a single desktop computer. Optimizing the script in R already yielded a significant reduction in computing time (2 weeks). By using R-MPI and SparkR, we were able to parallelize the computation and reduce the time to less than three hours (2.6 h) on already available, standard office computers. While MPI is a proven approach in high-performance clusters, it requires rather static, dedicated nodes. SparkR and its Hadoop siblings allow for a dynamic, elastic environment with automated failure handling. SparkR also scales better with the number of nodes in the cluster than MPI due to optimized data communication.
  
  
  CONCLUSIONS
  R is a popular environment for clinical data analysis. The new SparkR solution offers elastic resources and allows supporting big data analysis using R even on non-dedicated resources with minimal change to the original code. To unleash the full potential, additional efforts should be invested to customize and improve the algorithms, especially with regards to data distribution.
authors.affiliation: Friedrich-Alexander University Erlangen-Nürnberg, Wetterkreuz 13, 91058 Erlangen, Germany. Electronic address: martin.sedlmayr@fau.de.
authors.forename: Martin
authors.initials: M
authors.lastname: Sedlmayr
authors.affiliation: Friedrich-Alexander University Erlangen-Nürnberg, Wetterkreuz 13, 91058 Erlangen, Germany.
authors.forename: Tobias
authors.initials: T
authors.lastname: Würfl
authors.affiliation: Friedrich-Alexander University Erlangen-Nürnberg, Wetterkreuz 13, 91058 Erlangen, Germany.
authors.forename: Christian
authors.initials: C
authors.lastname: Maier
authors.affiliation: Department of Gynecology and Obstetrics, Erlangen University Hospital, Universitätsstrasse 21-23, 91054 Erlangen, Germany.
authors.forename: Lothar
authors.initials: L
authors.lastname: Häberle
authors.affiliation: Department of Gynecology and Obstetrics, Erlangen University Hospital, Universitätsstrasse 21-23, 91054 Erlangen, Germany.
authors.forename: Peter
authors.initials: P
authors.lastname: Fasching
authors.affiliation: Friedrich-Alexander University Erlangen-Nürnberg, Wetterkreuz 13, 91058 Erlangen, Germany.
authors.forename: Hans-Ulrich
authors.initials: HU
authors.lastname: Prokosch
authors.affiliation: Friedrich-Alexander University Erlangen-Nürnberg, Wetterkreuz 13, 91058 Erlangen, Germany.
authors.forename: Jan
authors.initials: J
authors.lastname: Christoph
mesh_terms: D035843:Biomedical Research; D016000:Cluster Analysis; D003205:Computing Methodologies; D055106:Genome-Wide Association Study; D006801:Humans
keywords: Big data; Cluster computing; Computing methodologies; Genome-wide association study; SparkR