title: VariantSpark: population scale clustering of genotype information.
abstract: BACKGROUND
  Genomic information is increasingly used in medical practice giving rise to the need for efficient analysis methodology able to cope with thousands of individuals and millions of variants. The widely used Hadoop MapReduce architecture and associated machine learning library, Mahout, provide the means for tackling computationally challenging tasks. However, many genomic analyses do not fit the Map-Reduce paradigm. We therefore utilise the recently developed SPARK engine, along with its associated machine learning library, MLlib, which offers more flexibility in the parallelisation of population-scale bioinformatics tasks. The resulting tool, VARIANTSPARK provides an interface from MLlib to the standard variant format (VCF), offers seamless genome-wide sampling of variants and provides a pipeline for visualising results.
  
  
  RESULTS
  To demonstrate the capabilities of VARIANTSPARK, we clustered more than 3,000 individuals with 80 Million variants each to determine the population structure in the dataset. VARIANTSPARK is 80 % faster than the SPARK-based genome clustering approach, ADAM, the comparable implementation using Hadoop/Mahout, as well as ADMIXTURE, a commonly used tool for determining individual ancestries. It is over 90 % faster than traditional implementations using R and Python.
  
  
  CONCLUSIONS
  The benefits of speed, resource consumption and scalability enables VARIANTSPARK to open up the usage of advanced, efficient machine learning algorithms to genomic data.
authors.affiliation: CSIRO, Health & Biosecurity Flagship, 11 Julius Av, Sydney, 2113, Australia.|School of Biomedical Sciences and Pharmacy, Faculty of Health, Newcastle, 2308, Australia.
authors.forename: Aidan R
authors.initials: AR
authors.lastname: O'Brien
authors.affiliation: CSIRO, Health & Biosecurity Flagship, 11 Julius Av, Sydney, 2113, Australia.
authors.forename: Neil F W
authors.initials: NF
authors.lastname: Saunders
authors.affiliation: CSIRO, Data61, Sydney, 2052, Australia. Yi.Guo@csiro.au.
authors.forename: Yi
authors.initials: Y
authors.lastname: Guo
authors.affiliation: Cancer Epigenetics Program, Cancer Research Division, Kinghorn Cancer Centre, Garvan Institute of Medical Research, 384 Victoria St, Sydney, 2010, Australia.|UNSW Medicine, University of New South Wales, Sydney, 2052, Australia.
authors.forename: Fabian A
authors.initials: FA
authors.lastname: Buske
authors.affiliation: School of Biomedical Sciences and Pharmacy, Faculty of Health, Newcastle, 2308, Australia. rodney.scott@newcastle.edu.au.
authors.forename: Rodney J
authors.initials: RJ
authors.lastname: Scott
authors.affiliation: CSIRO, Health & Biosecurity Flagship, 11 Julius Av, Sydney, 2113, Australia. Denis.Bauer@CSIRO.au.
authors.forename: Denis C
authors.initials: DC
authors.lastname: Bauer
mesh_terms: D000465:Algorithms; D016000:Cluster Analysis; D019295:Computational Biology / Q000379:methods*; D005838:Genotype; D006801:Humans; D020641:Polymorphism, Single Nucleotide; D012984:Software