title: Alignathon: a competitive assessment of whole-genome alignment methods.
abstract: Multiple sequence alignments (MSAs) are a prerequisite for a wide variety of evolutionary analyses. Published assessments and benchmark data sets for protein and, to a lesser extent, global nucleotide MSAs are available, but less effort has been made to establish benchmarks in the more general problem of whole-genome alignment (WGA). Using the same model as the successful Assemblathon competitions, we organized a competitive evaluation in which teams submitted their alignments and then assessments were performed collectively after all the submissions were received. Three data sets were used: Two were simulated and based on primate and mammalian phylogenies, and one was comprised of 20 real fly genomes. In total, 35 submissions were assessed, submitted by 10 teams using 12 different alignment pipelines. We found agreement between independent simulation-based and statistical assessments, indicating that there are substantial accuracy differences between contemporary alignment tools. We saw considerable differences in the alignment quality of differently annotated regions and found that few tools aligned the duplications analyzed. We found that many tools worked well at shorter evolutionary distances, but fewer performed competitively at longer distances. We provide all data sets, submissions, and assessment programs for further study and provide, as a resource for future benchmarking, a convenient repository of code and data for reproducing the simulation assessments.
authors.affiliation: Center for Biomolecular Science and Engineering, University of California Santa Cruz, Santa Cruz, California 95064, USA; Biomolecular Engineering Department, University of California Santa Cruz, Santa Cruz, California 95064, USA;
authors.forename: Dent
authors.initials: D
authors.lastname: Earl
authors.affiliation: Center for Biomolecular Science and Engineering, University of California Santa Cruz, Santa Cruz, California 95064, USA; Biomolecular Engineering Department, University of California Santa Cruz, Santa Cruz, California 95064, USA;
authors.forename: Ngan
authors.initials: N
authors.lastname: Nguyen
authors.affiliation: School of Computer Science, McGill University, Montreal, QC H3A 0G4, Canada;
authors.forename: Glenn
authors.initials: G
authors.lastname: Hickey
authors.affiliation: Department of Biology, The Pennsylvania State University, University Park, Pennsylvania 16801, USA;
authors.forename: Robert S
authors.initials: RS
authors.lastname: Harris
authors.affiliation: European Molecular Biology Laboratory, European Bioinformatics Institute, Wellcome Trust Genome Campus, Hinxton, Cambridge, CB10 1SD, United Kingdom;
authors.forename: Stephen
authors.initials: S
authors.lastname: Fitzgerald
authors.affiliation: European Molecular Biology Laboratory, European Bioinformatics Institute, Wellcome Trust Genome Campus, Hinxton, Cambridge, CB10 1SD, United Kingdom;
authors.forename: Kathryn
authors.initials: K
authors.lastname: Beal
authors.affiliation: Softberry Inc., Mount Kisco, New York 10549, USA;
authors.forename: Igor
authors.initials: I
authors.lastname: Seledtsov
authors.affiliation: Softberry Inc., Mount Kisco, New York 10549, USA;
authors.forename: Vladimir
authors.initials: V
authors.lastname: Molodtsov
authors.affiliation: Center for Biomolecular Science and Engineering, University of California Santa Cruz, Santa Cruz, California 95064, USA;
authors.forename: Brian J
authors.initials: BJ
authors.lastname: Raney
authors.affiliation: Center for Biomolecular Science and Engineering, University of California Santa Cruz, Santa Cruz, California 95064, USA;
authors.forename: Hiram
authors.initials: H
authors.lastname: Clawson
authors.affiliation: Department of Animal Biotechnology, Konkuk University, Seoul 143-701, Korea;
authors.forename: Jaebum
authors.initials: J
authors.lastname: Kim
authors.affiliation: Centre For Genomic Regulation (CRG), 08003 Barcelona, Spain; Universitat Pompeu Fabra (UPF), 08003 Barcelona, Spain; Westfalian Wilhelms University, Institute of Evolution and Biodiversity, 48149 Muenster, Germany;
authors.forename: Carsten
authors.initials: C
authors.lastname: Kemena
authors.affiliation: Centre For Genomic Regulation (CRG), 08003 Barcelona, Spain; Universitat Pompeu Fabra (UPF), 08003 Barcelona, Spain; Institute of Human Genetics (IGH), UPR 1142, CNRS, Montpellier, France;
authors.forename: Jia-Ming
authors.initials: JM
authors.lastname: Chang
authors.affiliation: Centre For Genomic Regulation (CRG), 08003 Barcelona, Spain; Universitat Pompeu Fabra (UPF), 08003 Barcelona, Spain;
authors.forename: Ionas
authors.initials: I
authors.lastname: Erb
authors.affiliation: Department of Energy Joint Genome Institute, Walnut Creek, California 94598, USA;
authors.forename: Alexander
authors.initials: A
authors.lastname: Poliakov
authors.affiliation: Department of Computer Science, Northern Illinois University, DeKalb, Illinois 60115, USA;
authors.forename: Minmei
authors.initials: M
authors.lastname: Hou
authors.affiliation: European Molecular Biology Laboratory, European Bioinformatics Institute, Wellcome Trust Genome Campus, Hinxton, Cambridge, CB10 1SD, United Kingdom; The Genome Analysis Centre, Norwich Research Park, Norwich, NR4 7UH, United Kingdom;
authors.forename: Javier
authors.initials: J
authors.lastname: Herrero
authors.affiliation: Center for Biomolecular Science and Engineering, University of California Santa Cruz, Santa Cruz, California 95064, USA; Biomolecular Engineering Department, University of California Santa Cruz, Santa Cruz, California 95064, USA;
authors.forename: William James
authors.initials: WJ
authors.lastname: Kent
authors.affiliation: Softberry Inc., Mount Kisco, New York 10549, USA;
authors.forename: Victor
authors.initials: V
authors.lastname: Solovyev
authors.affiliation: ithree Institute, University of Technology Sydney, NSW 2007, Australia;
authors.forename: Aaron E
authors.initials: AE
authors.lastname: Darling
authors.affiliation: Department of Bioengineering and Institute for Genomic Biology, University of Illinois at Urbana-Champaign, Illinois 61801, USA;
authors.forename: Jian
authors.initials: J
authors.lastname: Ma
authors.affiliation: Centre For Genomic Regulation (CRG), 08003 Barcelona, Spain; Universitat Pompeu Fabra (UPF), 08003 Barcelona, Spain;
authors.forename: Cedric
authors.initials: C
authors.lastname: Notredame
authors.affiliation: Department of Computer Science and the Donnelly Centre, University of Toronto, Toronto, ON M5S 3G4, Canada; Centre for Computational Medicine and the Genetics and Genome Biology Program, Hospital for Sick Children, Toronto, ON M5G 1X8, Canada;
authors.forename: Michael
authors.initials: M
authors.lastname: Brudno
authors.affiliation: Department of Energy Joint Genome Institute, Walnut Creek, California 94598, USA; Lawrence Berkeley National Laboratory, Berkeley, California 94710, USA;
authors.forename: Inna
authors.initials: I
authors.lastname: Dubchak
authors.affiliation: Center for Biomolecular Science and Engineering, University of California Santa Cruz, Santa Cruz, California 95064, USA; Biomolecular Engineering Department, University of California Santa Cruz, Santa Cruz, California 95064, USA; Howard Hughes Medical Institute, Chevy Chase, Maryland 20815-6789, USA.
authors.forename: David
authors.initials: D
authors.lastname: Haussler
authors.affiliation: Center for Biomolecular Science and Engineering, University of California Santa Cruz, Santa Cruz, California 95064, USA; Biomolecular Engineering Department, University of California Santa Cruz, Santa Cruz, California 95064, USA;
authors.forename: Benedict
authors.initials: B
authors.lastname: Paten
mesh_terms: D000818:Animals; D019295:Computational Biology / Q000379:methods; D003198:Computer Simulation; D066264:Datasets as Topic; D016678:Genome; D055106:Genome-Wide Association Study; D023281:Genomics / Q000379:methods*; D006801:Humans; D008322:Mammals / Q000235:genetics; D010802:Phylogeny; D015203:Reproducibility of Results; D016415:Sequence Alignment / Q000379:methods*; D012984:Software