title: DREAMTools: a Python package for scoring collaborative challenges.
abstract: DREAM challenges are community competitions designed to advance computational methods and address fundamental questions in system biology and translational medicine. Each challenge asks participants to develop and apply computational methods to either predict unobserved outcomes or to identify unknown model parameters given a set of training data. Computational methods are evaluated using an automated scoring metric, scores are posted to a public leaderboard, and methods are published to facilitate community discussions on how to build improved methods. By engaging participants from a wide range of science and engineering backgrounds, DREAM challenges can comparatively evaluate a wide range of statistical, machine learning, and biophysical methods. Here, we describe DREAMTools, a Python package for evaluating DREAM challenge scoring metrics. DREAMTools provides a command line interface that enables researchers to test new methods on past challenges, as well as a framework for scoring new challenges. As of March 2016, DREAMTools includes more than 80% of completed DREAM challenges. DREAMTools complements the data, metadata, and software tools available at the DREAM website http://dreamchallenges.org and on the Synapse platform at https://www.synapse.org.
  
  
  BACKGROUND
  DREAMTools is a Python package. Releases and documentation are available at http://pypi.python.org/pypi/dreamtools. The source code is available at http://github.com/dreamtools/dreamtools.
authors.affiliation: European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI),Wellcome Trust Genome Campus, Cambridge, UK; Bioinformatics and Biostatistics Hub, C3BI, Institut Pasteur, Paris, France.
authors.forename: Thomas
authors.initials: T
authors.lastname: Cokelaer
authors.affiliation: Department of Systems Biology, Columbia University, New York, USA.
authors.forename: Mukesh
authors.initials: M
authors.lastname: Bansal
authors.affiliation: Sage Bionetworks, Seattle, WA, USA.
authors.forename: Christopher
authors.initials: C
authors.lastname: Bare
authors.affiliation: IBM, TJ Watson, Computational Biology Center, New York, USA.
authors.forename: Erhan
authors.initials: E
authors.lastname: Bilal
authors.affiliation: Sage Bionetworks, Seattle, WA, USA.
authors.forename: Brian M
authors.initials: BM
authors.lastname: Bot
authors.affiliation: Sage Bionetworks, Seattle, WA, USA.
authors.forename: Elias
authors.initials: E
authors.lastname: Chaibub Neto
authors.affiliation: European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI),Wellcome Trust Genome Campus, Cambridge, UK.
authors.forename: Federica
authors.initials: F
authors.lastname: Eduati
authors.affiliation: Leibniz Institute for Farm Animal Biology, Institute of Genetics and Biometry, Dummerstorf, Germany.
authors.forename: Alberto
authors.initials: A
authors.lastname: de la Fuente
authors.affiliation: Oregon Health & Science University, Portland, OR, USA.
authors.forename: Mehmet
authors.initials: M
authors.lastname: Gönen
authors.affiliation: MRC Biostatistics Unit, Cambridge Institute of Public Health, Cambridge, UK.
authors.forename: Steven M
authors.initials: SM
authors.lastname: Hill
authors.affiliation: Sage Bionetworks, Seattle, WA, USA.
authors.forename: Bruce
authors.initials: B
authors.lastname: Hoff
authors.affiliation: Department of Genetics & Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, USA.
authors.forename: Jonathan R
authors.initials: JR
authors.lastname: Karr
authors.affiliation: Institute of Bioinformatics and Systems Biology, German Research Center for Environmental Health, Munich, Germany.
authors.forename: Robert
authors.initials: R
authors.lastname: Küffner
authors.affiliation: European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI),Wellcome Trust Genome Campus, Cambridge, UK.
authors.forename: Michael P
authors.initials: MP
authors.lastname: Menden
authors.affiliation: IBM, TJ Watson, Computational Biology Center, New York, USA.
authors.forename: Pablo
authors.initials: P
authors.lastname: Meyer
authors.affiliation: IBM, TJ Watson, Computational Biology Center, New York, USA.
authors.forename: Raquel
authors.initials: R
authors.lastname: Norel
authors.affiliation: Sage Bionetworks, Seattle, WA, USA.
authors.forename: Abhishek
authors.initials: A
authors.lastname: Pratap
authors.affiliation: IBM Almaden Research Center, San Jose, CA, USA.
authors.forename: Robert J
authors.initials: RJ
authors.lastname: Prill
authors.affiliation: Center for Autoimmune Genomics and Etiology and Divisions of Biomedical Informatics and Developmental Biology, Cincinnati Children's Hospital, Cincinnati, OH, USA.
authors.forename: Matthew T
authors.initials: MT
authors.lastname: Weirauch
authors.affiliation: Department of Pharmacology, University of Colorado Anschutz Medical Campus, Aurora, CO, USA.
authors.forename: James C
authors.initials: JC
authors.lastname: Costello
authors.affiliation: IBM, TJ Watson, Computational Biology Center, New York, USA; Department of Genetics & Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, USA.
authors.forename: Gustavo
authors.initials: G
authors.lastname: Stolovitzky
authors.affiliation: European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI),Wellcome Trust Genome Campus, Cambridge, UK; RWTH Aachen University Medical Hospital, Joint Research Centre for Computational Biomedicine (JRCCOMBINE), Aachen, Germany.
authors.forename: Julio
authors.initials: J
authors.lastname: Saez-Rodriguez
keywords: DREAM; benchmarking; collaborative competition; crowdsourcing; machine learning; method evaluation; systems biology; translational medicine