A Web Resource for Standardized Benchmark Datasets, Metrics, and Rosetta Protocols for Macromolecular Modeling and Design.
The development and validation of computational macromolecular modeling and design methods depend on suitable benchmark datasets and informative metrics for comparing protocols. In addition, if a method is intended to be adopted broadly in diverse biological applications, there needs to be information on appropriate parameters for each protocol, as well as metrics describing the expected accuracy compared to experimental data. In certain disciplines, there exist established benchmarks and public resources where experts in a particular methodology are encouraged to supply their most efficient implementation of each particular benchmark. We aim to provide such a resource for protocols in macromolecular modeling and design. We present a freely accessible web resource (https://kortemmelab.ucsf.edu/benchmarks) to guide the development of protocols for protein modeling and design. The site provides benchmark datasets and metrics to compare the performance of a variety of modeling protocols using different computational sampling methods and energy functions, providing a "best practice" set of parameters for each method. Each benchmark has an associated downloadable benchmark capture archive containing the input files, analysis scripts, and tutorials for running the benchmark. The captures may be run with any suitable modeling method; we supply command lines for running the benchmarks using the Rosetta software suite. We have compiled initial benchmarks for the resource spanning three key areas: prediction of energetic effects of mutations, protein design, and protein structure prediction, each with associated state-of-the-art modeling protocols. With the help of the wider macromolecular modeling community, we hope to expand the variety of benchmarks included on the website and continue to evaluate new iterations of current methods as they become available.
California Institute for Quantitative Biosciences (QB3), University of California San Francisco, San Francisco, California, United States of America; Department of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, California, United States of America.
Shane
S
Ó Conchúir
Graduate Program in Bioinformatics, University of California San Francisco, San Francisco, California, United States of America.
Kyle A
KA
Barlow
California Institute for Quantitative Biosciences (QB3), University of California San Francisco, San Francisco, California, United States of America; Department of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, California, United States of America.
Roland A
RA
Pache
Graduate Program in Bioinformatics, University of California San Francisco, San Francisco, California, United States of America.
Noah
N
Ollikainen
Graduate Program in Biophysics, University of California San Francisco, San Francisco, California, United States of America.
Kale
K
Kundert
Department of Pharmaceutical Chemistry, University of California San Francisco, San Francisco, California, United States of America.
Matthew J
MJ
O'Meara
California Institute for Quantitative Biosciences (QB3), University of California San Francisco, San Francisco, California, United States of America; Department of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, California, United States of America; Graduate Program in Bioinformatics, University of California San Francisco, San Francisco, California, United States of America.
Colin A
CA
Smith
California Institute for Quantitative Biosciences (QB3), University of California San Francisco, San Francisco, California, United States of America; Department of Bioengineering and Therapeutic Sciences, University of California San Francisco, San Francisco, California, United States of America; Graduate Program in Bioinformatics, University of California San Francisco, San Francisco, California, United States of America; Graduate Program in Biophysics, University of California San Francisco, San Francisco, California, United States of America.
Tanja
T
Kortemme
D000596:Amino Acids / Q000737:chemistry; D019985:Benchmarking; D066264:Datasets as Topic; D019418:Evolution, Chemical; D020407:Internet; D008958:Models, Molecular; D009154:Mutation; D011506:Proteins / Q000737:chemistry* / Q000235:genetics; D013816:Thermodynamics
D000596:Amino Acids; D011506:Proteins