title: CSAR 2014: A Benchmark Exercise Using Unpublished Data from Pharma.
abstract: The 2014 CSAR Benchmark Exercise was the last community-wide exercise that was conducted by the group at the University of Michigan, Ann Arbor. For this event, GlaxoSmithKline (GSK) donated unpublished crystal structures and affinity data from in-house projects. Three targets were used: tRNA (m1G37) methyltransferase (TrmD), Spleen Tyrosine Kinase (SYK), and Factor Xa (FXa). A particularly strong feature of the GSK data is its large size, which lends greater statistical significance to comparisons between different methods. In Phase 1 of the CSAR 2014 Exercise, participants were given several protein-ligand complexes and asked to identify the one near-native pose from among 200 decoys provided by CSAR. Though decoys were requested by the community, we found that they complicated our analysis. We could not discern whether poor predictions were failures of the chosen method or an incompatibility between the participant's method and the setup protocol we used. This problem is inherent to decoys, and we strongly advise against their use. In Phase 2, participants had to dock and rank/score a set of small molecules given only the SMILES strings of the ligands and a protein structure with a different ligand bound. Overall, docking was a success for most participants, much better in Phase 2 than in Phase 1. However, scoring was a greater challenge. No particular approach to docking and scoring had an edge, and successful methods included empirical, knowledge-based, machine-learning, shape-fitting, and even those with solvation and entropy terms. Several groups were successful in ranking TrmD and/or SYK, but ranking FXa ligands was intractable for all participants. Methods that were able to dock well across all submitted systems include MDock,1 Glide-XP,2 PLANTS,3 Wilma,4 Gold,5 SMINA,6 Glide-XP2/PELE,7 FlexX,8 and MedusaDock.9 In fact, the submission based on Glide-XP2/PELE7 cross-docked all ligands to many crystal structures, and it was particularly impressive to see success across an ensemble of protein structures for multiple targets. For scoring/ranking, submissions that showed statistically significant achievement include MDock1 using ITScore1,10 with a flexible-ligand term,11 SMINA6 using Autodock-Vina,12,13 FlexX8 using HYDE,14 and Glide-XP2 using XP DockScore2 with and without ROCS15 shape similarity.16 Of course, these results are for only three protein targets, and many more systems need to be investigated to truly identify which approaches are more successful than others. Furthermore, our exercise is not a competition.
authors.affiliation: Department of Medicinal Chemistry, College of Pharmacy, University of Michigan , 428 Church St., Ann Arbor, Michigan 48109-1065, United States.
authors.forename: Heather A
authors.initials: HA
authors.lastname: Carlson
authors.affiliation: Department of Medicinal Chemistry, College of Pharmacy, University of Michigan , 428 Church St., Ann Arbor, Michigan 48109-1065, United States.
authors.forename: Richard D
authors.initials: RD
authors.lastname: Smith
authors.affiliation: Department of Medicinal Chemistry, College of Pharmacy, University of Michigan , 428 Church St., Ann Arbor, Michigan 48109-1065, United States.
authors.forename: Kelly L
authors.initials: KL
authors.lastname: Damm-Ganamet
authors.affiliation: Center for Structural Biology, University of Michigan , 3358E Life Sciences Institute, 210 Washtenaw Ave., Ann Arbor, Michigan 48109-2216, United States.
authors.forename: Jeanne A
authors.initials: JA
authors.lastname: Stuckey
authors.affiliation: Department of Medicinal Chemistry, College of Pharmacy, University of Michigan , 428 Church St., Ann Arbor, Michigan 48109-1065, United States.
authors.forename: Aqeel
authors.initials: A
authors.lastname: Ahmed
authors.affiliation: Computational and Structural Sciences, Medicines Research Centre, GlaxoSmithKline Research & Development , Gunnels Wood Road, Stevenage, Hertfordshire SG1 2NY, United Kingdom.
authors.forename: Maire A
authors.initials: MA
authors.lastname: Convery
authors.affiliation: Computational and Structural Sciences, Medicines Research Centre, GlaxoSmithKline Research & Development , Gunnels Wood Road, Stevenage, Hertfordshire SG1 2NY, United Kingdom.
authors.forename: Donald O
authors.initials: DO
authors.lastname: Somers
authors.affiliation: Computational and Structural Sciences, Medicines Research Centre, GlaxoSmithKline Research & Development , Gunnels Wood Road, Stevenage, Hertfordshire SG1 2NY, United Kingdom.
authors.forename: Michael
authors.initials: M
authors.lastname: Kranz
authors.affiliation: Computational and Structural Sciences, GlaxoSmithKline Research & Development , 1250 South Collegeville Road, Collegeville, Pennsylvania 19426, United States.
authors.forename: Patricia A
authors.initials: PA
authors.lastname: Elkins
authors.affiliation: Computational and Structural Sciences, GlaxoSmithKline Research & Development , 1250 South Collegeville Road, Collegeville, Pennsylvania 19426, United States.
authors.forename: Guanglei
authors.initials: G
authors.lastname: Cui
authors.affiliation: Computational and Structural Sciences, GlaxoSmithKline Research & Development , 1250 South Collegeville Road, Collegeville, Pennsylvania 19426, United States.
authors.forename: Catherine E
authors.initials: CE
authors.lastname: Peishoff
authors.affiliation: Computational and Structural Sciences, GlaxoSmithKline Research & Development , 1250 South Collegeville Road, Collegeville, Pennsylvania 19426, United States.
authors.forename: Millard H
authors.initials: MH
authors.lastname: Lambert
authors.affiliation: Department of Medicinal Chemistry, College of Pharmacy, University of Michigan , 428 Church St., Ann Arbor, Michigan 48109-1065, United States.
authors.forename: James B
authors.initials: JB
authors.lastname: Dunbar
mesh_terms: D019985:Benchmarking; D062313:Databases, Pharmaceutical; D015195:Drug Design; D015951:Factor Xa / Q000737:chemistry / Q000378:metabolism; D008024:Ligands; D062105:Molecular Docking Simulation; D011487:Protein Conformation; D011506:Proteins / Q000737:chemistry / Q000378:metabolism*; D013329:Structure-Activity Relationship; D000072377:Syk Kinase / Q000737:chemistry / Q000378:metabolism; D012359:tRNA Methyltransferases / Q000737:chemistry / Q000378:metabolism
chemical_list: D008024:Ligands; D011506:Proteins; D012359:tRNA Methyltransferases; D000072377:Syk Kinase; D015951:Factor Xa