title: CSAR benchmark exercise 2011-2012: evaluation of results from docking and relative ranking of blinded congeneric series.
abstract: The Community Structure-Activity Resource (CSAR) recently held its first blinded exercise based on data provided by Abbott, Vertex, and colleagues at the University of Michigan, Ann Arbor. A total of 20 research groups submitted results for the benchmark exercise where the goal was to compare different improvements for pose prediction, enrichment, and relative ranking of congeneric series of compounds. The exercise was built around blinded high-quality experimental data from four protein targets: LpxC, Urokinase, Chk1, and Erk2. Pose prediction proved to be the most straightforward task, and most methods were able to successfully reproduce binding poses when the crystal structure employed was co-crystallized with a ligand from the same chemical series. Multiple evaluation metrics were examined, and we found that RMSD and native contact metrics together provide a robust evaluation of the predicted poses. It was notable that most scoring functions underpredicted contacts between the hetero atoms (i.e., N, O, S, etc.) of the protein and ligand. Relative ranking was found to be the most difficult area for the methods, but many of the scoring functions were able to properly identify Urokinase actives from the inactives in the series. Lastly, we found that minimizing the protein and correcting histidine tautomeric states positively trended with low RMSD for pose prediction but minimizing the ligand negatively trended. Pregenerated ligand conformations performed better than those that were generated on the fly. Optimizing docking parameters and pretraining with the native ligand had a positive effect on the docking performance as did using restraints, substructure fitting, and shape fitting. Lastly, for both sampling and ranking scoring functions, the use of the empirical scoring function appeared to trend positively with the RMSD. Here, by combining the results of many methods, we hope to provide a statistically relevant evaluation and elucidate specific shortcomings of docking methodology for the community.
authors.affiliation: Department of Medicinal Chemistry, University of Michigan, Ann Arbor, Michigan 48109-1065, USA.
authors.forename: Kelly L
authors.initials: KL
authors.lastname: Damm-Ganamet
authors.forename: Richard D
authors.initials: RD
authors.lastname: Smith
authors.forename: James B
authors.initials: JB
authors.lastname: Dunbar
authors.forename: Jeanne A
authors.initials: JA
authors.lastname: Stuckey
authors.forename: Heather A
authors.initials: HA
authors.lastname: Carlson
mesh_terms: D019985:Benchmarking; D062313:Databases, Pharmaceutical; D015195:Drug Design; D008024:Ligands; D062105:Molecular Docking Simulation / Q000379:methods*; D011487:Protein Conformation; D011506:Proteins / Q000737:chemistry / Q000378:metabolism; D013329:Structure-Activity Relationship
chemical_list: D008024:Ligands; D011506:Proteins
references.citation: Acta Crystallogr D Biol Crystallogr. 2011 Apr;67(Pt 4):235-42
references.pmid: 21460441
references.citation: J Med Chem. 2005 Apr 7;48(7):2534-47
references.pmid: 15801843
references.citation: J Comput Aided Mol Des. 2008 Mar-Apr;22(3-4):179-90
references.pmid: 18217218
references.citation: J Comput Aided Mol Des. 2008 Mar-Apr;22(3-4):133-9
references.pmid: 18338228
references.citation: J Chem Inf Model. 2007 Jan-Feb;47(1):195-207
references.pmid: 17238265
references.citation: J Chem Inf Model. 2011 Sep 26;51(9):2036-46
references.pmid: 21728306
references.citation: Curr Opin Pharmacol. 2010 Dec;10(6):770-4
references.pmid: 20888294
references.citation: J Chem Inf Model. 2011 Sep 26;51(9):2115-31
references.pmid: 21809884
references.citation: Bioorg Med Chem. 2007 Apr 1;15(7):2759-67
references.pmid: 17287122
references.citation: J Am Chem Soc. 2007 Jul 4;129(26):8225-35
references.pmid: 17555316
references.citation: J Med Chem. 2004 Jan 15;47(2):337-44
references.pmid: 14711306
references.citation: J Comput Aided Mol Des. 2012 Jun;26(6):687-99
references.pmid: 22569590
references.citation: BMC Bioinformatics. 2007 Aug 25;8:310
references.pmid: 17718923
references.citation: Drug Discov Today. 2002 Oct 15;7(20):1047-55
references.pmid: 12546894
references.citation: J Chem Inf Model. 2009 Jun;49(6):1455-74
references.pmid: 19476350
references.citation: Proteins. 2002 Dec 1;49(4):457-71
references.pmid: 12402356
references.citation: J Med Chem. 2007 Jan 11;50(1):74-82
references.pmid: 17201411
references.citation: J Med Chem. 2010 May 27;53(10):3862-86
references.pmid: 20158188
references.citation: J Med Chem. 2008 Feb 28;51(4):769-79
references.pmid: 18215013
references.citation: J Comput Chem. 2008 Jun;29(8):1316-31
references.pmid: 18172838
references.citation: Curr Protein Pept Sci. 2006 Oct;7(5):407-20
references.pmid: 17073693
references.citation: Proteins. 2005 Aug 15;60(3):325-32
references.pmid: 15937897
references.citation: J Med Chem. 2004 Mar 25;47(7):1739-49
references.pmid: 15027865
references.citation: Acta Crystallogr D Biol Crystallogr. 2004 Dec;60(Pt 12 Pt 1):2240-9
references.pmid: 15572777
references.citation: Curr Top Med Chem. 2011;11(2):192-210
references.pmid: 20939788
references.citation: J Chem Inf Model. 2011 Oct 24;51(10):2455-66
references.pmid: 21870862
references.citation: Phys Chem Chem Phys. 2010 Oct 28;12(40):12899-908
references.pmid: 20730182
references.citation: J Med Chem. 2008 Oct 23;51(20):6237-55
references.pmid: 18785728
references.citation: Sci Am. 2000 Oct;283(4):82-7
references.pmid: 11011389
references.citation: J Chem Inf Model. 2008 Jul;48(7):1411-22
references.pmid: 18598022
references.citation: Curr Opin Chem Biol. 2002 Aug;6(4):447-52
references.pmid: 12133719
references.citation: J Med Chem. 2007 Feb 22;50(4):726-41
references.pmid: 17300160
references.citation: J Comput Aided Mol Des. 2012 Jun;26(6):787-99
references.pmid: 22576241
references.citation: J Comput Aided Mol Des. 2007 May;21(5):281-306
references.pmid: 17387436
references.citation: J Med Chem. 2003 Feb 13;46(4):499-511
references.pmid: 12570372
references.citation: Proteins. 1999 Nov 1;37(2):228-41
references.pmid: 10584068
references.citation: J Med Chem. 2006 Oct 5;49(20):5851-5
references.pmid: 17004700
references.citation: AAPS J. 2012 Mar;14(1):133-41
references.pmid: 22281989
references.citation: J Mol Biol. 2000 Jan 14;295(2):337-56
references.pmid: 10623530
references.citation: Chem Biol. 2011 Jan 28;18(1):38-47
references.pmid: 21167751
references.citation: Science. 2004 Mar 19;303(5665):1813-8
references.pmid: 15031495
references.citation: Drug Discov Today. 2012 Dec;17(23-24):1270-81
references.pmid: 22728777
references.citation: Br J Pharmacol. 2008 Mar;153 Suppl 1:S7-26
references.pmid: 18037925
references.citation: J Chem Inf Model. 2013 Aug 26;53(8):1842-52
references.pmid: 23617227
references.citation: Biophys J. 2006 Jun 15;90(12):4558-73
references.pmid: 16565070
references.citation: J Med Chem. 2004 Jan 15;47(2):303-24
references.pmid: 14711304
references.citation: J Chem Inf Model. 2010 Apr 26;50(4):572-84
references.pmid: 20235588
references.citation: J Med Chem. 2004 Dec 30;47(27):6804-11
references.pmid: 15615529
references.citation: J Comput Aided Mol Des. 2010 Apr;24(4):257-8
references.pmid: 20425136
references.citation: J Comput Aided Mol Des. 2012 Jun;26(6):775-86
references.pmid: 22566074
references.citation: Int J Mol Sci. 2010 Aug 18;11(8):3016-34
references.pmid: 21152288
references.citation: J Comput Aided Mol Des. 2009 Jun;23(6):355-74
references.pmid: 19340588
references.citation: J Med Chem. 2009 Oct 22;52(20):6362-8
references.pmid: 19827834
references.citation: Curr Pharm Des. 2002;8(17):1571-8
references.pmid: 12052201
references.citation: J Med Chem. 2006 Oct 5;49(20):5912-31
references.pmid: 17004707
references.citation: J Chem Inf Model. 2009 Apr;49(4):1079-93
references.pmid: 19358517
references.citation: J Med Chem. 2004 Mar 25;47(7):1750-9
references.pmid: 15027866
references.citation: J Comput Aided Mol Des. 2008 May;22(5):269-86
references.pmid: 18273558
references.citation: J Phys Chem B. 2009 Apr 9;113(14):4501-7
references.pmid: 19338360
references.citation: J Chem Inf Comput Sci. 2004 May-Jun;44(3):871-81
references.pmid: 15154752