title: Beware of machine learning-based scoring functions-on the danger of developing black boxes.
abstract: Training machine learning algorithms with protein-ligand descriptors has recently gained considerable attention to predict binding constants from atomic coordinates. Starting from a series of recent reports stating the advantages of this approach over empirical scoring functions, we could indeed reproduce the claimed superiority of Random Forest and Support Vector Machine-based scoring functions to predict experimental binding constants from protein-ligand X-ray structures of the PDBBind dataset. Strikingly, these scoring functions, trained on simple protein-ligand element-element distance counts, were almost unable to enrich virtual screening hit lists in true actives upon docking experiments of 10 reference DUD-E datasets; this is a a feature that, however, has been verified for an a priori less-accurate empirical scoring function (Surflex-Dock). By systematically varying ligand poses from true X-ray coordinates, we show that the Surflex-Dock scoring function is logically sensitive to the quality of docking poses. Conversely, our machine-learning based scoring functions are totally insensitive to docking poses (up to 10 Å root-mean square deviations) and just describe atomic element counts. This report does not disqualify using machine learning algorithms to design scoring functions. Protein-ligand element-element distance counts should however be used with extreme caution and only applied in a meaningful way. To avoid developing novel but meaningless scoring functions, we propose that two additional benchmarking tests must be systematically done when developing novel scoring functions: (i) sensitivity to docking pose accuracy, and (ii) ability to enrich hit lists in true actives upon structure-based (docking, receptor-ligand pharmacophore) virtual screening of reference datasets.
authors.affiliation: Laboratoire d'Innovation Thérapeutique, UMR 7200 CNRS-Université de Strasbourg , 74 route du Rhin, F-67400 Illkirch, France.
authors.forename: Joffrey
authors.initials: J
authors.lastname: Gabel
authors.forename: Jérémy
authors.initials: J
authors.lastname: Desaphy
authors.forename: Didier
authors.initials: D
authors.lastname: Rognan
mesh_terms: D016477:Artifacts; D001665:Binding Sites; D018360:Crystallography, X-Ray; D062126:Databases, Chemical; D004791:Enzyme Inhibitors / Q000737:chemistry*; D057166:High-Throughput Screening Assays; D006801:Humans; D008024:Ligands; D062105:Molecular Docking Simulation; D056004:Molecular Dynamics Simulation; D011485:Protein Binding; D011506:Proteins / Q000037:antagonists & inhibitors / Q000737:chemistry*; D012107:Research Design; D054852:Small Molecule Libraries / Q000737:chemistry*; D060388:Support Vector Machine; D013816:Thermodynamics; D014584:User-Computer Interface
chemical_list: D004791:Enzyme Inhibitors; D008024:Ligands; D011506:Proteins; D054852:Small Molecule Libraries