title: Validation tools for variable subset regression.
abstract: Variable selection is applied frequently in QSAR research. Since the selection process influences the characteristics of the finally chosen model, thorough validation of the selection technique is very important. Here, a validation protocol is presented briefly and two of the tools which are part of this protocol are introduced in more detail. The first tool, which is based on permutation testing, allows to assess the inflation of internal figures of merit (such as the cross-validated prediction error). The other tool, based on noise addition, can be used to determine the complexity and with it the stability of models generated by variable selection. The obtained statistical information is important in deciding whether or not to trust the predictive abilities of a specific model. The graphical output of the validation tools is easily accessible and provides a reliable impression of model performance. Among others, the tools were employed to study the influence of leave-one-out and leave-multiple-out cross-validation on model characteristics. Here, it was confirmed that leave-multiple-out cross-validation yields more stable models. To study the performance of the entire validation protocol, it was applied to eight different QSAR data sets with default settings. In all cases internal and external model performance was good, indicating that the protocol serves its purpose quite well.
authors.affiliation: Department of Pharmacy and Food Chemistry, University of Wuerzburg, Am Hubland, D-97074 Wuerzburg, Germany. knut.baumann@mail.uni-wuerzburg.de
authors.forename: Knut
authors.initials: K
authors.lastname: Baumann
authors.forename: Nikolaus
authors.initials: N
authors.lastname: Stiefl
mesh_terms: D021281:Quantitative Structure-Activity Relationship; D012044:Regression Analysis
references.citation: Bioorg Med Chem. 1999 Nov;7(11):2283-92
references.pmid: 10632038
references.citation: J Math Psychol. 2000 Mar;44(1):41-61
references.pmid: 10733857
references.citation: J Med Chem. 1972 Oct;15(10):1066-8
references.pmid: 5069775
references.citation: J Med Chem. 2001 Nov 8;44(23):3786-94
references.pmid: 11689065
references.citation: J Med Chem. 2003 Apr 10;46(8):1390-407
references.pmid: 12672239
references.citation: J Clin Lab Anal. 1996;10(3):119-24
references.pmid: 8731497
references.citation: J Comput Aided Mol Des. 2000 Mar;14(3):293-306
references.pmid: 10756483
references.citation: J Comput Aided Mol Des. 2003 May-Jun;17(5-6):347-65
references.pmid: 14635727
references.citation: J Am Chem Soc. 1988 Aug 1;110(18):5959-67
references.pmid: 22148765
references.citation: J Med Chem. 1999 Feb 25;42(4):573-83
references.pmid: 10052964
references.citation: J Comput Aided Mol Des. 2001 May;15(5):395-410
references.pmid: 11394735
references.citation: J Med Chem. 1979 Oct;22(10):1238-44
references.pmid: 513071
references.citation: J Med Chem. 1995 Feb 17;38(4):659-68
references.pmid: 7861414
references.citation: J Med Chem. 1998 Jul 2;41(14):2553-64
references.pmid: 9651159
references.citation: Toxicol Sci. 2001 Feb;59(2):335-45
references.pmid: 11158727
references.citation: J Mol Graph Model. 2002 Jan;20(4):269-76
references.pmid: 11858635
references.citation: J Med Chem. 1997 Dec 19;40(26):4347-59
references.pmid: 9435904
references.citation: J Med Chem. 1990 Jan;33(1):136-42
references.pmid: 2296013
references.citation: J Chem Inf Comput Sci. 2002 Jan-Feb;42(1):26-35
references.pmid: 11855963