Validation tools for variable subset regression.
Variable selection is applied frequently in QSAR research. Since the selection process influences the characteristics of the finally chosen model, thorough validation of the selection technique is very important. Here, a validation protocol is presented briefly and two of the tools which are part of this protocol are introduced in more detail. The first tool, which is based on permutation testing, allows to assess the inflation of internal figures of merit (such as the cross-validated prediction error). The other tool, based on noise addition, can be used to determine the complexity and with it the stability of models generated by variable selection. The obtained statistical information is important in deciding whether or not to trust the predictive abilities of a specific model. The graphical output of the validation tools is easily accessible and provides a reliable impression of model performance. Among others, the tools were employed to study the influence of leave-one-out and leave-multiple-out cross-validation on model characteristics. Here, it was confirmed that leave-multiple-out cross-validation yields more stable models. To study the performance of the entire validation protocol, it was applied to eight different QSAR data sets with default settings. In all cases internal and external model performance was good, indicating that the protocol serves its purpose quite well.
Baumann
Knut
K
Department of Pharmacy and Food Chemistry, University of Wuerzburg, Am Hubland, D-97074 Wuerzburg, Germany. knut.baumann@mail.uni-wuerzburg.de
Stiefl
Nikolaus
N
D021281:Quantitative Structure-Activity Relationship; D012044:Regression Analysis
Bioorg Med Chem. 1999 Nov;7(11):2283-92
10632038
J Math Psychol. 2000 Mar;44(1):41-61
10733857
J Med Chem. 1972 Oct;15(10):1066-8
5069775
J Med Chem. 2001 Nov 8;44(23):3786-94
11689065
J Med Chem. 2003 Apr 10;46(8):1390-407
12672239
J Clin Lab Anal. 1996;10(3):119-24
8731497
J Comput Aided Mol Des. 2000 Mar;14(3):293-306
10756483
J Comput Aided Mol Des. 2003 May-Jun;17(5-6):347-65
14635727
J Am Chem Soc. 1988 Aug 1;110(18):5959-67
22148765
J Med Chem. 1999 Feb 25;42(4):573-83
10052964
J Comput Aided Mol Des. 2001 May;15(5):395-410
11394735
J Med Chem. 1979 Oct;22(10):1238-44
513071
J Med Chem. 1995 Feb 17;38(4):659-68
7861414
J Med Chem. 1998 Jul 2;41(14):2553-64
9651159
Toxicol Sci. 2001 Feb;59(2):335-45
11158727
J Mol Graph Model. 2002 Jan;20(4):269-76
11858635
J Med Chem. 1997 Dec 19;40(26):4347-59
9435904
J Med Chem. 1990 Jan;33(1):136-42
2296013
J Chem Inf Comput Sci. 2002 Jan-Feb;42(1):26-35
11855963