title: Blocked 3×2 cross-validated t-test for comparing supervised classification learning algorithms.
abstract: In the research of machine learning algorithms for classification tasks, the comparison of the performances of algorithms is extremely important, and a statistical test of significance for generalization error is often used to perform it in the machine learning literature. In view of the randomness of partitions in cross-validation, a new blocked 3×2 cross-validation is proposed to estimate generalization error in this letter. We then conduct an analysis of variance of the blocked 3×2 cross-validated estimator. A relatively conservative variance estimator that considers the correlation between any two two-fold cross-validations, and was previously neglected in 5×2 cross-validated t and F-tests is put forward. A corresponding test using this variance estimator is presented to compare the performances of algorithms. Simulated results show that the performance of our test is comparable with that of 5×2 cross-validated tests but with less computation complexity.
authors.affiliation: Computer Center of Shanxi University, Taiyuan 030006, P.R.C. wangyu@sxu.edu.cn.
authors.forename: Wang
authors.initials: W
authors.lastname: Yu
authors.forename: Wang
authors.initials: W
authors.lastname: Ruibo
authors.forename: Jia
authors.initials: J
authors.lastname: Huichen
authors.forename: Li
authors.initials: L
authors.lastname: Jihong