title: Combined 5 x 2 cv F test for comparing supervised classification learning algorithms.
abstract: Dietterich (1998) reviews five statistical tests and proposes the 5 x 2 cv t test for determining whether there is a significant difference between the error rates of two classifiers. In our experiments, we noticed that the 5 x 2 cv t test result may vary depending on factors that should not affect the test, and we propose a variant, the combined 5 x 2 cv F test, that combines multiple statistics to get a more robust test. Simulation results show that this combined version of the test has lower type I error and higher power than 5 x 2 cv proper.
authors.affiliation: IDIAP, CP 592 CH-1920 Martigny, Switzerland, and Department of Computer Engineering, Boğaziçi University, TR-80815 Istanbul, Turkey.
authors.forename: E
authors.initials: E
authors.lastname: Alpaydin
mesh_terms: D000465:Algorithms; D002965:Classification; D007858:Learning; D016571:Neural Networks, Computer; D013223:Statistics as Topic