Honest assessments of automatic learning algorithm performance.
OBJECTIVE
To compare methods of evaluating probabilistic predictors in systems that learn from examples.


METHODS
The performance of four automatic learning algorithms, representing current machine learning technology, were assessed using four methodologies in the task of separating normal squamous intermediate cervical cells from all other segmented objects in digital images. Two of the methodologies were carefully constructed to model sources of variation associated with the choice of training and test sets. These assessments were statistically compared with assessments using both standard and a modified version of cross-validation.


RESULTS
The investigation illustrates the tradeoffs involved in obtaining statistical rigor as compared with the cost of collecting data. While cross-validation makes frugal use of data, it can produce misleading assessments of algorithm performance in terms of both bias and variance. The modified version produces more reliable assessments but in some cases may also be misleading.


CONCLUSIONS
We suggest that users of learning algorithms should exercise judicious care in evaluating learning algorithm performance in order to avoid unnecessary bias and large variance in their assessments.
Revow
M
M
Department of Computer Science, University of Toronto, Ontario, Canada.
Maclean
D
D
D000465:Algorithms; D000704:Analysis of Variance; D001185:Artificial Intelligence; D002584:Cervix Uteri / Q000166:cytology; D005069:Evaluation Studies as Topic; D005260:Female; D006801:Humans; D019044:Image Cytometry / Q000379:methods*; D007091:Image Processing, Computer-Assisted / Q000379:methods; D008403:Mass Screening / Q000379:methods; D015233:Models, Statistical; D015588:Observer Variation; D015203:Reproducibility of Results; D014626:Vaginal Smears