title: Honest assessments of automatic learning algorithm performance.
abstract: OBJECTIVE
  To compare methods of evaluating probabilistic predictors in systems that learn from examples.
  
  
  METHODS
  The performance of four automatic learning algorithms, representing current machine learning technology, were assessed using four methodologies in the task of separating normal squamous intermediate cervical cells from all other segmented objects in digital images. Two of the methodologies were carefully constructed to model sources of variation associated with the choice of training and test sets. These assessments were statistically compared with assessments using both standard and a modified version of cross-validation.
  
  
  RESULTS
  The investigation illustrates the tradeoffs involved in obtaining statistical rigor as compared with the cost of collecting data. While cross-validation makes frugal use of data, it can produce misleading assessments of algorithm performance in terms of both bias and variance. The modified version produces more reliable assessments but in some cases may also be misleading.
  
  
  CONCLUSIONS
  We suggest that users of learning algorithms should exercise judicious care in evaluating learning algorithm performance in order to avoid unnecessary bias and large variance in their assessments.
authors.affiliation: Department of Computer Science, University of Toronto, Ontario, Canada.
authors.forename: M
authors.initials: M
authors.lastname: Revow
authors.forename: D
authors.initials: D
authors.lastname: Maclean
mesh_terms: D000465:Algorithms; D000704:Analysis of Variance; D001185:Artificial Intelligence; D002584:Cervix Uteri / Q000166:cytology; D005069:Evaluation Studies as Topic; D005260:Female; D006801:Humans; D019044:Image Cytometry / Q000379:methods*; D007091:Image Processing, Computer-Assisted / Q000379:methods; D008403:Mass Screening / Q000379:methods; D015233:Models, Statistical; D015588:Observer Variation; D015203:Reproducibility of Results; D014626:Vaginal Smears