title: Validation of Monte Carlo estimates of three-class ideal observer operating points for normal data.
abstract: OBJECTIVE
  Traditional two-class receiver operating characteristic (ROC) analysis is inadequate for the complete evaluation of observer performance in tasks with more than two classes.
  
  
  METHODS
  Here, a Monte Carlo estimation method for operating point coordinates on a three-class ROC surface is developed and compared with analytically calculated coordinates in two special cases: (1) univariate and (2) restricted bivariate trinormal underlying data.
  
  
  RESULTS
  In both cases, the statistical estimates were found to be good in the sense that the analytical values lay within the 95% confidence interval of the estimated values about 95% of the time.
  
  
  CONCLUSIONS
  The statistical estimation method should be key in the development of a pragmatic performance metric for evaluation of observers in classification tasks with three or more classes.
authors.affiliation: The Department of Physiology, 303 E Superior St., Northwestern University, Chicago, IL 60611, USA. darrin.c.edwards@gmail.com
authors.forename: Darrin C
authors.initials: DC
authors.lastname: Edwards
mesh_terms: D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D006801:Humans; D009010:Monte Carlo Method; D012372:ROC Curve; D015203:Reproducibility of Results