title: Automatic scoring of virtual mastoidectomies using expert examples.
abstract: OBJECTIVE
  Automatic scoring of resident performance on a virtual mastoidectomy simulation system is needed to achieve consistent and efficient evaluations. By not requiring immediate expert intervention, the system provides a completely objective assessment of performance as well as a self-driven user assessment mechanism.
  
  
  METHODS
  An iconic temporal bone with surgically important regions defined into a fully partitioned segmented dataset was created. Comparisons between expert-drilled bones and student-drilled bones were computed based on gradations with both Euclidean and Earth Mover's Distance. Using the features derived from these comparisons, a decision tree was constructed. This decision tree was used to determine scores of resident surgical performance. The algorithm was applied on multiple expert comparison bones and the scores averaged to provide reliability metric.
  
  
  RESULTS
  The reliability metrics for the multi-grade scoring system are better in some cases than previously reported binary classification metrics. The two scoring methods given provide a trade-off between accuracy and speed.
  
  
  CONCLUSIONS
  Comparison of virtually drilled bones with expert examples on a voxel level provides sufficient information to score them and provide several specific quality metrics. By merging scores from different expert examples, two related metrics were developed; one is slightly faster and less accurate, while a second is more accurate but takes more processing time.
authors.affiliation: Ohio Supercomputer Center, Columbus, OH, USA. kerwin@osc.edu
authors.forename: Thomas
authors.initials: T
authors.lastname: Kerwin
authors.forename: Gregory
authors.initials: G
authors.lastname: Wiet
authors.forename: Don
authors.initials: D
authors.lastname: Stredney
authors.forename: Han-Wei
authors.initials: HW
authors.lastname: Shen
mesh_terms: D000465:Algorithms; D002983:Clinical Competence; D003663:Decision Trees; D006801:Humans; D007396:Internship and Residency; D008416:Mastoid / Q000601:surgery*; D013517:Otorhinolaryngologic Surgical Procedures / Q000193:education*; D015203:Reproducibility of Results; D014584:User-Computer Interface