title: Vision-based online recognition of surgical activities.
abstract: OBJECTIVE
  Surgical processes are complex entities characterized by expressive models and data. Recognizable activities define each surgical process. The principal limitation of current vision-based recognition methods is inefficiency due to the large amount of information captured during a surgical procedure. To overcome this technical challenge, we introduce a surgical gesture recognition system using temperature-based recognition.
  
  
  METHODS
  An infrared thermal camera was combined with a hierarchical temporal memory and was used during surgical procedures. The recordings were analyzed for recognition of surgical activities. The image sequence information acquired included hand temperatures. This datum was analyzed to perform gesture extraction and recognition based on heat differences between the surgeon's warm hands and the colder background of the environment.
  
  
  RESULTS
  The system was validated by simulating a functional endoscopic sinus surgery, a common type of otolaryngologic surgery. The thermal camera was directed toward the hands of the surgeon while handling different instruments. The system achieved an online recognition accuracy of 96% with high precision and recall rates of approximately 60%.
  
  
  CONCLUSIONS
  Vision-based recognition methods are the current best practice approaches for monitoring surgical processes. Problems of information overflow and extended recognition times in vision-based approaches were overcome by changing the spectral range to infrared. This change enables the real-time recognition of surgical activities and provides online monitoring information to surgical assistance systems and workflow management systems.
authors.affiliation: Innovation Center Computer Assisted Surgery, University of Leipzig, Semmelweisstr. 14, Leipzig, 04103, Germany. michael.unger@medizin.uni-leipzig.de.
authors.forename: Michael
authors.initials: M
authors.lastname: Unger
authors.affiliation: Innovation Center Computer Assisted Surgery, University of Leipzig, Semmelweisstr. 14, Leipzig, 04103, Germany.
authors.forename: Claire
authors.initials: C
authors.lastname: Chalopin
authors.affiliation: Innovation Center Computer Assisted Surgery, University of Leipzig, Semmelweisstr. 14, Leipzig, 04103, Germany.
authors.forename: Thomas
authors.initials: T
authors.lastname: Neumuth
mesh_terms: D004724:Endoscopy / Q000379:methods*; D005868:Gestures; D006225:Hand; D006801:Humans; D008962:Models, Theoretical; D010256:Paranasal Sinuses / Q000601:surgery*; D010363:Pattern Recognition, Automated; D025321:Surgery, Computer-Assisted / Q000379:methods; D057188:Workflow
keywords: Computer-assisted surgery; Intraoperative monitoring; Surgical activity recognition; Surgical process model; Thermal imaging; Workflow