title: Data-driven spatio-temporal RGBD feature encoding for action recognition in operating rooms.
abstract: OBJECTIVE
  Context-aware systems for the operating room (OR) provide the possibility to significantly improve surgical workflow through various applications such as efficient OR scheduling, context-sensitive user interfaces, and automatic transcription of medical procedures. Being an essential element of such a system, surgical action recognition is thus an important research area. In this paper, we tackle the problem of classifying surgical actions from video clips that capture the activities taking place in the OR.
  
  
  METHODS
  We acquire recordings using a multi-view RGBD camera system mounted on the ceiling of a hybrid OR dedicated to X-ray-based procedures and annotate clips of the recordings with the corresponding actions. To recognize the surgical actions from the video clips, we use a classification pipeline based on the bag-of-words (BoW) approach. We propose a novel feature encoding method that extends the classical BoW approach. Instead of using the typical rigid grid layout to divide the space of the feature locations, we propose to learn the layout from the actual 4D spatio-temporal locations of the visual features. This results in a data-driven and non-rigid layout which retains more spatio-temporal information compared to the rigid counterpart.
  
  
  RESULTS
  We classify multi-view video clips from a new dataset generated from 11-day recordings of real operations. This dataset is composed of 1734 video clips of 15 actions. These include generic actions (e.g., moving patient to the OR bed) and actions specific to the vertebroplasty procedure (e.g., hammering). The experiments show that the proposed non-rigid feature encoding method performs better than the rigid encoding one. The classifier's accuracy is increased by over 4 %, from 81.08 to 85.53 %.
  
  
  CONCLUSIONS
  The combination of both intensity and depth information from the RGBD data provides more discriminative power in carrying out the surgical action recognition task as compared to using either one of them alone. Furthermore, the proposed non-rigid spatio-temporal feature encoding scheme provides more discriminative histogram representations than the rigid counterpart. To the best of our knowledge, this is also the first work that presents action recognition results on multi-view RGBD data recorded in the OR.
authors.affiliation: ICube Laboratory, University of Strasbourg, CNRS, IHU Strasbourg, Strasbourg, France, twinanda@unistra.fr.
authors.forename: Andru P
authors.initials: AP
authors.lastname: Twinanda
authors.forename: Emre O
authors.initials: EO
authors.lastname: Alkan
authors.forename: Afshin
authors.initials: A
authors.lastname: Gangi
authors.forename: Michel
authors.initials: M
authors.lastname: de Mathelin
authors.forename: Nicolas
authors.initials: N
authors.lastname: Padoy
mesh_terms: D000465:Algorithms; D003199:Computer Systems; D006801:Humans; D009873:Operating Rooms; D025321:Surgery, Computer-Assisted; D014741:Video Recording
references.citation: Int J Comput Assist Radiol Surg. 2015 Aug;10(8):1181-91
references.pmid: 25712916
references.citation: IEEE Trans Biomed Eng. 2012 Apr;59(4):966-76
references.pmid: 22203700
references.citation: Med Image Comput Comput Assist Interv. 2010;13(Pt 3):400-7
references.pmid: 20879425
references.citation: Med Image Anal. 2012 Apr;16(3):632-41
references.pmid: 21195015
references.citation: Med Image Anal. 2013 Oct;17(7):732-45
references.pmid: 23706754
references.citation: AMIA Jt Summits Transl Sci Proc. 2013 Mar 18;2013:136-40
references.pmid: 24303253