title: Machine learning-based augmented reality for improved surgical scene understanding.
abstract: In orthopedic and trauma surgery, AR technology can support surgeons in the challenging task of understanding the spatial relationships between the anatomy, the implants and their tools. In this context, we propose a novel augmented visualization of the surgical scene that mixes intelligently the different sources of information provided by a mobile C-arm combined with a Kinect RGB-Depth sensor. Therefore, we introduce a learning-based paradigm that aims at (1) identifying the relevant objects or anatomy in both Kinect and X-ray data, and (2) creating an object-specific pixel-wise alpha map that permits relevance-based fusion of the video and the X-ray images within one single view. In 12 simulated surgeries, we show very promising results aiming at providing for surgeons a better surgical scene understanding as well as an improved depth perception.
authors.affiliation: Computer Aided Medical Procedures, Technische Universität, München, Germany; Institute of Biomathematics and Biometry, Helmholtz Zentrum, München, Germany. Electronic address: olivier.pauly@tum.de.
authors.forename: Olivier
authors.initials: O
authors.lastname: Pauly
authors.affiliation: Computer Aided Medical Procedures, Technische Universität, München, Germany.
authors.forename: Benoit
authors.initials: B
authors.lastname: Diotte
authors.affiliation: Computer Aided Medical Procedures, Technische Universität, München, Germany.
authors.forename: Pascal
authors.initials: P
authors.lastname: Fallavollita
authors.affiliation: Chirurgische Klinik und Poliklinik Innenstadt, München, Germany.
authors.forename: Simon
authors.initials: S
authors.lastname: Weidert
authors.affiliation: Chirurgische Klinik und Poliklinik Innenstadt, München, Germany.
authors.forename: Ekkehard
authors.initials: E
authors.lastname: Euler
authors.affiliation: Computer Aided Medical Procedures, Technische Universität, München, Germany.
authors.forename: Nassir
authors.initials: N
authors.lastname: Navab
mesh_terms: D000465:Algorithms; D004867:Equipment Design; D019544:Equipment Failure Analysis; D006801:Humans; D007089:Image Enhancement / Q000295:instrumentation / Q000379:methods; D007090:Image Interpretation, Computer-Assisted / Q000295:instrumentation / Q000379:methods*; D021621:Imaging, Three-Dimensional / Q000295:instrumentation / Q000379:methods; D000069550:Machine Learning; D064847:Multimodal Imaging / Q000295:instrumentation / Q000379:methods; D010363:Pattern Recognition, Automated / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D025321:Surgery, Computer-Assisted / Q000295:instrumentation / Q000379:methods*; D014057:Tomography, X-Ray Computed / Q000295:instrumentation / Q000379:methods*; D014584:User-Computer Interface; D014741:Video Recording / Q000295:instrumentation / Q000379:methods