title: Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation.
abstract: Autostereoscopic 3D image overlay for augmented reality (AR) based surgical navigation has been studied and reported many times. For the purpose of surgical overlay, the 3D image is expected to have the same geometric shape as the original organ, and can be transformed to a specified location for image overlay. However, how to generate a 3D image with high geometric fidelity and quantitative evaluation of 3D image's geometric accuracy have not been addressed. This paper proposes a graphics processing unit (GPU) based computer-generated integral imaging pipeline for real-time autostereoscopic 3D display, and an automatic closed-loop 3D image calibration paradigm for displaying undistorted 3D images. Based on the proposed methods, a novel AR device for 3D image surgical overlay is presented, which mainly consists of a 3D display, an AR window, a stereo camera for 3D measurement, and a workstation for information processing. The evaluation on the 3D image rendering performance with 2560×1600 elemental image resolution shows the rendering speeds of 50-60 frames per second (fps) for surface models, and 5-8 fps for large medical volumes. The evaluation of the undistorted 3D image after the calibration yields sub-millimeter geometric accuracy. A phantom experiment simulating oral and maxillofacial surgery was also performed to evaluate the proposed AR overlay device in terms of the image registration accuracy, 3D image overlay accuracy, and the visual effects of the overlay. The experimental results show satisfactory image registration and image overlay accuracy, and confirm the system usability.
authors.affiliation: Graduate School of Engineering, The University of Tokyo, Tokyo, Japan. Electronic address: wangjunchen@gmail.com.
authors.forename: Junchen
authors.initials: J
authors.lastname: Wang
authors.affiliation: Department of Oral-Maxillofacial Surgery, Dentistry and Orthodontics, The University of Tokyo Hospital, Tokyo, Japan.
authors.forename: Hideyuki
authors.initials: H
authors.lastname: Suenaga
authors.affiliation: Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China. Electronic address: liao@tsinghua.edu.cn.
authors.forename: Hongen
authors.initials: H
authors.lastname: Liao
authors.affiliation: Department of Oral-Maxillofacial Surgery, Dentistry and Orthodontics, The University of Tokyo Hospital, Tokyo, Japan.
authors.forename: Kazuto
authors.initials: K
authors.lastname: Hoshi
authors.affiliation: Graduate School of Engineering, The University of Tokyo, Tokyo, Japan.
authors.forename: Liangjing
authors.initials: L
authors.lastname: Yang
authors.affiliation: Graduate School of Engineering, The University of Tokyo, Tokyo, Japan.
authors.forename: Etsuko
authors.initials: E
authors.lastname: Kobayashi
authors.affiliation: Graduate School of Engineering, The University of Tokyo, Tokyo, Japan.
authors.forename: Ichiro
authors.initials: I
authors.lastname: Sakuma
mesh_terms: D000465:Algorithms; D002138:Calibration / Q000592:standards; D003199:Computer Systems; D006801:Humans; D007089:Image Enhancement / Q000379:methods / Q000592:standards; D007090:Image Interpretation, Computer-Assisted / Q000379:methods* / Q000592:standards; D021621:Imaging, Three-Dimensional / Q000379:methods* / Q000592:standards; D019647:Oral Surgical Procedures / Q000379:methods* / Q000592:standards; D010363:Pattern Recognition, Automated / Q000379:methods* / Q000592:standards; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D025321:Surgery, Computer-Assisted / Q000379:methods* / Q000592:standards; D014584:User-Computer Interface
keywords: 3D image calibration; Augmented reality; Integral imaging; Stereo tracking; Surgical navigation