title: Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration.
abstract: OBJECTIVE
  To investigate a markerless tracking system for real-time stereo-endoscopic visualization of preoperative computed tomographic imaging as an augmented display during robot-assisted laparoscopic partial nephrectomy.
  
  
  METHODS
  Stereoscopic video segments of a patient undergoing robot-assisted laparoscopic partial nephrectomy for tumor and another for a partial staghorn renal calculus were processed to evaluate the performance of a three-dimensional (3D)-to-3D registration algorithm. After both cases, we registered a segment of the video recording to the corresponding preoperative 3D-computed tomography image. After calibrating the camera and overlay, 3D-to-3D registration was created between the model and the surgical recording using a modified iterative closest point technique. Image-based tracking technology tracked selected fixed points on the kidney surface to augment the image-to-model registration.
  
  
  RESULTS
  Our investigation has demonstrated that we can identify and track the kidney surface in real time when applied to intraoperative video recordings and overlay the 3D models of the kidney, tumor (or stone), and collecting system semitransparently. Using a basic computer research platform, we achieved an update rate of 10 Hz and an overlay latency of 4 frames. The accuracy of the 3D registration was 1 mm.
  
  
  CONCLUSIONS
  Augmented reality overlay of reconstructed 3D-computed tomography images onto real-time stereo video footage is possible using iterative closest point and image-based surface tracking technology that does not use external navigation tracking systems or preplaced surface markers. Additional studies are needed to assess the precision and to achieve fully automated registration and display for intraoperative use.
authors.affiliation: Department of Urology, University of Florida College of Medicine, Gainesville, Florida 32610, USA. sulm@urology.ufl.edu
authors.forename: Li-Ming
authors.initials: LM
authors.lastname: Su
authors.forename: Balazs P
authors.initials: BP
authors.lastname: Vagvolgyi
authors.forename: Rahul
authors.initials: R
authors.lastname: Agarwal
authors.forename: Carol E
authors.initials: CE
authors.lastname: Reiley
authors.forename: Russell H
authors.initials: RH
authors.lastname: Taylor
authors.forename: Gregory D
authors.initials: GD
authors.lastname: Hager
mesh_terms: D000465:Algorithms; D003199:Computer Systems; D005240:Feasibility Studies; D006801:Humans; D021621:Imaging, Three-Dimensional; D007669:Kidney Calculi / Q000175:diagnosis* / Q000601:surgery*; D007680:Kidney Neoplasms / Q000175:diagnosis* / Q000601:surgery*; D010535:Laparoscopy / Q000379:methods*; D009392:Nephrectomy / Q000379:methods*; D012371:Robotics; D025321:Surgery, Computer-Assisted; D014057:Tomography, X-Ray Computed; D014741:Video Recording