Hand gesture guided robot-assisted surgery based on a direct augmented reality interface.
Radiofrequency (RF) ablation is a good alternative to hepatic resection for treatment of liver tumors. However, accurate needle insertion requires precise hand-eye coordination and is also affected by the difficulty of RF needle navigation. This paper proposes a cooperative surgical robot system, guided by hand gestures and supported by an augmented reality (AR)-based surgical field, for robot-assisted percutaneous treatment. It establishes a robot-assisted natural AR guidance mechanism that incorporates the advantages of the following three aspects: AR visual guidance information, surgeon's experiences and accuracy of robotic surgery. A projector-based AR environment is directly overlaid on a patient to display preoperative and intraoperative information, while a mobile surgical robot system implements specified RF needle insertion plans. Natural hand gestures are used as an intuitive and robust method to interact with both the AR system and surgical robot. The proposed system was evaluated on a mannequin model. Experimental results demonstrated that hand gesture guidance was able to effectively guide the surgical robot, and the robot-assisted implementation was found to improve the accuracy of needle insertion. This human-robot cooperative mechanism is a promising approach for precise transcutaneous ablation therapy.
Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore 117576, Singapore.
Rong
R
Wen
Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore 117583, Singapore.
Wei-Liang
WL
Tay
Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore 117576, Singapore. Electronic address: mpenp@nus.edu.sg.
Binh P
BP
Nguyen
Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore 117576, Singapore.
Chin-Boon
CB
Chng
Department of Mechanical Engineering, National University of Singapore, 9 Engineering Drive 1, Singapore 117576, Singapore.
Chee-Kong
CK
Chui
D017115:Catheter Ablation / Q000295:instrumentation; D005868:Gestures; D006801:Humans; D008113:Liver Neoplasms / Q000601:surgery; D008348:Manikins; D009339:Needles; D019047:Phantoms, Imaging; D065287:Robotic Surgical Procedures / Q000295:instrumentation; D012371:Robotics / Q000295:instrumentation; D025321:Surgery, Computer-Assisted / Q000295:instrumentation; D014584:User-Computer Interface
Augmented interaction; Augmented reality; Human–robot cooperation; Image-guided surgery; Projector-camera system; Visual guidance