title: Optical merger of direct vision with virtual images for scaled teleoperation.
abstract: Scaled teleoperation is increasingly prevalent in medicine, as well as in other applications of robotics. Visual feedback in such systems is essential and should make maximal use of natural hand-eye coordination. This paper describes a new method of visual feedback for scaled teleoperation in which the operator manipulates the handle of a remote tool in the presence of a registered virtual image of the target in real time. The method adapts a concept already used successfully in a new medical device called the Sonic Flashlight, which permits direct in situ visualization of ultrasound during invasive procedures. The Sonic Flashlight uses a flat-panel monitor and a half-silvered mirror to merge the visual outer surface of a patient with a simultaneous ultrasound scan of the patient's interior. Adapting the concept to scaled teleoperation involves removing the imaging device and the target to a remote location and adding a master-slave control device. This permits the operator to see his hands, along with what appears to be the tool, and the target, merged in a workspace that preserves natural hand-eye coordination. Three functioning prototypes are described, one based on ultrasound and two on light microscopy. The limitations and potential of the new approach are discussed.
authors.affiliation: School of Medicine, University of Pittsburgh, PA 15214, USA. sclanton@oeic.net
authors.forename: Samuel T
authors.initials: ST
authors.lastname: Clanton
authors.forename: David C
authors.initials: DC
authors.lastname: Wang
authors.forename: Vikram S
authors.initials: VS
authors.lastname: Chib
authors.forename: Yoky
authors.initials: Y
authors.lastname: Matsuoka
authors.forename: George D
authors.initials: GD
authors.lastname: Stetten
mesh_terms: D000465:Algorithms; D003196:Computer Graphics; D004867:Equipment Design; D019544:Equipment Failure Analysis; D006801:Humans; D007089:Image Enhancement / Q000295:instrumentation; D007090:Image Interpretation, Computer-Assisted / Q000295:instrumentation*; D021621:Imaging, Three-Dimensional / Q000295:instrumentation*; D009716:Numerical Analysis, Computer-Assisted; D012371:Robotics / Q000295:instrumentation* / Q000379:methods; D012815:Signal Processing, Computer-Assisted; D013382:Subtraction Technique; D017216:Telemedicine / Q000295:instrumentation* / Q000379:methods; D014584:User-Computer Interface; D014796:Visual Perception / Q000502:physiology*