title: Optimal Modality Selection for Cooperative Human-Robot Task Completion.
abstract: Human-robot cooperation in complex environments must be fast, accurate, and resilient. This requires efficient communication channels where robots need to assimilate information using a plethora of verbal and nonverbal modalities such as hand gestures, speech, and gaze. However, even though hybrid human-robot communication frameworks and multimodal communication have been studied, a systematic methodology for designing multimodal interfaces does not exist. This paper addresses the gap by proposing a novel methodology to generate multimodal lexicons which maximizes multiple performance metrics over a wide range of communication modalities (i.e., lexicons). The metrics are obtained through a mixture of simulation and real-world experiments. The methodology is tested in a surgical setting where a robot cooperates with a surgeon to complete a mock abdominal incision and closure task by delivering surgical instruments. Experimental results show that predicted optimal lexicons significantly outperform predicted suboptimal lexicons (p <; 0.05) in all metrics validating the predictability of the methodology. The methodology is validated in two scenarios (with and without modeling the risk of a human-robot collision) and the differences in the lexicons are analyzed.
authors.forename: Mithun George
authors.initials: MG
authors.lastname: Jacob
authors.forename: Juan P
authors.initials: JP
authors.lastname: Wachs
mesh_terms: D000465:Algorithms; D003198:Computer Simulation; D003491:Cybernetics / Q000379:methods* / Q000592:standards*; D005403:Fixation, Ocular; D013502:General Surgery / Q000193:education; D005868:Gestures; D006801:Humans; D012371:Robotics / Q000379:methods*; D013647:Task Performance and Analysis; D014584:User-Computer Interface