title: Exploiting Listener Gaze to Improve Situated Communication in Dynamic Virtual Environments.
abstract: Beyond the observation that both speakers and listeners rapidly inspect the visual targets of referring expressions, it has been argued that such gaze may constitute part of the communicative signal. In this study, we investigate whether a speaker may, in principle, exploit listener gaze to improve communicative success. In the context of a virtual environment where listeners follow computer-generated instructions, we provide two kinds of support for this claim. First, we show that listener gaze provides a reliable real-time index of understanding even in dynamic and complex environments, and on a per-utterance basis. Second, we show that a language generation system that uses listener gaze to provide rapid feedback improves overall task performance in comparison with two systems that do not use gaze. Aside from demonstrating the utility of listener gaze in situated communication, our findings open the door to new methods for developing and evaluating multi-modal models of situated interaction.
authors.affiliation: HERE Berlin. konstantina.garoufi@here.com.
authors.forename: Konstantina
authors.initials: K
authors.lastname: Garoufi
authors.affiliation: Department of Computational Linguistics, Saarland University.
authors.forename: Maria
authors.initials: M
authors.lastname: Staudte
authors.affiliation: Department of Linguistics, University of Potsdam.
authors.forename: Alexander
authors.initials: A
authors.lastname: Koller
authors.affiliation: Department of Computational Linguistics, Saarland University.
authors.forename: Matthew W
authors.initials: MW
authors.lastname: Crocker
mesh_terms: D000328:Adult; D001288:Attention / Q000502:physiology*; D003142:Communication; D032882:Comprehension / Q000502:physiology*; D005133:Eye Movements / Q000502:physiology*; D005260:Female; D006801:Humans; D007802:Language; D008297:Male; D000076142:Virtual Reality
keywords: Eye-tracking; Listener gaze; Referential understanding; Situated communication; Virtual environments