title: Multimodal emotional state recognition using sequence-dependent deep hierarchical features.
abstract: Emotional state recognition has become an important topic for human-robot interaction in the past years. By determining emotion expressions, robots can identify important variables of human behavior and use these to communicate in a more human-like fashion and thereby extend the interaction possibilities. Human emotions are multimodal and spontaneous, which makes them hard to be recognized by robots. Each modality has its own restrictions and constraints which, together with the non-structured behavior of spontaneous expressions, create several difficulties for the approaches present in the literature, which are based on several explicit feature extraction techniques and manual modality fusion. Our model uses a hierarchical feature representation to deal with spontaneous emotions, and learns how to integrate multiple modalities for non-verbal emotion recognition, making it suitable to be used in an HRI scenario. Our experiments show that a significant improvement of recognition accuracy is achieved when we use hierarchical features and multimodal information, and our model improves the accuracy of state-of-the-art approaches from 82.5% reported in the literature to 91.3% for a benchmark dataset on spontaneous emotion expressions.
authors.affiliation: Department of Informatics, University of Hamburg, Knowledge Technology, Vogt-Koelln-Strasse 30, 22527 Hamburg, Germany. Electronic address: barros@informatik.uni-hamburg.de.
authors.forename: Pablo
authors.initials: P
authors.lastname: Barros
authors.affiliation: Department of Informatics, University of Hamburg, Knowledge Technology, Vogt-Koelln-Strasse 30, 22527 Hamburg, Germany. Electronic address: jirak@informatik.uni-hamburg.de.
authors.forename: Doreen
authors.initials: D
authors.lastname: Jirak
authors.affiliation: Department of Informatics, University of Hamburg, Knowledge Technology, Vogt-Koelln-Strasse 30, 22527 Hamburg, Germany. Electronic address: weber@informatik.uni-hamburg.de.
authors.forename: Cornelius
authors.initials: C
authors.lastname: Weber
authors.affiliation: Department of Informatics, University of Hamburg, Knowledge Technology, Vogt-Koelln-Strasse 30, 22527 Hamburg, Germany. Electronic address: wermter@informatik.uni-hamburg.de.
authors.forename: Stefan
authors.initials: S
authors.lastname: Wermter
mesh_terms: D004644:Emotions; D006801:Humans; D007858:Learning; D021641:Recognition, Psychology; D012371:Robotics
keywords: Convolutional Neural Networks; Deep learning; Emotion recognition; Hierarchical features; Human Robot Interaction