title: Towards an intelligent framework for multimodal affective data analysis.
abstract: An increasingly large amount of multimodal content is posted on social media websites such as YouTube and Facebook everyday. In order to cope with the growth of such so much multimodal data, there is an urgent need to develop an intelligent multi-modal analysis framework that can effectively extract information from multiple modalities. In this paper, we propose a novel multimodal information extraction agent, which infers and aggregates the semantic and affective information associated with user-generated multimodal data in contexts such as e-learning, e-health, automatic video content tagging and human-computer interaction. In particular, the developed intelligent agent adopts an ensemble feature extraction approach by exploiting the joint use of tri-modal (text, audio and video) features to enhance the multimodal information extraction process. In preliminary experiments using the eNTERFACE dataset, our proposed multi-modal system is shown to achieve an accuracy of 87.95%, outperforming the best state-of-the-art system by more than 10%, or in relative terms, a 56% reduction in error rate.
authors.affiliation: School of Natural Sciences, University of Stirling, UK. Electronic address: soujanya.poria@cs.stir.ac.uk.
authors.forename: Soujanya
authors.initials: S
authors.lastname: Poria
authors.affiliation: School of Computer Engineering, Nanyang Technological University, Singapore. Electronic address: cambria@ntu.edu.sg.
authors.forename: Erik
authors.initials: E
authors.lastname: Cambria
authors.affiliation: School of Natural Sciences, University of Stirling, UK. Electronic address: ahu@cs.stir.ac.uk.
authors.forename: Amir
authors.initials: A
authors.lastname: Hussain
authors.affiliation: School of Electrical & Electronic Engineering, Nanyang Technological University, Singapore. Electronic address: egbhuang@ntu.edu.sg.
authors.forename: Guang-Bin
authors.initials: GB
authors.lastname: Huang
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D056667:Biometric Identification / Q000379:methods*; D006801:Humans; D016247:Information Storage and Retrieval / Q000379:methods*
keywords: Affective computing; Emotion analysis; Facial expressions; Multimodal; Multimodal sentiment analysis; Speech; Text