title: Combining Video, Audio and Lexical Indicators of Affect in Spontaneous Conversation via Particle Filtering.
abstract: We present experiments on fusing facial video, audio and lexical indicators for affect estimation during dyadic conversations. We use temporal statistics of texture descriptors extracted from facial video, a combination of various acoustic features, and lexical features to create regression based affect estimators for each modality. The single modality regressors are then combined using particle filtering, by treating these independent regression outputs as measurements of the affect states in a Bayesian filtering framework, where previous observations provide prediction about the current state by means of learned affect dynamics. Tested on the Audio-visual Emotion Recognition Challenge dataset, our single modality estimators achieve substantially higher scores than the official baseline method for every dimension of affect. Our filtering-based multi-modality fusion achieves correlation performance of 0.344 (baseline: 0.136) and 0.280 (baseline: 0.096) for the fully continuous and word level sub challenges, respectively.
authors.affiliation: Department of Radiology, Information Science University of Pennsylvania Philadelphia, PA 19104.
authors.forename: Arman
authors.initials: A
authors.lastname: Savran
authors.affiliation: Department of Computer & Information Science University of Pennsylvania Philadelphia, PA 19104.
authors.forename: Houwei
authors.initials: H
authors.lastname: Cao
authors.forename: Miraj
authors.initials: M
authors.lastname: Shah
authors.forename: Ani
authors.initials: A
authors.lastname: Nenkova
authors.forename: Ragini
authors.initials: R
authors.lastname: Verma
keywords: adaboost; affective computing; class-spectral features; emotion dynamics; emotion recognition; lexical analysis; local binary patterns; multi-modality fusion; particle filtering; svm