Multimodal emotion recognition by combining physiological signals and facial expressions: a preliminary study.
Lately, multimodal approaches for automatic emotion recognition have gained significant scientific interest. In this paper, emotion recognition by combining physiological signals and facial expressions was studied. Heart rate variability parameters, respiration frequency, and facial expressions were used to classify person's emotions while watching pictures with emotional content. Three classes were used for both valence and arousal. The preliminary results show that, over the proposed channels, detecting arousal seem to be easier compared to valence. While the classification performance of 54.5% was attained with arousal, only 38.0% of the samples were classified correctly in terms of valence. In future, additional modalities as well as feature selection will be utilized to improve the results.
Department of Computer Science and Engineering, BOX 4500, FIN-90014 University of Oulu, Finland. jukka.kortelainen@ee.oulu.fi
Jukka
J
Kortelainen
Suvi
S
Tiinanen
Xiaohua
X
Huang
Xiaobai
X
Li
Seppo
S
Laukka
Matti
M
Pietikäinen
Tapio
T
Seppänen
D000465:Algorithms; D004562:Electrocardiography / Q000379:methods*; D004644:Emotions / Q000502:physiology*; D005149:Facial Expression; D006339:Heart Rate / Q000502:physiology*; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D010363:Pattern Recognition, Automated / Q000379:methods*; D010865:Pilot Projects; D011594:Psychometrics / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity