title: CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset.
abstract: People convey their emotional state in their face and voice. We present an audio-visual data set uniquely suited for the study of multi-modal emotion expression and perception. The data set consists of facial and vocal emotional expressions in sentences spoken in a range of basic emotional states (happy, sad, anger, fear, disgust, and neutral). 7,442 clips of 91 actors with diverse ethnic backgrounds were rated by multiple raters in three modalities: audio, visual, and audio-visual. Categorical emotion labels and real-value intensity values for the perceived emotion were collected using crowd-sourcing from 2,443 raters. The human recognition of intended emotion for the audio-only, visual-only, and audio-visual data are 40.9%, 58.2% and 63.6% respectively. Recognition rates are highest for neutral, followed by happy, anger, disgust, fear, and sad. Average intensity levels of emotion are rated highest for visual-only perception. The accurate recognition of disgust and fear requires simultaneous audio-visual cues, while anger and happiness can be well recognized based on evidence from a single modality. The large dataset we introduce can be used to probe other questions concerning the audio-visual perception of emotion.
authors: Radiology Department at the University of Pennsylvania, 3600 Market Street, Suite 380, Philadelphia, PA 19104. Houwei.Cao@uphs.upnne.edu.
authors: Houwei
authors: H
authors: Cao
authors: Math and Computer Science Department at Ursinus College, 601 E. Main Street, Collegeville, PA, 19426. dgc@alumni.cmu.edu.
authors: David G
authors: DG
authors: Cooper
authors: Department of Psychology at the University of Illinois at Chicago, 1007 West Harrison Street, M/C 285, Chicago, IL, 60607. michaelk@alumni.upenn.edu .
authors: Michael K
authors: MK
authors: Keutmann
authors: Neuropsychiatry section of the Psychiatry Department of the University of Pennsylvania 3400 Spruce Street, 10th Floor, Gates Bldg.,and the Philadelphia Veterans Administration Medical Center, Philadelphia, PA 19104. gur@mail.med.upenn.edu.
authors: Ruben C
authors: RC
authors: Gur
authors: Department of Computer and Information Science, University of Pennsylvania, 3330 Walnut Street, Philadelphia, PA 19104. nenkova@seas.upenn.edu .
authors: Ani
authors: A
authors: Nenkova
authors: Radiology Department at the University of Pennsylvania, 3600 Market Street, Suite 380, Philadelphia, PA 19104.
authors: Ragini
authors: R
authors: Verma
keywords: Emotional corpora; facial expression; multi-modal recognition; voice expression
references: J Acoust Soc Am. 1995 Jul;98(1):88-98
references: 7608410
references: Behav Res Methods. 2008 Nov;40(4):935-9
references: 19001384
references: J Acoust Soc Am. 1993 Feb;93(2):1097-108
references: 8445120
references: Neuroimage. 2011 Feb 1;54(3):2257-66
references: 20974266
references: Brain Res. 2008 Nov 25;1242:126-35
references: 18495094
references: Neural Netw. 2005 May;18(4):371-88
references: 15961273
references: J Acoust Soc Am. 1997 Apr;101(4):2267-77
references: 9104028
references: Rev Neurosci. 2012;23(4):381-92
references: 23089604
references: IEEE Trans Pattern Anal Mach Intell. 2005 May;27(5):812-6
references: 15875802
references: J Neurosci. 2011 Sep 21;31(38):13635-43
references: 21940454
references: Proc Natl Acad Sci U S A. 2010 Feb 9;107(6):2408-12
references: 20133790
references: Neuroimage. 2012 Mar;60(1):553-61
references: 22182770
references: J Pers Soc Psychol. 1996 Mar;70(3):614-36
references: 8851745