title: Behavioral Signal Processing: Deriving Human Behavioral Informatics From Speech and Language: Computational techniques are presented to analyze and model expressed and perceived human behavior-variedly characterized as typical, atypical, distressed, and disordered-from speech and language cues and their applications in health, commerce, education, and beyond.
abstract: The expression and experience of human behavior are complex and multimodal and characterized by individual and contextual heterogeneity and variability. Speech and spoken language communication cues offer an important means for measuring and modeling human behavior. Observational research and practice across a variety of domains from commerce to healthcare rely on speech- and language-based informatics for crucial assessment and diagnostic information and for planning and tracking response to an intervention. In this paper, we describe some of the opportunities as well as emerging methodologies and applications of human behavioral signal processing (BSP) technology and algorithms for quantitatively understanding and modeling typical, atypical, and distressed human behavior with a specific focus on speech- and language-based communicative, affective, and social behavior. We describe the three important BSP components of acquiring behavioral data in an ecologically valid manner across laboratory to real-world settings, extracting and analyzing behavioral cues from measured data, and developing models offering predictive and decision-making support. We highlight both the foundational speech and language processing building blocks as well as the novel processing and modeling opportunities. Using examples drawn from specific real-world applications ranging from literacy assessment and autism diagnostics to psychotherapy for addiction and marital well being, we illustrate behavioral informatics applications of these signal processing techniques that contribute to quantifying higher level, often subjectively described, human behavior in a domain-sensitive fashion.
authors.affiliation: The authors are with the Ming Hsieh Department of Electrical Engineering, University of Southern California, Los Angeles, CA 90089 USA.
authors.forename: Shrikanth
authors.initials: S
authors.lastname: Narayanan
authors.forename: Panayiotis G
authors.initials: PG
authors.lastname: Georgiou
keywords: Affective computing; behavior; computational psychology; computational social sciences; emotions; health applications; multimodal signal processing; natural language processing; speech understanding
references.citation: J Autism Dev Disord. 2007 Apr;37(4):613-27
references.pmid: 17180459
references.citation: IEEE Trans Audio Speech Lang Process. 2007 Nov 1;15(8):2190-2201
references.pmid: 20428476
references.citation: J Speech Lang Hear Res. 2007 Aug;50(4):1015-28
references.pmid: 17675602
references.citation: J Autism Dev Disord. 2000 Jun;30(3):205-23
references.pmid: 11055457
references.citation: J Consult Clin Psychol. 2004 Apr;72(2):176-91
references.pmid: 15065953
references.citation: IEEE Trans Pattern Anal Mach Intell. 2013 Apr;35(4):769-83
references.pmid: 22732663
references.citation: Psychol Assess. 2005 Sep;17(3):288-307
references.pmid: 16262455
references.citation: IEEE Trans Audio Speech Lang Process. 2008 Jan;16(1):216-228
references.pmid: 19122857
references.citation: J Exp Psychol Hum Percept Perform. 2005 Feb;31(1):62-79
references.pmid: 15709863
references.citation: IEEE Trans Audio Speech Lang Process. 2008;16(4):797-811
references.pmid: 19603083
references.citation: J Speech Lang Hear Res. 2001 Oct;44(5):1097-115
references.pmid: 11708530
references.citation: IEEE Trans Audio Speech Lang Process. 2009 Jul 1;17(5):1009-1024
references.pmid: 20084186
references.citation: Pediatrics. 2010 Jan;125(1):e17-23
references.pmid: 19948568
references.citation: IEEE Trans Inf Technol Biomed. 2010 Mar;14(2):215-23
references.pmid: 20064760
references.citation: J Fam Psychol. 2004 Mar;18(1):58-71
references.pmid: 14992610
references.citation: J Psycholinguist Res. 2003 May;32(3):335-54
references.pmid: 12845943
references.citation: IEEE Trans Pattern Anal Mach Intell. 2005 Nov;27(11):1805-19
references.pmid: 16285378
references.citation: Psychol Assess. 2012 Dec;24(4):983-94
references.pmid: 22708571
references.citation: Dev Psychopathol. 1997 Spring;9(2):421-52
references.pmid: 9201451
references.citation: Proc Natl Acad Sci U S A. 2010 Jul 27;107(30):13354-9
references.pmid: 20643944
references.citation: Signal Inf Process Assoc Annu Summit Conf APSIPA Asia Pac. 2012 Dec;2012:
references.pmid: 27602411
references.citation: J Consult Clin Psychol. 2003 Oct;71(5):862-78
references.pmid: 14516235
references.citation: J Acoust Soc Am. 1972 Oct;52(4):1238-50
references.pmid: 4638039
references.citation: IEEE Trans Audio Speech Lang Process. 2007 Feb 1;15(2):690-701
references.pmid: 20454538
references.citation: Clin Child Fam Psychol Rev. 1998 Dec;1(4):195-213
references.pmid: 11324091
references.citation: IEEE Trans Pattern Anal Mach Intell. 2007 Feb;29(2):300-12
references.pmid: 17170482
references.citation: IEEE Trans Neural Syst Rehabil Eng. 2010 Aug;18(4):369-80
references.pmid: 20699202
references.citation: Psychol Bull. 2000 Nov;126(6):829-43
references.pmid: 11107878
references.citation: J Fam Psychol. 2005 Mar;19(1):157-67
references.pmid: 15796661
references.citation: Psychol Bull. 1968 Oct;70(4):213-20
references.pmid: 19673146
references.citation: IEEE Signal Process Lett. 2009 Sep;16(9):810-813
references.pmid: 24453471
references.citation: J Consult Clin Psychol. 2009 Feb;77(1):160-73
references.pmid: 19170462
references.citation: J Acoust Soc Am. 2006 Apr;119(4):2382-93
references.pmid: 16642851
references.citation: Annu Rev Clin Psychol. 2008;4:1-32
references.pmid: 18509902
references.citation: J Phys Act Health. 2012 Mar;9(3):432-41
references.pmid: 21934162