title: Automatic Behavior Analysis During a Clinical Interview with a Virtual Human.
abstract: SimSensei is a Virtual Human (VH) interviewing platform that uses off-the-shelf sensors (i.e., webcams, Microsoft Kinect and a microphone) to capture and interpret real-time audiovisual behavioral signals from users interacting with the VH system. The system was specifically designed for clinical interviewing and health care support by providing a face-to-face interaction between a user and a VH that can automatically react to the inferred state of the user through analysis of behavioral signals gleaned from the user's facial expressions, body gestures and vocal parameters. Akin to how non-verbal behavioral signals have an impact on human-to-human interaction and communication, SimSensei aims to capture and infer user state from signals generated from user non-verbal communication to improve engagement between a VH and a user and to quantify user state from the data captured across a 20 minute interview. Results from of sample of service members (SMs) who were interviewed before and after a deployment to Afghanistan indicate that SMs reveal more PTSD symptoms to the VH than they report on the Post Deployment Health Assessment. Pre/Post deployment facial expression analysis indicated more sad expressions and few happy expressions at post deployment.
authors.affiliation: University of Southern California, Institute for Creative Technologies.
authors.forename: Albert
authors.initials: A
authors.lastname: Rizzo
authors.affiliation: University of Southern California, Institute for Creative Technologies.
authors.forename: Gale
authors.initials: G
authors.lastname: Lucas
authors.affiliation: University of Southern California, Institute for Creative Technologies.
authors.forename: Jonathan
authors.initials: J
authors.lastname: Gratch
authors.affiliation: University of Southern California, Institute for Creative Technologies.
authors.forename: Giota
authors.initials: G
authors.lastname: Stratou
authors.affiliation: Carnegie Mellon University.
authors.forename: Louis-Philippe
authors.initials: LP
authors.lastname: Morency
authors.affiliation: Colorado National Guard.
authors.forename: Kenneth
authors.initials: K
authors.lastname: Chavez
authors.affiliation: U.S. Dept. of Education.
authors.forename: Russ
authors.initials: R
authors.lastname: Shilling
authors.affiliation: University of Southern California, Institute for Creative Technologies.
authors.forename: Stefan
authors.initials: S
authors.lastname: Scherer
mesh_terms: D000328:Adult; D001185:Artificial Intelligence; D003936:Diagnosis, Computer-Assisted / Q000379:methods*; D005260:Female; D005868:Gestures; D006801:Humans; D021621:Imaging, Three-Dimensional / Q000379:methods*; D007406:Interview, Psychological / Q000379:methods*; D008297:Male; D008328:Man-Machine Systems; D008875:Middle Aged; D008889:Military Personnel; D009633:Nonverbal Communication; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D013068:Speech Production Measurement / Q000379:methods*; D013313:Stress Disorders, Post-Traumatic / Q000175:diagnosis*; D014481:United States; D014584:User-Computer Interface