title: Accurate visible speech synthesis based on concatenating variable length motion capture data.
abstract: We present a novel approach to synthesizing accurate visible speech based on searching and concatenating optimal variable-length units in a large corpus of motion capture data. Based on a set of visual prototypes selected on a source face and a corresponding set designated for a target face, we propose a machine learning technique to automatically map the facial motions observed on the source face to the target face. In order to model the long distance coarticulation effects in visible speech, a large-scale corpus that covers the most common syllables in English was collected, annotated and analyzed. For any input text, a search algorithm to locate the optimal sequences of concatenated units for synthesis is desrcribed. A new algorithm to adapt lip motions from a generic 3D face model to a specific 3D face model is also proposed. A complete, end-to-end visible speech animation system is implemented based on the approach. This system is currently used in more than 60 kindergarten through third grade classrooms to teach students to read using a lifelike conversational animated agent. To evaluate the quality of the visible speech produced by the animation system, both subjective evaluation and objective evaluation are conducted. The evaluation results show that the proposed approach is accurate and powerful for visible speech synthesis.
authors.affiliation: Center for Spoken Language Research, University of Colorado at Boulder, CO 80309-0594, USA. jiyong@cslr.colorado.edu
authors.forename: Jiyong
authors.initials: J
authors.lastname: Ma
authors.forename: Ron
authors.initials: R
authors.lastname: Cole
authors.forename: Bryan
authors.initials: B
authors.lastname: Pellom
authors.forename: Wayne
authors.initials: W
authors.lastname: Ward
authors.forename: Barbara
authors.initials: B
authors.lastname: Wise
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D003196:Computer Graphics; D003198:Computer Simulation; D005145:Face / Q000033:anatomy & histology* / Q000502:physiology*; D007089:Image Enhancement / Q000379:methods; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D021621:Imaging, Three-Dimensional / Q000379:methods*; D016247:Information Storage and Retrieval / Q000379:methods; D008954:Models, Biological; D009068:Movement / Q000502:physiology*; D010363:Pattern Recognition, Automated / Q000379:methods; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D013060:Speech / Q000502:physiology*; D013068:Speech Production Measurement / Q000379:methods; D014584:User-Computer Interface; D014741:Video Recording / Q000379:methods