title: Audio-visual enhancement of speech in noise.
abstract: A key problem for telecommunication or human-machine communication systems concerns speech enhancement in noise. In this domain, a certain number of techniques exist, all of them based on an acoustic-only approach--that is, the processing of the audio corrupted signal using audio information (from the corrupted signal only or additive audio information). In this paper, an audio-visual approach to the problem is considered, since it has been demonstrated in several studies that viewing the speaker's face improves message intelligibility, especially in noisy environments. A speech enhancement prototype system that takes advantage of visual inputs is developed. A filtering process approach is proposed that uses enhancement filters estimated with the help of lip shape information. The estimation process is based on linear regression or simple neural networks using a training corpus. A set of experiments assessed by Gaussian classification and perceptual tests demonstrates that it is indeed possible to enhance simple stimuli (vowel-plosive-vowel sequences) embedded in white Gaussian noise.
authors: Institut de la Communication Parlée, INPG/Université Stendhal/CNRS UMR 5009, Grenoble, France. girin@icp.inpg.fr
authors: L
authors: L
authors: Girin
authors: J L
authors: JL
authors: Schwartz
authors: G
authors: G
authors: Feng
mesh_terms: D000328:Adult; D006801:Humans; D008954:Models, Biological; D009622:Noise / Q000009:adverse effects*; D010700:Phonetics; D013061:Speech Acoustics; D013067:Speech Perception / Q000502:physiology*; D014796:Visual Perception / Q000502:physiology*