title: Development of a Low-Cost, Noninvasive, Portable Visual Speech Recognition Program.
abstract: OBJECTIVE
  Loss of speech following tracheostomy and laryngectomy severely limits communication to simple gestures and facial expressions that are largely ineffective. To facilitate communication in these patients, we seek to develop a low-cost, noninvasive, portable, and simple visual speech recognition program (VSRP) to convert articulatory facial movements into speech.
  
  
  METHODS
  A Microsoft Kinect-based VSRP was developed to capture spatial coordinates of lip movements and translate them into speech. The articulatory speech movements associated with 12 sentences were used to train an artificial neural network classifier. The accuracy of the classifier was then evaluated on a separate, previously unseen set of articulatory speech movements.
  
  
  RESULTS
  The VSRP was successfully implemented and tested in 5 subjects. It achieved an accuracy rate of 77.2% (65.0%-87.6% for the 5 speakers) on a 12-sentence data set. The mean time to classify an individual sentence was 2.03 milliseconds (1.91-2.16).
  
  
  CONCLUSIONS
  We have demonstrated the feasibility of a low-cost, noninvasive, portable VSRP based on Kinect to accurately predict speech from articulation movements in clinically trivial time. This VSRP could be used as a novel communication device for aphonic patients.
authors.affiliation: Department of Otolaryngology-Head and Neck Surgery, Columbia University College of Physicians and Surgeons, New York, New York, USA.
authors.forename: Gavriel D
authors.initials: GD
authors.lastname: Kohlberg
authors.affiliation: Department of Information Systems Engineering, Ben-Gurion University of the Negev, Beer-Sheva, Israel.
authors.forename: Ya'akov Kobi
authors.initials: YK
authors.lastname: Gal
authors.affiliation: Department of Otolaryngology-Head and Neck Surgery, Columbia University College of Physicians and Surgeons, New York, New York, USA anil.lalwani@columbia.edu.
authors.forename: Anil K
authors.initials: AK
authors.lastname: Lalwani
mesh_terms: D000328:Adult; D001044:Aphonia / Q000628:therapy*; D003143:Communication Devices for People with Disabilities; D005260:Female; D006801:Humans; D007825:Laryngectomy / Q000009:adverse effects; D008046:Lip / Q000502:physiology*; D008297:Male; D009068:Movement / Q000502:physiology*; D049250:Speech Recognition Software; D014139:Tracheostomy / Q000009:adverse effects
keywords: communication aids for disabled; lipreading; silent speech interface; user computer interface; visual speech recognition