Joint optimization of word alignment and epenthesis generation for Chinese to Taiwanese sign synthesis.
This work proposes a novel approach to translate Chinese to Taiwanese sign language and to synthesize sign videos. An aligned bilingual corpus of Chinese and Taiwanese Sign Language (TSL) with linguistic and signing information is also presented for sign language translation. A two-pass alignment in syntax level and phrase level is developed to obtain the optimal alignment between Chinese sentences and Taiwanese sign sequences. For sign video synthesis, a scoring function is presented to develop motion transition-balanced sign videos with rich combinations of intersign transitions. Finally, the maximum a posteriori (MAP) algorithm is employed for sign video synthesis based on joint optimization of two-pass word alignment and intersign epenthesis generation. Several experiments are conducted in an educational environment to evaluate the performance on the comprehension of sign expression. The proposed approach outperforms the IBM Model 2 in sign language translation. Moreover, deaf students perceived sign videos generated by the proposed method to be satisfactory.
Chiu
Yu-Hsien
YH
Home Network Technology Center, Industrial Technology Research Institute, N200, ITRI Bldg. R1, No. 31, Gongye 2nd Rd., Annan District, Tainan City 709, Taiwan, ROC. chiuyh@itri.org.tw
Wu
Chung-Hsien
CH
Su
Hung-Yu
HY
Cheng
Chih-Jen
CJ
D000465:Algorithms; D001185:Artificial Intelligence; D006225:Hand / Q000033:anatomy & histology*; D006801:Humans; D007089:Image Enhancement / Q000379:methods; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D021621:Imaging, Three-Dimensional / Q000379:methods; D016247:Information Storage and Retrieval / Q000379:methods*; D019303:Multilingualism; D010363:Pattern Recognition, Automated / Q000379:methods*; D010781:Photography / Q000379:methods; D015203:Reproducibility of Results; D012660:Semantics; D012680:Sensitivity and Specificity; D012813:Sign Language; D013382:Subtraction Technique; D014741:Video Recording / Q000379:methods*