title: Real-time 3D human capture system for mixed-reality art and entertainment.
abstract: A real-time system for capturing humans in 3D and placing them into a mixed reality environment is presented in this paper. The subject is captured by nine cameras surrounding her. Looking through a head-mounted-display with a camera in front pointing at a marker, the user can see the 3D image of this subject overlaid onto a mixed reality scene. The 3D images of the subject viewed from this viewpoint are constructed using a robust and fast shape-from-silhouette algorithm. The paper also presents several techniques to produce good quality and speed up the whole system. The frame rate of our system is around 25 fps using only standard Intel processor-based personal computers. Besides a remote live 3D conferencing and collaborating system, we also describe an application of the system in art and entertainment, named Magic Land, which is a mixed reality environment where captured avatars of human and 3D computer generated virtual animations can form an interactive story and play with each other. This system demonstrates many technologies in human computer interaction: mixed reality, tangible interaction, and 3D communication. The result of the user study not only emphasizes the benefits, but also addresses some issues of these technologies.
authors.affiliation: Interactive and Entertainment Research Center, Level 02-02, Research Techno Plaza, Nanyang Technological University, 50 Nanyang Drive, Singapore 637553. duynguyen@mixedrealitylab.org
authors.forename: Ta Huynh Duy
authors.initials: TH
authors.lastname: Nguyen
authors.forename: Tran Cong Thien
authors.initials: TC
authors.lastname: Qui
authors.forename: Ke
authors.initials: K
authors.lastname: Xu
authors.forename: Adrian David
authors.initials: AD
authors.lastname: Cheok
authors.forename: Sze Lee
authors.initials: SL
authors.lastname: Teo
authors.forename: ZhiYing
authors.initials: Z
authors.lastname: Zhou
authors.forename: Asitha
authors.initials: A
authors.lastname: Mallawaarachchi
authors.forename: Shang Ping
authors.initials: SP
authors.lastname: Lee
authors.forename: Wei
authors.initials: W
authors.lastname: Liu
authors.forename: Hui Siang
authors.initials: HS
authors.lastname: Teo
authors.forename: Le Nam
authors.initials: le N
authors.lastname: Thang
authors.forename: Yu
authors.initials: Y
authors.lastname: Li
authors.forename: Hirokazu
authors.initials: H
authors.lastname: Kato
mesh_terms: D001154:Art; D003198:Computer Simulation; D003199:Computer Systems; D003626:Data Display; D004777:Environment; D004867:Equipment Design; D019544:Equipment Failure Analysis; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000295:instrumentation* / Q000379:methods*; D021621:Imaging, Three-Dimensional / Q000379:methods*; D007899:Leisure Activities; D008328:Man-Machine Systems; D008954:Models, Biological; D019212:Multimedia; D009862:Online Systems; D012815:Signal Processing, Computer-Assisted / Q000295:instrumentation; D014584:User-Computer Interface