Tencent Launches AI Sign Language Digital Interpreter Lingyu
Shenzhen-based internet giant Tencent has recently launched a 3D sign language digital interpreter called “Lingyu” (聆语Língyǔ) on China’s state broadcasting network CCTV, to provide translation services for the ongoing Beijing Winter Olympics. In addition, “Xiaocong” (小聪 Xiǎocōng), another 3D sign language digital character developed by Tencent to help those with impaired hearing, will be launched on Tencent Sports.
Tencent said that Lingyu was jointly created by Tencent Cloud and the AI team for the company’s Platform and Content Business Group (PCG), as well as other technical teams. Up to now, the vocabulary and sentence coverage of Lingyu has exceeded 1.6 million, it has been optimized for sporting events, and its intelligibility of sign language has surpassed 90%.
Underpinning Lingyu and Xiaocong are integrated technologies that span 3D digital human modeling, voice recognition, semantic understanding, machine translation and image rendering.
Tencent has also developed a sign language translation system. The system can generate sign language with high accuracy through machine translation with a low delay. For example, when a user inputs a sentence “he is my sign language teacher,” the system will preprocess the sentence as “he, is, my, sign language, teacher” and will deliver a translated result as “he, me, sign language, teacher, is.”
Next, Tencent‘s system will allow the sign language digital character to express the message accurately. Based on its multimodal end-to-end generation model, joint modeling and prediction will be carried out to generate high-accuracy sequences of actions, expressions and lip movements, so as to achieve natural, professional and easy-to-understand sign language effects.
Moreover, Tencent has developed a visualized action editing platform, which provides a friendly tool for professional sign language teachers, enabling them to refine all sign language actions efficiently.
This interpreting and commentary service launched by Tencent was used to describe the first gold medal of the Chinese team at the current Beijing Winter Olympics, namely the short-track speedskating mixed-gender team relay. The technology helps people with impaired hearing feel the grand occasion of the Games, further enhancing their experience.
