AI Sign Language Generation in Concert (Ongoing)

• Project Goal: Develop an AI-driven system to generate real-time American Sign Language (ASL) during concerts, providing accessibility to deaf and hard-of-hearing audiences.
• Utilized Whisper API for accurate real-time speech-to-text conversion. Integrated ChatGPT API to transform text into ASL gloss. Constructed an ASL lexicon by leveraging the WLASL dataset. Converted ASL gloss into ASL poses.
• Developed a two-channel server socket and built an interface with Unreal Engine to enable real-time ASL performance by a 3D avatar based on live audio input from the client.
• Developed the workflow that enables real-time conversion from speech to text, text to gloss, gloss to ASL poses, and poses to 3D Avatar, improving the accessibility and inclusivity of live performances.
• The code of this project and more technical details can be found at this https URL.