392265 Project: SignSpeak: Generating Sign Language Videos from Speech using Deep Generative Models (Pj) (SoSe 2026)

Current AI systems can accurately transcribe speech and generate text-based translations. However, generating sign language videos directly from speech remains a challenging problem, as sign languages are complex visual languages that rely on hand gestures, facial expressions, and body movements.
This project explores how multimodal representations extracted from speech can be used to generate sign-language video sequences with deep generative models.
The goal is to design and implement a prototype pipeline that maps speech features to temporally coherent sign language representations and generates corresponding video output. The system will be trained and evaluated using multimodal datasets containing aligned speech, text, and sign language videos.
Depending on the number of students and the project scope, the project can also include:
• Evaluation of the intelligibility and realism of generated sign language videos
• Analysis of temporal alignment between speech and generated signs
• Comparison of direct speech-based vs. speech-to-text-to-sign pipelines

Requirements for participation, required level

• Good programming skills in Python (ideally PyTorch)
• Basic knowledge of machine learning / deep learning
• Interest in generative AI and multimodal systems
• (Preferably) experience or strong interest in speech processing or video processing (e.g., speech recognition, audio features, or temporal data such as video or pose sequences)
Upon completion of this project, we will work hand in hand to publish the results in a well-established conference or journal in Human–Computer Interaction (HCI) or Computer Vision (CV)