392266 Project: Speech2Symbols: Generating AAC Symbol Sequences from Speech using Generative AI (Pj) (SoSe 2026)

Current AI systems can accurately transcribe speech and generate text-based outputs. However, automatically translating speech into symbol-based representations for augmentative and alternative communication (AAC) remains a challenging problem, as it requires mapping spoken language to semantically meaningful and contextually appropriate visual symbols.
This project explores how speech input can be transformed into sequences of AAC symbols using generative AI models.
The goal is to design and implement a prototype pipeline that converts speech into semantic representations and generates corresponding symbol-based visual sequences. The system will be based on open AAC symbol sets (e.g., Mulberry Symbols, Global Symbols) and will be trained and evaluated using suitable multimodal or text-based datasets.

Depending on the number of students and the project scope, the project can also include:
• Evaluation of the semantic accuracy and usability of generated symbol sequences
• Analysis of temporal alignment between speech and symbol output
• Comparison of rule-based, speech-to-text-based, and generative approaches
• Exploration of real-time or interactive prototype implementations

Teilnahmevoraussetzungen, notwendige Vorkenntnisse

• Good programming skills in Python (ideally PyTorch)
• Basic knowledge of machine learning / deep learning
• Interest in generative AI and accessibility applications
• (Preferably) experience or strong interest in speech processing, NLP, or sequence modeling
Upon completion of this project, we will work hand in hand to publish the results in a well-established conference or journal in Human–Computer Interaction (HCI) or Computer Vision (CV)