392254 Project: Referential Understanding for Multi-Agent Interaction in the Cooperative Cuisine Environment (Pj) (WiSe 2025/2026)

The Cooperative Cuisine research environment provides a virtual kitchen scenario for studying collaboration between humans and autonomous agents. While speech-based interaction with a single agent is already possible, the system currently lacks mechanisms for distinguishing between multiple agents in shared scenarios.
This project aims to extend the system’s communicative and perceptual capabilities by introducing referential understanding: enabling agents to interpret and respond to spoken references based on spatial relations (e.g. “the agent near the cutting board”), unique identifiers (e.g.
names), and visual or semantic attributes (e.g. color, tool, role).
Additionally, references to previous actions (“Agent A, another one”) should be supported to facilitate more natural, contextual communication.
To achieve this, you will design and implement a modular LLM-based reference resolution system, integrate it into the existing agent, and extend the visualization layer with GUI overlays showing agent identifiers and relevant contextual information.

Key Tasks:
- Implement reference resolution mechanisms (spatial, contextual, name-/feature-based).
- Design and integrate a modular LLM-based architecture for interpreting user commands.
- Evaluate system performance through interactive kitchen scenarios as well as a systematic benchmark with test cases

Expected Outcome:
A prototype enabling task delegation through natural language in a
multi-agent human–AI setting in the virtual kitchen, advancing research
in embodied communication and referential language understanding.