392192 Geometry Foundation Models meet Robotics (S) (WiSe 2026/2027)

Feed-forward geometric foundation models such as DUSt3R and VGGT have made dense, calibration-free 3D reconstruction from images practical — and increasingly central to robotics: precise scene geometry, real2sim2real transfer, dynamics understanding for planning, and 6D object pose estimation for manipulation.

This seminar closes a gap between existing offerings at the Technische Fakultät, the classical Computer Vision course (Prof. Dr. Wachsmuth) and the deep-learning-focused courses of the Visual AI for Extended Reality group (Prof. Dr. Rhodin) - by tracing how recent geometric foundation models connect to robotic perception, simulation, and manipulation.

We cover four themes, moving from the models themselves to their robotics applications:

• 3D reconstruction backbones — feed-forward multi-view geometry models and how they represent cameras and scenes.
• Digital twins (real2sim2real) for robotics — reconstructing simulation-ready digital twins of real scenes and objects to close the sim-to-real gap.
• Manipulation from digital twins — how these reconstructions enable calibration, 6D pose estimation, and grasping.
• Dynamics from vision for planning — recovering scene and object dynamics from visual observation for robot planning.

Requirements for participation, required level

Required:

• Foundations in machine learning (an introductory ML/DL course)
• Working knowledge of Python
• Basic linear algebra and 3D geometry (rotations, projective transformations) - briefly reviewed in the first sessions

Not required: prior experience with robotics or with 3D computer vision. The course is designed to be equally accessible to computer-vision-oriented and robotics-oriented students.

Bibliography

A short set of orientation references - the full, session-by-session reading list follows separately at the start of the semester.

• Hartley & Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004.
• VGGT: Visual Geometry Grounded Transformer, Wang et al., CVPR 2025.
• RialTo: Real-to-Sim-to-Real, Torne et al., RSS 2024.