392247 ISY Project: Robomantic2021 - Object recognition in haptic (augmented) point clouds using a tactile sensitive data-glove and ML methods. (Pj) (SoSe 2021)

Object recognition in haptic (augmented) point clouds using a tactile sensitive data-glove and ML methods. Within the ISY-Proj. Robomantic2021 we investigate in the nature of things under their (polygon) surface. Our goal is to enrich the virtual representations of captured real-world scenes with haptic information. E.g. how does a box feel that the Kinect has captured? Is it heavy, rigid, filled with sand etc.? The current capturing methods do not provide such information. But with a tactile sensitive data-glove (µGlove) we can record them!
In this project, we:
- Record rich object point-cloud data within the ROS environment (at your home, first with your marker tracked hand, later in the lab with) a state of the art multimodal data-glove (touch, posture, inertial measurements).
- Lean about the functioning of our implemented forward kinematic model of the tracked hand which compute the contact point cloud.
- Improve the accuracy of the point cloud generation via ground truth and cross-modality calibration technics.
- Employ state of the art methods to filter the multidimensional time-series data and to learn and recognize features of soft and hard objects in the object point-cloud.
- Derive essential details from the dataset, such as form, weight, mass distribution, ductility, etc.
- Measure our results: e.g by finding objects that trick out a pure visually generated point cloud (e.g. from Kinect).
-> as surplus:
- Fuse visual keys and tactile features to one multimodal object model.
- Study data-driven approaches to model and learn human manipulation skill. e.g. understand how humans use their hands and haptic perception to manipulate, classify and reconstruct objects.

FEEL WHAT MATTER!S
In the ISY project Robomantic2021, your team will provide weight to a virtual cloud of points.
The Kinect revolutionized robotics with its highly informative point cloud. First trigonometrically, then via the time of flight of light particles. Now it provides a relatively good optical scan of the scenery, spatially and temporally. This information helps us (our robots) to navigate, recognize and detect objects. The recognition of objects is done by matching them with an internal representation (already stored objects). As manifold the environment is, as diverse are the properties of its objects. Optical properties are only the visible tip of the iceberg. Beneath the outer shell of the objects we find their inner nature - their softness/elasticity/mass distribution etc. given via their inner structure. The world is not only made of polygons! The aim of our ISY project Robomantic1 is to capture the inner values of the objects in our environment.
The hardware we use for this is the µGlove. A data glove that records spatial and temporal information about its position (finger position) and contact forces to touched surfaces.
The system for generating point clouds using the µGlove was developed as part of Julia Niermann's master's thesis. Except for the µGlove, it also works without expensive laboratory equipment, so that the experiments can be planned well from home. (e.g. recording dry (weightless) clouds using marker cubes for bar hand tracking).
We test different architectures to classify and model the object point cloud. The special challenge for advanced progress is to model soft surfaces as well.
The results of the ISY project Robomantic1 can be used in the field of extended (haptic) VR. Primarily, however, in robotics we benefit from the new possibilities to fill real objects in mental representations with meaningful content. This improves the handling of objects in terms of e.g. better / more dexterous handling by a robotic hand or a robotic hand-prosthesis.
You will learn about the elementary importance of tactile sensitivity. We will support you in setting up the ROS environment and the additional features. And we will accompany you in developing ideas for the practical realisation of the project.