392235 Project: Multimodal Behavior Processing - Projects (see description) (Pj) (WiSe 2024/2025)

1)
Title: I can see your heartbeat – Implementing rPPG-Methods to Assess Heart Rates in a Web Application
Description: Remote photoplethysmography (rPPG) is the contactless monitoring of the blood volume
pulse, and consequently the heart rate, using a regular camera. Classical rPPG methods make use of
signal processing techniques, while recent methods utilize large deep learning models. We developed the
Simulated Interaction Task (SIT), a digital application designed to measure qualitative and quantitative
differences in social behaviour. The SIT is currently in use in multiple clinical studies, for example to detect
Autism Spectrum Disorder (ASD) in adults or assess social anxiety in children. Participants perform the
test on a laptop while their faces are recorded via a web cam. You will implement rPPG Methods in the
existing SIT web application, providing users with immediate feedback of their heart rate during the test.
You will integrate a pretrained rPPG model within the existing code and infrastructure (JavaScript, JATOS
backend).

2)
Title: You look stressed so I look stressed? Analysis of Facial Mimicry during a social stress test (TSST)
Description: In interpersonal exchanges, individuals often spontaneously mimic the facial expressions of
others, a phenomenon commonly termed facial mimicry. This project aims to analyze facial mimicry and
empathic stress during the Trier Social Stress Test (TSST). The TSST is considered the gold standard in
human experimental stress research, where participants complete a 5-minute mock job interview and a 5-
minute mental arithmetic task in front of an evaluating committee. Your task is to analyze facial mimicry in
video recordings of participant and committee members during the TSST. The goal is to develop digital
facial mimicry indices that can be further used for analyzing social interactions and prediction stress levels
based on observers' facial expressions & mimicry.

3)
Title: Predicting Breathing Rates from Smartphone Front-Camera Videos of the Digital Stress Test (DST)
Description: In an on-going laboratory study we’re currently using our recently developed Digital Stress
Test (DST) for the induction of psychosocial stress. Participants perform the test with a smartphone while
their faces are recorded via the front-camera. At the same time, we continuously monitor various
physiological signals (i.e., breathing rate, heart rate, blood volume pulse). With this machine learning
project, you will analyze how well breathing rates can be detected based on the front camera videos. You
will explore several pre-existing algorithms for video-based breathing detection, implement and evaluate
them on our dataset. Further, you will implement your own algorithms for predicting breathing rates
based on the experiences with existing algorithms and try to improve their performance.