Speech Source Separation for Multi-Speaker EEG Hyperscanning Recordings
Background
We recorded 13 EEG hyperscanning experiments in which groups of three people who know each other held free conversations. Each experiment contains 12 trials spanning three conversation types (free speech, consensus finding, and conflict). While they talked we recorded 96-channel EEG at 2.5 kHz and audio at 44.1 kHz.
The audio setup is the problem this thesis addresses. Each participant has a directional microphone placed 2 m away, with 120 degrees between neighbouring microphone axes, and a single omnidirectional microphone sits at the centre of the triangle. Directionality is not enough at this distance: every channel contains substantial speech bleed from the other two participants.
This leaks into the neuroscience. Speaker diarization currently relies on energy gating of the directional channels, so bleed produces false speaking intervals, which in turn corrupts the listening segments on which all our neural metrics are computed. Worse, the temporal response function (TRF), which measures how strongly a listener’s brain tracks speech, is currently estimated from the omnidirectional envelope. That envelope is a mixture of all three talkers, so we cannot presently ask the question we actually care about: does a listener’s cortex track the person they are listening to?
A clean per-speaker audio stream would fix all three of these at once.
Objective
Implement, validate, and deploy a source separation algorithm that recovers three individual speech streams from our four-channel recordings.
Requirements
- Solid Python; PyTorch experience is a clear advantage.
- Background in digital signal processing (filtering, spectral analysis, and ideally array processing).
- Interest in working with real, imperfect experimental data.
- German language skills are helpful for judging separation quality by ear but are not required.
Resources
Access to a fully recorded multimodal dataset, a working analysis codebase, GPU compute, and a result that immediately unblocks an ongoing PhD project. Positive outcomes will be considered for publication.
Gaze Behaviour and Conversational Dynamics in Triadic Interaction
Background
We recorded 13 EEG hyperscanning experiments in which groups of three acquainted people held 12 conversational trials each, across free speech, consensus finding, and conflict conditions. The sessions were filmed, and the video is already photodiode-synchronised to the 96-channel EEG. Audio from three directional microphones and one omnidirectional microphone gives us diarized speaking intervals, so we know at every moment who was talking.
We would like to consider the sight. This matters more in a triad than in a dyad. With two people, gaze direction carries limited information: you are either looking at the other person or you are not. With three, gaze becomes a genuine addressing mechanism. A speaker selects the next speaker by looking at them. A listener signals alignment with one participant over another. Someone can be excluded from an exchange without a word being said. The classic findings on gaze in conversation, that speakers avert gaze while planning and return it near turn ends to hand over the floor, were largely established in dyads, and the triadic case is where they should become both more complex and more consequential.
Our seating geometry helps: participants sat 120 degrees apart at known, fixed positions, which turns gaze target identification from an open-ended regression problem into a well-posed three-way classification.
Objective
Extract gaze and eye-related measures from the video recordings and relate them to the temporal dynamics of the conversation, to the neural metrics computed in our pipeline, and to the behavioural ratings.
Requirements
- Solid Python; computer vision experience is an advantage.
- Interest in human behaviour and social interaction, not only in the signal processing.
- Willingness to do manual annotation work. It is unglamorous and it is what makes the rest of the thesis defensible.
Resources
A recorded and synchronised multimodal dataset, an established EEG analysis pipeline, GPU compute.
Facial Expression States and Their Neural Correlates in Triadic Conversation
Background
We recorded 13 EEG hyperscanning experiments in which groups of three acquainted people held 12 conversational trials each, spanning free speech, consensus finding, and conflict. Alongside 96-channel EEG at 2.5 kHz and multi-microphone audio, every session was filmed. The video is stored as .mkv and is already synchronised to the EEG through a photodiode signal, so frame-accurate alignment to the neural data is a solved problem you can build on rather than a risk you have to absorb.
So far the video has been used for exactly one thing: extracting lip motion to help verify who was speaking. This is a considerable waste. The face is the richest continuously available signal of a person’s affective state during conversation, and we have three faces recorded simultaneously alongside their three brains.
Our published work from this dataset shows that occipital alpha power and delta-band inter-brain synchrony both predict how engaged participants report feeling. What we do not know is whether these neural markers have a visible affective signature, and whether the emotional dynamics of a conversation explain variance that the self-report ratings miss.
Objective
Extract facial features from the video recordings, classify them into affective states, and relate those states to the EEG metrics already computed in our pipeline and to the behavioural ratings.
Requirements
- Solid Python; experience with computer vision or deep learning frameworks is an advantage.
- Comfort with statistics beyond a t-test, or willingness to learn mixed models.
Resources
A recorded multimodal dataset, an established EEG analysis pipeline with metrics already computed, GPU compute.
Bachelor and Master thesis in Sensory Neurotechnology
Please contact Tobias Reichenbach by email if you are interested in a Bachelor or Master thesis in the group. Please include a CV and a transcript of your grades at FAU.