Our new open-access paper is out in NeuroImage.
Seeing a speaker’s face makes it much easier to follow a conversation in a noisy room. AI-generated talking avatars could offer the same help in, e.g., train-station announcements, in particular for people with hearing loss. But how does the brain respond to an artificial face?
We recorded brain activity with MEG in 32 volunteers while they listened to sentences in background babble. Each sentence came with a still image, a natural video, a simplified “edge-only” video, or an AI avatar.
What we found:
✅ Avatars help. Word comprehension rose from 66% to 78% with the avatar. Natural video was still best at 84%.
⏱️ A late brain signal tracks understanding. Brain activity following speech rhythms (1–4 Hz) about 600 ms after the sound predicted how well people understood.
👄 Very slow brain rhythms (below 1 Hz) relate to lip reading, and intriguingly the avatar disrupted them. The altered neural response to the avatar may come from its unfaithful mouth, jaw or head movements at these low frequencies, and offers opportunities for future improvement.
J. Riegel, A. Schüller, C. Jehn, A. Wißmann, S. Zeiler, D. Kolossa, T. Reichenbach,
Relationships between speech-in-noise comprehension, lip reading ability and cortical speech tracking in talking avatars as well as natural talking heads,
NeuroImage 342:122282 (2026) [bioRxiv][pdf]
The complete MEG data is available on zenodo.org.