Seeing a talker’s face helps us follow speech in a noisy room. Computer-generated facial avatars can increasingly be used to supply that visual support. We wondered how much such an avatar really helps, and how the brain combines its visual infomation with the audio. Volunteers listened to German sentences embedded in the chatter of four background talkers while we recorded their brain activity with magnetoencephalography, a technique that picks up the faint magnetic fields produced by active nerve cells. The same sentences were presented in four ways: with a still image, so effectively sound alone; with an avatar generated by a deep neural network; with a heavily degraded video showing little more than the outlines of the face; and with natural video of the talker.
Speech comprehension rose from 66 percent with sound alone to 78 percent with the avatar and 76 percent with the degraded video, and reached 84 percent with natural video. We then examined one of the earliest and fastest brain responses to a voice, which follows the speaker’s pitch. This so-called frequency following response (FFR) was unchanged across all four conditions and did not account for the differences in comprehension. Published in the European Journal of Neuroscience, this is the first measurement of this early cortical response for continuous audiovisual speech and for artificial avatars, and it shows that the benefit an avatar brings does not depend on shaping how the brain encodes vocal pitch at this early stage.
J. Riegel, A. Schüller, C. Jehn, A. Wißmann, S. Zeiler, D. Kolossa, T. Reichenbach,
The cortical contribution to the speech-FFR is not modulated by visual information,
Eur. J. Neurosci. 64:e70680 (2026) [bioRxiv] [pdf]
The complete data is available on zenodo.org.