The role of multimodal information in speech perception

Abstract

The second lecture was devoted to multisensory integration, and in particular the integration of auditory and visual signals, which is both a condition for survival in the environment and a prerequisite for speech perception. The notion of audio-visual integration was introduced, illustrated by three examples :

the ventriloquist effect, in which a speaker holds a puppet whose lips he moves in synchrony with the speech he utters, while keeping his own face motionless. As a result, the audience has the impression that it's the puppet speaking. This illusion involves the fusion of a sound stimulus with an independent but synchronous visual stimulus, just as happens when watching a film ;

the McGurck effect, another classic effect first demonstrated in 1976 by McGurck and MacDonald while studying speech perception in adults. In this experiment, a subject perceives a syllable such as " ba " presented to him repeatedly, while watching a video showing a speaker pronouncing one of the syllables " ba ", " da ", " ga " or " fa ". Depending on the movement of the speaker's lips, the subject perceives a different sound, whereas the sound presented (" ba ") is always exactly the same. This is a striking example of audio-visual fusion, which occurs unconsciously ;

the " double flash " sound-induced visual illusion (or Sifi, sound-induced flash illusion), in which a regular stream of double " beeps " (two short sounds close together in time) are presented to a public at the same time as a series of visual flashes. If, after a series of synchronously presented double flashes and double beeps, the subject is presented with a single flash concomitant with the first beep, the second beep may induce the perception of a second flash of light when none has been presented.

One of the mechanisms underlying multisensory fusion probably involves brain oscillations, which are known to rhythmically modulate neuronal excitability and play a role in the phasing of different brain regions. In this context, we have reviewed a study by Vincenzo Romei's group [6], in which the involvement of alpha occipital waves in the double flash illusion was demonstrated. Using encephalographic recordings, the authors show that when a series of double beeps and light flashes are presented simultaneously, the phase of the occipital alpha oscillations matches the rhythm of the sounds, increasing the excitability of the visual system, so that a light flash can be perceived even in the absence of light for a period of around 100 ms around the instant when the second sound is produced. In this experiment, alpha waves thus appear to act as an internal clock, capable of synchronizing with sound stimulation to increase the excitability of the visual pathways.

Behavioral experiments such as the McGurk effect show that, conversely, bringing the temporal properties of a visual stimulus into coherence with a sound stream modifies the perception of the auditory scene, in a way that enhances the sensitivity of listening to a particular sound of interest, whereas the temporal incoherence of a light stream with a sound disrupts its listening, even relegating it to an inaudible " background noise ". A recent study by Jennifer Bizley's group [7], which we have discussed, provides a neurophysiological basis for these effects, highlighting a multisensory binding mechanism that leads to the formation of distinct audio-visual objects in the auditory cortex. The results of this study show that a contrast enhancement occurs in the cortical representation of a sound when a visual stimulus temporally coherent with the sound is presented together. Importantly, this improvement affects the sound characteristics represented in the auditory cortex, whether or not they are correlated with the luminous flux (in this case, the intensity of the sound is correlated with the luminous flux, but the timbre is not). We can therefore speak of the formation of a distinct audio-visual object in the cortical representation, which " inherits " the attributes of the sound and visual objects from which it is formed. The authors also show that this contrast enhancement once again involves brain oscillations : their results suggest that visual stimulation induces phase synchronization of slow waves in the auditory cortex, making it possible for a temporal correlation between visual and auditory inputs to emerge. However, this synchronization occurs irrespective of whether or not there is temporal coherence between sound and light. The processing center of this correlation and of the formation of an audio-visual linked object therefore remains to be determined, but would probably lie beyond the primary auditory and visual cortices (A1 and V1). This is also suggested by a third study by Tim Rohe and Uta Noppeney [8], whose findings we have briefly summarized here. Based on functional imaging recordings made on subjects during a sound and visual localization experiment, these authors show that audio-visual integration involves the activity of the associative parietal regions of the brain, beyond the primary sensory regions A1 and V1.

The final part of the lecture focused on the current state of knowledge concerning multisensory (audio-visual) integration in children. Psychophysical studies show that, from their first year of life, children are sensitive to the temporal coincidence of rhythmic audiovisual stimuli, a sign of early multisensory integration of audio-visual signals, particularly speech. Nevertheless, this integration remains sub-optimal, and shows a slow maturation until at least the age of 8 , at which point a flexible and efficient use of multisensory information is established. Several studies have shown that between the ages of 9 and 12 children with autistic disorders have a significantly poorer multisensory (audio-visual) perception of speech in noise than non-autistic children of the same age. Autistic children also showed less sensitivity to the McGurk effect, as well as to the double flash illusion. Remarkably, this deficit in audio-visual integration disappears around the age of 13 . These data suggest a delay in the onset of audiovisual integration in autistic children, leading to a deficit in audio-visual perception that diminishes with age.

Finally, we have discussed a study by Scott Murray's group [9], which highlights a defect in cortical adaptation in children with autism, whose cognitive abilities are otherwise normal and whose brain activity is measured by functional imaging. Neuronal adaptation, an essential property of the auditory cortex, is manifested by a progressive decrease in activity in response to repeated exposure to the same stimulus. A failure to adapt can be interpreted as an inability of the cortex to generate predictions from previous stimuli forming an individual's " sensory context ". The data collected by the authors show that a significant adaptation deficit is present in the auditory cortex of children with autism. The severity of this deficit is, moreover, correlated with a behavioral index (ADOS index) measuring the severity of autistic impairment. Notably, a similar adaptation defect was not observed in the visual cortex of these children. This suggests a specific adaptation defect in the auditory cortex, which could play a role in the audio-visual integration deficits discussed above.

In conclusion, we returned to the essential role of audio-visual integration in speech perception, emphasizing its importance in lip-reading. This role is put to good use in particular in the completed spoken language (LPC) practiced by hearing-impaired people known as " oralisantes ", in which the combination of the speaker's lip and hand configurations removes the ambiguities of lip-reading. The LPC is widely used in schools for hearing-impaired children. It has been adapted to over fifty languages. It is also an effective compensatory tool for people suffering from progressive hearing loss (including presbycusis).