Presentation
Search Abstracts | Symposia | Slide Sessions | Poster Sessions | Poster Slams
Repeatedly experiencing the McGurk Effect drives long-lasting changes in auditory-only speech perception
Poster C3 in Poster Session C, Friday, October 7, 10:15 am - 12:00 pm EDT, Millennium Hall
Michael Beauchamp1, Anastasia Lado1, Yue Zhang1, John Magnotti1; 1Department of Neurosurgery, University of Pennsylvania Perelman School of Medicine
Auditory information from the voice of the talker and visual information from the face of the talker provide independent cues about the contents of speech. Cue conflict resulting from audiovisual incongruency can produce unexpected percepts. In the McGurk effect, integration of auditory "ba" and visual "ga" can produce a "da" percept. We report the surprising finding that repeatedly experiencing the McGurk effect results in long-lasting, exemplar-specific changes during auditory-only speech perception. Thirty online participants completed a 16-day experimental protocol. On day 1 ("pre-test") participants reported their perception of 27 auditory-only recordings of "ba", "da" and "ga" syllables from eighteen different talkers, including talkers AM and AN. On days 2 to 15 ("training") participants viewed one minute of McGurk syllables (auditory "ba"+visual "ga") spoken by talkers AM and AN. On day 16 ("post-test") participants reported their perception of the same 27 auditory-only recordings presented during the pre-test. In the pre-test, accuracy was uniformly high, indicating that participants were attentive to the stimuli; "ba" recordings from talkers AM and AN were always perceived as "ba". During training, McGurk stimuli were perceived as "da" on 35% of presentations, with considerable variability across participants (range 0% to 100%). In the post-test, participants accurately reported the identity of the auditory-only syllables with two exceptions. Surprisingly, many participants (12/30; 40%) reported at least one "da" percept in response to the auditory-only "ba" recordings from talkers AM and AN, even though no visual information was presented. For these 12 participants, the rate of McGurk effect during training was four-fold higher than for other participants (66% vs. 14%, p=0.0000042). Our interpretation of these results is that repeatedly experiencing the McGurk effect remapped participants' representation of the auditory component of the McGurk stimulus. The error signal provided by experiencing the McGurk effect led participants to adjust their internal representations so that what was initially perceived as a "ba" became mapped to the "da" region of perceptual space, allowing multisensory integration circuits to minimize the incongruity between the encoded auditory and visual cues. Participants who did not experience the McGurk effect did not show any perceptual remapping during the post-test, demonstrating that simple exposure was not sufficient and the error signal required experiencing the McGurk effect. The phenomenon was specific to individual talkers; the "ba" recordings of talkers AM and AN were perceived as "da", while "ba" recordings from other talkers were unaffected. Importantly, the remapping was unsupervised, since no experimenter feedback was given at any time. The remapped internal representation persisted, so that even when the visual information was absent, auditory "ba" continued to be perceived as "da". The perceptual change endured in two additional testing sessions conducted two weeks and four weeks after the post-test. The long duration and talker-specificity of the phenomenon suggests a possible link to learning the accent of non-native speakers. While initially non-native speakers can be difficult to understand, after repeated interactions, perceivers acquire and retain the ability to understand the accent.
Topic Areas: Perception: Speech Perception and Audiovisual Integration, Development