Poster C76, Wednesday, August 21, 2019, 10:45 am – 12:30 pm, Restaurant Hall
Musical background affects audiovisual modulation of speech and music at N1 - an ERP study
Marzieh Sorati1, Dawn M. Behne;1Norwegian University of Science and Technology (NTNU)
Previous research on audiovisual speech perception has shown that mouth movements predicting the upcoming sound give an anticipatory effect which can behaviorally and electrophysiologically modulate speech perception (van Wassenhove et al., 2005; Paris et al., 2013). Similarly, in audiovisual music perception, hand movements provide predictable regularities for the note being played in a musical event. Practicing a musical instrument is a rich multimodal experience, and while extensive musical training is known to enhance audio perception (Zatorre et al., 2007), an unattended question is whether this enhancement is fixed to unimodal auditory perception for music or transfers to audiovisual perception, and speech, by superior prediction of what sound is coming and when. The current study compares musicians and non-musicians’ audiovisual modulation in music and speech based on the visual cues (hand and mouth movements) predicting the upcoming sound and providing an anticipatory effect which can modulate perception. Event-related potentials (ERPs) were recorded with seven musicians and seven non-musicians while presented music (keyboard note, C4) and speech (/ba/). The music and speech stimuli were presented in three conditions: audio-only (AO), video-only (VO) and audiovisual (AV). In the AO condition, analysis of N1 amplitudes and latencies showed that for music stimuli musicians have a higher N1 amplitude than non-musicians while for speech stimuli no group difference was observed. These results are consistent with previous ERP research showing that musicians have improved auditory perception in music (Shahin et al., 2005). Next, to isolate the anticipatory effect of hand movements in music stimuli and mouth movements in speech, for each stimulus ERP waveforms for the VO conditions were subtracted from the AV conditions (AV-VO) and compared with the corresponding AO ERPs. For music stimuli, a two-way analysis of variance with musical background (musicians vs. non-musicians) and condition (AO vs. AV-VO) showed a significant interaction for N1 amplitude. The difference between AO and AV-VO was hence further analyzed for each group, and results showed that while musicians` N1 amplitude was significantly lower for AV-VO compared to the AO condition, no difference was observed for non-musicians. For the speech stimuli, despite no significant interaction, the same main effect for condition was observed for N1 amplitude, with the musicians’ N1 amplitude lower in AV-VO compared to AO, and no N1 suppression for non-musicians. Notably, N1 latency was shorter for AV-VO than AO for both groups, and this pattern was the same for music and speech stimuli. These findings show that N1 amplitude is affected by musical experience both for music and speech perception, while, N1 latency is lower for AV-VO compared to AO condition independently of expertise. In other words, while both groups showed audiovisual modulation of N1 latency with anticipatory visual cues, musicians showed a greater audiovisual modulation of N1 amplitude than non-musicians, suggestion greater prediction from the visual information for what sound is coming, and when. These findings also imply that enhancement in audio perception for musicians is transferable to audiovisual perception both for music and for speech.
Themes: Perception: Speech Perception and Audiovisual Integration, Speech Perception
Method: Electrophysiology (MEG/EEG/ECOG)