Slide Slam R6
An increased influence of biological motion on speech perception in those with superior kinesthetic working memory
Jacob Momsen1, Seana Coulson2; 1UC San Diego/San Diego State University, 2UC San Diego
Because spontaneously produced gestures often unfold in tandem with prosodic fluctuations in speech, gestures can influence discourse processing by inducing expectations about the precise timing of upcoming speech. Biological motion that correlates with sonic energy peaks can help optimize how the brain samples auditory information by modulating phase relationships between ongoing neural oscillations and the speech signal. Cross-modal coordination of this sort would be indexed by an increase in the phase alignment of low frequency activity (~3-7Hz) to speech that is preceded by the onset of a gesture. Here we investigate the role of gestures in tuning expectations about speech onset and whether kinesthetic working memory (KWM), that is, memory for body movements, supports these processes. KWM was assessed in 25 healthy adults using a movement span task that quantified their ability to memorize and reproduce a series of meaningless body postures. In a subsequent study, EEG was recorded as those participants viewed multimodal discourse clips with either congruent or incongruent speech and gestures. Cluster-based permutation tests revealed a main effect of speech-gesture congruency on ERPs to content words in the speech (p<0.001). Words in incongruent discourse clips elicited more negative ERPs than those accompanied by congruent gestures beginning approximately 350ms after onset. This N400 effect confirms the facilitatory role of gestures in semantic retrieval. To explore the role of KWM in phase resetting of theta band activity to speech, we measured the intertrial phase coherence of EEG elicited during congruent versus incongruent discourse clips. Analysis across all participants indicated phase alignment in low frequency activity was more pronounced in congruent relative to incongruent speech-gesture pairings. Cluster-based permutation tests on the interaction of this congruency effect and KWM revealed a relationship between KWM scores and more positive intertrial phase coherence in the theta band (3-5Hz) from 80-150ms after speech onset in congruent relative to incongruent videos (p<0.05). Results suggest participants with greater KWM capacity were more sensitive to how visual biological motion cues in gesture influenced theta resetting to the speech signal. Overall, these data support a role for KWM in multimodal discourse processing by promoting sensitivity to dynamic features of biological motion in a way that affords efficient auditory processing of the speech signal.