Slide Slam N12
Kinesthetic working memory is used to align gestures with speech and visuospatial working memory is recruited to understand gestures
Seana Coulson1,2, Jacob Momsen1,2; 1University of California, San Diego, 2San Diego State University
Prior work suggests both visuospatial and kinesthetic working memory (WM) capacity are related to behavioral measures of sensitivity to co-speech gestures. Here we examine how individual differences in each WM capacity relate to neural indices of real-time speech and gesture processing. Visuospatial WM was assessed in 25 healthy adults via a computerized version of the Corsi block task, while kinesthetic WM was assessed via the movement span task that tracks the ability to accurately reproduce short sequences of meaningless movements of the hands and arms. Following WM assessment, EEG was recorded as participants viewed a series of discourse videos with either congruent or incongruent pairings of speech and gesture. Following each discourse clip, picture probes related to speech were presented to assess how multimodal discourse primed the visual system for subsequent input. A nonparametric cluster-based permutation analysis of event related spectral perturbation (ERSP) data induced by the discourse videos revealed greater alpha band (8-10Hz) suppression during incongruent than congruent speech-gesture pairings (p<0.05). A separate analysis revealed changes in low beta band activity (14-16Hz) in response to speech gesture congruity were related to individual differences in kinesthetic WM ability, an effect focused over left temporo-parietal sites and beginning approximately 100ms after video onset. Individual differences in visuospatial WM ability were not related to gesture congruity effects in ERSP to the videos, but post hoc analysis indicated a relationship between visuospatial WM scores and alpha suppression levels during all discourse videos (p<0.01), suggesting greater engagement with videos in participants with superior visuospatial abilities. Visuospatial WM scores were also associated with gesture congruity effects on the N300 and N400 components of the event-related potentials (ERPs) elicited by picture probes (p<0.025). After the EEG experiment, participants were again shown discourse videos and asked to make explicit judgments about the relationship between the speech and the gesture. Performance on this offline gesture congruity task (d’) was positively associated with kinesthetic WM scores, replicating previous work and providing evidence that kinesthetic WM helps promote sensitivity to speech-gesture congruity (b=0.40, t=2.94, p<0.01). Offline sensitivity to the relationship between speech and gestures may be related to the beta-suppression effect observed during real time viewing of multimodal discourse. The timing of that effect (beginning ~100ms after video onset) and the hypothesized role for beta band activity in the synchronization of neural events suggests kinesthetic WM helps to coordinate the perception of prosodic fluctuations in speech with movements of the speaker’s body. The relationship between visuospatial WM and gesture congruency effects on picture probes suggests visuospatial resources related to the generation and maintenance of imagery mediate sensitivity to the meaning of co-speech gestures.