Presentation
Search Abstracts | Symposia | Slide Sessions | Poster Sessions | Lightning Talks
The influence of speaker gaze on addressees’ response planning: evidence from behavioral and EEG data
Poster B18 in Poster Session B, Tuesday, October 24, 3:30 - 5:15 pm CEST, Espace Vieux-Port
This poster is part of the Sandbox Series.
Alexandra Emmendorfer1,2, Judith Holler1,2; 1Donders Institute for Brain, Cognition, and Behaviour, 2Max Planck Institute for Psycholinguistics
Classical models of speech processing are based primarily on unimodal accounts of language, and neglect the rich multimodal environment in which face-to-face communication takes place (Holler & Levinson, 2019), where the speech signal is accompanied by a broad range of visual communicative signals. A growing body of literature suggests that these visual signals may facilitate efficient turn-taking behavior at various stages in speech processing, such as facilitating the recognition of social actions (Nota et al., 2022), allowing the prediction of upcoming words (ter Bekke et al., 2020), or increasing inter-brain synchrony between interlocuters (Drijvers & Holler, 2022). One prominent visual communicative signal is speaker gaze, however its role in response planning is not well understood. Corpus analyses reveal that questions with static direct gaze receive faster responses compared to questions where the speaker averts their gaze from the addressee, suggesting that speaker gaze may function as a response mobilization cue (Stivers & Rossano, 2010). This mobilizing function may take effect at various cognitive stages of turn-taking behavior. We shed light on this in two experiments (online behavioral, EEG) investigating the effect of speaker gaze on response time and EEG correlates of response planning and motor preparation. In both experiments, participants are presented with animated videos of an avatar. Participants are instructed to respond to 240 polar questions as fast and as accurately as possible via button-press (yes/no). The questions are presented in one of three gaze conditions: (1) static direct, where the avatar’s gaze remains fixed toward the participant throughout the video, (2) dynamic direct, where the avatar’s gaze starts at an ambiguous position 15 degrees averted, and shifts toward the participant at the beginning of the question, (3) dynamic averted, where the avatar’s gaze starts at the same ambiguous position, but shifts further away from the participant at the beginning of the question. To examine the effect of speaker gaze on response planning, the questions were designed to have either an early or late answer point. Contrary to prior corpus analyses, the behavioral experiment (n=56) did not reveal a significant main effect of speaker gaze on response time. However, an interaction between speaker gaze and answer point approached significance, where questions with static direct gaze showed a smaller effect of answer point on response time. This pattern may indicate that speaker gaze influences turn-taking behavior early on in the response planning stage. In the ongoing EEG experiment (currently n = 2, target n = 30), we plan to analyze the planning potential (e.g., Bögels et al., 2015) as a marker of early response planning, as well as the readiness potential as a marker of motor preparation, to examine which stage of response preparation is more strongly influenced by speaker gaze. In light of the current behavioral findings, we expect an interaction effect of speaker gaze and answer point on the planning potential, where questions with early answer points show a larger planning potential in the static gaze condition compared to both dynamic gaze conditions.
Topic Areas: Meaning: Discourse and Pragmatics, Multisensory or Sensorimotor Integration