Slide Slam M6
Cortical tracking and the relationship between structure and meaning
Cas Coopmans1,2, Helen de Hoop2, Peter Hagoort1,3, Andrea Martin1,3; 1Max Planck Institute for Psycholinguistics, 2Centre for Language Studies, Radboud University, 3Donders Institute for Brain, Cognition and Behavior, Radboud University
It is well-established that brain activity ‘tracks’ low-level aspects of speech stimuli, such as the amplitude envelope (e.g., Peelle & Davis, 2012). Recent studies show that cortical tracking also occurs at the presentation rate of abstract information, such as syntactic phrases (Ding et al., 2016; Keitel et al., 2018), and that such phrase structure tracking is modulated by the content of these phrases. In a recent EEG study, Kaufeld et al. (2020) found that phrase structure tracking is stronger for compositional sentences than for control stimuli that had lexical content but no structure (i.e., word lists) and prosodic and syntactic structure but no lexical content (i.e., jabberwocky), suggesting that this neural response is driven by compositional structure and meaning (Martin, 2020). Following up on this idea, the current EEG study examines to what extent cortical tracking of linguistic structure is modulated by the compositionality of that structure. We measured EEG of 38 participants who listened to naturally produced stimuli in five different conditions, which systematically modulated the amount of linguistic information. We compared sentences (+syntax, +lexical meaning, +composition) to idioms (+syntax, +lexical meaning, ~composition), syntactic prose (+syntax, +lexical meaning, ~composition), prosodic jabberwocky (+syntax, –lexical meaning), and word lists (–syntax, +lexical meaning), and included backward versions of sentences and word lists as acoustic controls. Based on manual annotations of all speech recordings, we derived frequency bands corresponding to the presentation rate of phrases (1.1-2.1 Hz), words (2.3-4.7 Hz) and syllables (3.4-4.9 Hz). Tracking was quantified through Mutual Information (MI) between the EEG data and the envelope of the speech stimuli in each of these frequency bands. The higher the MI value, the stronger the statistical relationship between the two input signals. We consistently found that, in the phrase frequency band, MI between speech and EEG was higher for sentences than for prosodic jabberwocky, but not higher than for idioms or syntactic prose. This result was also found when MI was computed between the EEG signal and abstract annotations that represented either the presence of closing phrase boundaries or the number of closing phrase boundaries (‘bracket count’). Phrase structure tracking was also higher for sentences than for word lists, but this difference was also found for the backward versions of these stimuli and could therefore reflect a difference in their acoustics. Overall, phrase structure tracking was stronger for sentences than for stimuli that lacked either lexical meaning or syntactic structure, but it was not consistently different from stimuli which had lexical meaning and syntactic structure. These findings suggest that cortical tracking of linguistic structure reflects the generation of structure (Martin, 2020; Meyer et al., 2019), whether this structure straightforwardly maps onto semantic meaning or not. This is in line with models of language comprehension which make a functional distinction between syntactic structure building and semantic composition (Baggio, 2018; Hagoort, 2005). As our findings map most strongly onto the process of structure building, an important question for future research is how the brain computes the compositional meaning of this structure.