Slide Slam S15 Sandbox Series
Cortical signatures of the interaction between prosody and syntax during naturalistic language processing
Giulio Degano1, Alessandra Rampinini1, Peter Donhauser5, Paola Merlo6, Narly Golestani1,2,3,4; 1Department of Psychology, Faculty of Psychology and Educational Sciences, University of Geneva, Geneva, Switzerland., 2Brain and Language Lab, Cognitive Science Hub, University of Vienna., 3Dept of Behavioral and Cognitive Biology, Faculty of Life Sciences, University of Vienna., 4Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria, 5McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, Montreal, Quebec, Canada., 6Department of Linguistics, Faculty of Psychology and Educational Sciences, University of Geneva, Geneva, Switzerland
Recent studies have shown promising results in using computational modeling approaches for pinpointing the neural signatures underlying the processing of linguistic hierarchy at different levels (semantics: Broderick, 2019; Huth, 2016; syntax: Pallier, 2010; Brennan, 2012; acoustics: Rutten, 2018; Santoro, 2017). However, the interaction across these levels is poorly understood. Our goal is to investigate if the encoding of syntactic information in the brain is modulated by prosodic cues carried by naturalistic speech signals. Crucially, the influence between prosody and syntax is reciprocal (Bennett & Elfner, 2019), thus, a computational model representing these two different levels can uncover if and how the neural processing of linguistic (i.e., syntactic) information is facilitated by their interaction. To investigate the interaction between prosodic and syntactic information in the brain, we are currently analyzing a previously published MEG dataset consisting of data from 11 subjects recorded while listening to continuous speech (Donhauser, 2020). The stimuli that were presented to the participants consisted of four TED talks extracted from the TEDLium Dataset (Rousseau, 2012). Two sets of features were extracted from the TED talks to characterize the time course of the prosodic and syntactic model (PS model). The prosodic features are composed of intonational and rhythmic information obtained from a spectral analysis of the speech signal. The syntactic features consist of both statistical and linguistic information present in the talks. More specifically, these consist of two subsets: (1) a subset representing the contextual predictability of part-of-speech (PoS) for each word of the TED talks, derived via the state-of-the-art transformer GTP-2 (Radford, 2019); (2) a subset describing the dependency structure of each of the TED talk sentences, characterized by the number of left-side connections of each word obtained from a transition-based dependency parser (Honnibal, 2015; Lopopolo, 2021). We are currently planning to evaluate the prosody-syntax interaction in two steps. First, a regularized linear regression between the source-localized brain activity and the sets of features of the PS model will be used to assess the amount of variance explained by the two sets of spaces. Secondly, a variance partition analysis will be used to determine not only the amount of unique information explained by syntactic and prosodic features in the brain but also their interaction. The ongoing project will allow a deeper understanding of how brain dynamics involved in the processing of prosodic cues carried by the spectrotemporal components of the speech signal interact with those involved in the processing of syntactic information carrying different levels of complexity. Importantly, we predict that the exhaustive modeling of these two feature spaces can uncover their interaction not only from a temporal perspective but also pinpoint the brain areas involved in making use of paralinguistic information to boost the processing of higher-level, abstract linguistic features, ultimately facilitating speech comprehension.