Presentation
Search Abstracts | Symposia | Slide Sessions | Poster Sessions | Lightning Talks
Neural source dynamics of predictive and integratory structure building during natural story listening
There is a Poster PDF for this presentation, but you must be a current member or registered to attend SNL 2023 to view it. Please go to your Account Home page to register.
Poster A45 in Poster Session A, Tuesday, October 24, 10:15 am - 12:00 pm CEST, Espace Vieux-Port
Cas W. Coopmans1,2, Helen de Hoop3, Peter Hagoort1,2, Andrea E. Martin1,2; 1Max Planck Institute for Psycholinguistics, 2Donders Institute for Brain, Cognition and Behaviour, Radboud University, 3Centre for Language Studies, Radboud University
Neuro-computational language models have gained popularity as linguistically interpretable tools for studying language comprehension in naturalistic contexts. Here, we use this method to investigate to what extent three commonly used parsing strategies can account for neural activity related to Dutch sentence comprehension. In particular, we test how well the brain activity of people listening to Dutch audiobook stories is predicted by an integratory bottom-up parser, a predictive top-down parser, and a mildly predictive left-corner parser. Dutch syntax exhibits mixed headedness, which makes it more amenable to comparing different parsing strategies than English, because English phrases are strictly head-initial and therefore particularly well-suited for the left-corner parsing strategy. By comparing these parsers in terms of how well they reconstruct brain activity during natural story listening, we intend to uncover possible mechanisms underlying high-level language comprehension. Twenty-four Dutch participants listened to 49 minutes of Dutch audiobook stories while their brain activity was measured with magnetoencephalography. Each word in the audiobook was assigned a complexity metric corresponding to the number of nodes that would be visited by the three parsers when incrementally integrating the word into the hierarchical structure of the sentence. These syntactic complexity metrics were then mapped onto delta-band source activity using multivariate temporal response functions (TRFs). TRFs are linear kernels that describe how the brain responds to a representation of a (linguistic) feature. By additionally including lower-level features as predictors in our TRF models, we explicitly modeled the acoustic (i.e., acoustic spectrogram, acoustic onsets) and statistical (i.e., word frequency, entropy, surprisal) properties of the auditory stimuli. The results show that all three syntactic predictors explain variance in the left-hemispheric language network on top of the variance accounted for by lower-level predictors. Strikingly, activity in left inferior frontal and superior temporal regions most strongly reflects node counts derived by the top-down method, showing that predictive structure building is an important component of Dutch sentence comprehension. The effects of node count derived from bottom-up and left-corner parsing models, while significant, were considerably weaker. The weak effects of bottom-up node count in the presence of strong top-down effect suggests that predictive sentence comprehension (captured by strong effects of top-down node count) is accompanied by reduced demands on integratory processing (reflected in weak effects of bottom-up node count). Moreover, the absence of strong effects of left-corner node counts suggests that its mildly predictive strategy does not capture Dutch sentence comprehension well, in contrast to what has been found for English. This might be related to the fact that Dutch contains head-final phrases, whose left corner often contains multiple words, making the left-corner parsing strategy insufficiently incremental. In sum, using neuro-computational language models, we find that Dutch sentence comprehension is best modeled via a predictive parsing strategy, contrasting previous naturalistic studies conducted in English. These findings, though still from related languages, therefore underscore the need for more work on typologically diverse languages, whose structural properties are different from those of English and therefore invite different parsing strategies within the fronto-temporal language network.
Topic Areas: Syntax and Combinatorial Semantics, Computational Approaches