Presentation

SUBBA REDDY OOTA¹, Frederic Alexandre¹, Xavier Hinaut¹; ¹Inria Bordeaux, France

Recent neuroscience studies explored how multiple brain regions in a language network can be associated with syntactic and semantic representations of the stimulus words or sentences or naturalistic stimuli. Few recent papers have also obtained purely syntactic embeddings using constituency trees and fine-grained word syntactic features to show that these embeddings can indeed significantly predict many regions of the language system. However, previous works did not explore the explicit representation of syntactic features that fully encode the information present in dependency trees mainly for three reasons: (i) types of dependencies are not used, (ii) random walks cannot encode graph structure very well, and (iii) it is much harder to interpret the constituency tree-based embeddings when compared to simple fine-grained syntactic features. Moreover, most existing syntax studies have focused only on a few subjects involved in reading English text. In this study, we explicitly introduce syntactic word embeddings obtained from graph convolutional networks (GCN): (i) which utilizes the dependency context of a word that encodes information about the syntactic structure of sentences, (ii) remove the prior information of context embeddings by providing random initialization as node features. Using the features of pretrained syntactic embeddings and story text (reading and listening stories), we model the brain representation of syntax for reading and listening comprehension. First, we find that our syntactic structure-based features from GCN explain additional variance in the brain activity across multiple regions of the language system. At the same time, we see that regions in the language network are well-predicted by syntactic features during story listening compared to reading. We also notice that both syntax and semantics are overlapped and distributed in the language system for reading and listening than those suggested by classical studies (syntax is specific to Brodmann area 44). Overall, for both reading and listening stories, syntactic structure-based features from GCN consist of very low semantic information similar to constituency tree-based embeddings. It is reasonable to assume that any additional variance predicted by the syntactic embeddings from GCN compared to the BERT semantic feature spaces is mainly due to their syntactic information.

Topic Areas: Syntax, Meaning: Lexical Semantics