What can NLP systems teach us about language in the brain?
Friday, October 8, 2021, 6:00 - 8:00 am PDT Log In to set Timezone
Organizer: Mariya Toneva1,2,3; 1Carnegie Mellon University, 2Postdoctoral Fellow, Princeton University, 3Assistant Professor, Max Planck Institute for Software Systems
Presenters: Evelina Fedorenko, Jixing Li, Jean-Rémi King, Leila Wehbe, Alexander Huth
In the last few years, new computational tools have emerged for natural language processing (NLP) that significantly outperform previous methods across linguistic tasks, ranging from predicting upcoming words to answering comprehension questions. In particular, these methods learn to represent individual words and to flexibly combine these representations to account for the surrounding context and the task at hand, without enforcing specific constraints from linguistics. Recent work in neurolinguistics shows that these models can also predict brain activity during language comprehension to an impressive degree. How these new methods can improve our understanding of the neurobiology of language remains an open question. In this symposium, the speakers will discuss their perspective on the benefits and limitations of utilizing recent NLP systems for improved understanding of language in the brain. Our target audience is researchers who want to make scientific inferences about the neurobiology of language using powerful but complex computational methods.
The neural architecture of language: Integrative modeling converges on predictive processing
Evelina Fedorenko1; 1Associate Professor, MIT
Recent advances in machine learning have produced artificial neural networks (ANNs) that achieve remarkable performance on diverse language tasks, providing the first computationally precise models of how language might work in the brain. We have begun to investigate whether state-of-the-art ANN language models capture human brain activity in the language-selective network. Adapting a pipeline developed in vision, we tested 43 language models on several neural datasets (fMRI+ECoG) and found that the ‘transformer’ ANN models accurately predict neural responses, in some cases achieving near-perfect predictivity. Critically, model-to-brain fit correlates with model performance on the next-word prediction task, but not other language tasks, suggesting that optimizing for predictive representations may be a critical shared feature of biological and artificial neural networks for language. We are now investigating the most brain-like models in richer detail to isolate the key contributors to model-to-brain fit and to develop intuitive theories around their inner workings.
Cortical processing of pronoun resolution revealed by computational models
Jixing Li1; 1Postdoctoral Researcher, New York University Abu Dhabi
Human language processing involves not only combining word meanings in accordance with semantic and syntactic constraints, but also figuring out who and what is being referred to. Here we present the first study towards a mechanistic understanding of the neural basis for referential processing. Using both fMRI and MEG, we identified a consistent increase of activity in a network spanning the anterior and posterior left middle temporal gyrus and the angular gyrus for pronoun processing during naturalistic listening for both English and Chinese speakers. We then leveraged computational models to examine the cognitive processes underlying pronoun resolution. We evaluated the neural fit of three symbolic models that each formalizes a different explanation for pronoun resolution in the cognitive and linguistic literature, as well as two neural network models with different architectures. Our results favor the memory-based symbolic model, suggesting a domain-general mechanism of pronoun resolution that resembles memory retrieval.
When, where and why do deep nets learn to process language like the brain?
Jean-Rémi King1,2; 1Researcher, CNRS, 2École normale supérieure
Do deep learning models learn to process language similarly to humans, and is this similarity driven by specific principles? Here, we test whether the activations of artificial neural networks trained on (1) image/sound, (2) word and (3) sentence processing linearly map onto human brain responses to written words, as recorded with magneto-encephalography (MEG, n=204), functional Magnetic Resonance Imaging (fMRI, n=589), and intracranial electrodes (n=176 patients, 20K electrodes). Our results reveal that visual (or sound), word and language algorithms respectively correlate with distinct areas and dynamics of the left-lateralized cortical hierarchy of reading (or speech processing, respectively). However, only specific subsets of these algorithms converge towards brain-like representations during their training. Overall, these results reveal the structural and training principles that lead deep nets to converge to brain-like computations.
Testing neurobiology of language theories in the wild with natural language processing
Leila Wehbe1; 1Assistant Professor, Carnegie Mellon University
Do findings about the brain obtained from controlled language tasks also apply when processing natural language that is not constrained into clean categories? Naturalistic experiments, by using complex language stimuli, allow us to study language processes in the wild, and to test if theories built on controlled stimuli generalize to the natural setting. To analyse these complex experiments, natural language processing offers powerful tools that can be carefully combined to create in vitro, post-hoc, computationally-controlled experiments. These computationally-controlled experiments can test targeted hypotheses and break the tie between different theoretical models of language processing. To make these generalizability tests accessible, we have built BOLDpredictions, an online tool for simulating fMRI experiments and testing the generalizability of findings from controlled experiments to the natural setting.
Interpretable multi-timescale models for predicting fMRI responses to continuous natural speech
Alexander Huth1; 1Assistant Professor, University of Texas at Austin
Natural language contains information at multiple timescales. To understand how the human brain represents this information, one approach is to build encoding models that predict fMRI responses to natural language using representations extracted from neural network language models (LMs). However, these LM-derived representations do not explicitly separate information at different timescales, making it difficult to interpret the encoding models. In this work we construct interpretable multi-timescale representations by forcing individual units in an LSTM LM to integrate information over specific temporal scales. This allows us to explicitly and directly map the timescale of information encoded by each individual fMRI voxel. This approach outperforms other encoding models, particularly for voxels that represent long-timescale information. It also provides a finer-grained map of timescale information in the human language pathway. This serves as a framework for future work investigating temporal hierarchies across artificial and biological language systems.