What can NLP systems teach us about language in the brain?
Organizer: Mariya Toneva1,2,3; 1PhD Candidate, Carnegie Mellon University, 2Postdoctoral Fellow, Princeton University, 3Assistant Professor, Max Planck Institute for Software Systems
Presenters: Evelina Fedorenko, John Hale, Jean-Rémi King, Leila Wehbe, Alexander Huth
In the last few years, new computational tools have emerged for natural language processing (NLP) that significantly outperform previous methods across linguistic tasks, ranging from predicting upcoming words to answering comprehension questions. In particular, these methods learn to represent individual words and to flexibly combine these representations to account for the surrounding context and the task at hand, without enforcing specific constraints from linguistics. Recent work in neurolinguistics shows that these models can also predict brain activity during language comprehension to an impressive degree. How these new methods can improve our understanding of the neurobiology of language remains an open question. In this symposium, the speakers will discuss their perspective on the benefits and limitations of utilizing recent NLP systems for improved understanding of language in the brain. Our target audience is researchers who want to make scientific inferences about the neurobiology of language using powerful but complex computational methods.
The neural architecture of language: Integrative modeling converges on predictive processing
Evelina Fedorenko1; 1Associate Professor, MIT
Recent advances in machine learning have produced artificial neural networks (ANNs) that achieve remarkable performance on diverse language tasks, providing the first computationally precise models of how language might work in the brain. We have begun to investigate whether state-of-the-art ANN language models capture human brain activity in the language-selective network. Adapting a pipeline developed in vision, we tested 43 language models on several neural datasets (fMRI+ECoG) and found that the ‘transformer’ ANN models accurately predict neural responses, in some cases achieving near-perfect predictivity. Critically, model-to-brain fit correlates with model performance on the next-word prediction task, but not other language tasks, suggesting that optimizing for predictive representations may be a critical shared feature of biological and artificial neural networks for language. We are now investigating the most brain-like models in richer detail to isolate the key contributors to model-to-brain fit and to develop intuitive theories around their inner workings.
Mapping Combinatory Categorial Grammar to real places and times in the brains of human language-comprehenders
John Hale1; 1Professor, University of Georgia
Mapping between linguistics, where sentences derive from abstract grammatical principles, and real brains, where neurons need more oxygen 4-6 seconds after doing extra language-processing work, has always seemed daunting. This talk reports recent results bringing these two worlds closer together, using Combinatory Categorial Grammar to formalize what it means to “understand” a sentence by finding a grammatical analysis. Varying the order in which these analysis steps proceed yields a family of forward models that, in combination with fMRI, shed light on the “eagerness” of everyday human language processing in the brain.
When, where and why do deep nets learn to process language like the brain?
Jean-Rémi King1,2; 1Researcher, CNRS, 2École normale supérieure
Do deep learning models learn to process language similarly to humans, and is this similarity driven by specific principles? Here, we test whether the activations of artificial neural networks trained on (1) image/sound, (2) word and (3) sentence processing linearly map onto human brain responses to written words, as recorded with magneto-encephalography (MEG, n=204), functional Magnetic Resonance Imaging (fMRI, n=589), and intracranial electrodes (n=176 patients, 20K electrodes). Our results reveal that visual (or sound), word and language algorithms respectively correlate with distinct areas and dynamics of the left-lateralized cortical hierarchy of reading (or speech processing, respectively). However, only specific subsets of these algorithms converge towards brain-like representations during their training. Overall, these results reveal the structural and training principles that lead deep nets to converge to brain-like computations.
Testing neurobiology of language theories in the wild with natural language processing
Leila Wehbe1; 1Assistant Professor, Carnegie Mellon University
Do findings about the brain obtained from controlled language tasks also apply when processing natural language that is not constrained into clean categories? Naturalistic experiments, by using complex language stimuli, allow us to study language processes in the wild, and to test if theories built on controlled stimuli generalize to the natural setting. To analyse these complex experiments, natural language processing offers powerful tools that can be carefully combined to create in vitro, post-hoc, computationally-controlled experiments. These computationally-controlled experiments can test targeted hypotheses and break the tie between different theoretical models of language processing. To make these generalizability tests accessible, we have built BOLDpredictions, an online tool for simulating fMRI experiments and testing the generalizability of findings from controlled experiments to the natural setting.
Interpretable multi-timescale models for predicting fMRI responses to continuous natural speech
Alexander Huth1; 1Assistant Professor, University of Texas at Austin
Natural language contains information at multiple timescales. To understand how the human brain represents this information, one approach is to build encoding models that predict fMRI responses to natural language using representations extracted from neural network language models (LMs). However, these LM-derived representations do not explicitly separate information at different timescales, making it difficult to interpret the encoding models. In this work we construct interpretable multi-timescale representations by forcing individual units in an LSTM LM to integrate information over specific temporal scales. This allows us to explicitly and directly map the timescale of information encoded by each individual fMRI voxel. This approach outperforms other encoding models, particularly for voxels that represent long-timescale information. It also provides a finer-grained map of timescale information in the human language pathway. This serves as a framework for future work investigating temporal hierarchies across artificial and biological language systems.