Slide Slam B5
Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses
Richard Antonello1, Javier Turek2, Vy Vo2, Alexander Huth1; 1UT Austin, 2Intel Labs
There are a multitude of common techniques for analytically representing the information contained in natural language. Language representations can highlight specific linguistic properties, such as parts-of-speech or sentence chunks, or utilize well-known NLP models such as the intermediate layers of pretrained language models. In fields such as linguistics, natural language processing, and cognitive psychology, qualitative adjectives are often used to describe these language representations -- e.g. “low-level” or “high-level” and “syntactic” or “semantic”. The use of these words belies an unstated hypothesis about the nature of the space of language representations -- namely that this space is fundamentally low-dimensional, and therefore that the information from the representations in this space can be efficiently described using a few categorical descriptors. In this work, we attempt to directly map the low-dimensional space of language representations by generating “representation embeddings” using a method inspired by the work of Zamir et. al. This method uses the transfer properties between representations to map their relationships. Specifically, language representations are embedded based on how well the information contained in each representation can be used to regress to 100 prominent language representations from a large group. These representations include well-known word embedding spaces such as GloVe and FLAIR, as well as intermediate layers from common language models such as GPT-2, BERT, and Transformer-XL, and two machine translation models. Using these mapped 100-dimensional embeddings, we use multidimensional scaling to generate a low-dimensional structure. The principal dimension of this structure is especially interesting as it seems to capture an intuitive notion of a language representation hierarchy. Representations with negativelow values along the main dimension include word embeddings as well as the earliest layers of most of the language models and machine translation models, whereas higher valued representations with more positive values include the deeper layers of these models, as well as many interpretable syntactic and semantic representations. We show that mapping the principal dimension of the representation embeddings onto the brain recovers voxels that are thought to involve to higher-order, longer timescale language processing, such as those in prefrontal cortex and the precuneus. We also generate fMRI encoding models which try to predict BOLD response from a natural language stimulus represented by our 100 language representations. Using a discriminability metric, we show that representation embeddings can be correctly matched to their corresponding voxelwise encoding model performance maps over 90% of the time for all representations, and 100% of the time for the majority of representations. This suggests that the low-dimensional structure we generate is reflected in the hierarchy of linguistic representations in the brain.