Poster D37, Friday, August 17, 4:45 – 6:30 pm, Room 2000AB

An integrated neural decoder of experiential and linguistic meaning

Andrew Anderson1, Jeffrey Binder2, Leonardo Fernandino2, Colin Humphries2, Lisa Conant2, Douwe Kiela3, Rajeev Raizada1, Feng Lin1, Edmund Lalor1,4;1University of Rochester, 2Medical College of Wisconsin, 3Facebook, 4Trinity College Dublin

How the human brain codes the meaning of words is thought to draw on a combination of linguistic knowledge and non-linguistic experience interacting with the word. Despite extensive research into decoding meaning from brain activation there is minimal direct neural evidence that both linguistic and experiential aspects of meaning are present in neural activity elicited during natural reading. In part this is because neural decoding has been reliant on matching brain activation to models of word meaning that are either linguistic or experiential in nature. We rectify this. In the challenging task of decoding 14 participants functional Magnetic Resonance Imaging (fMRI) activation elicited in sentence reading, we integrate a state-of-the-art distributional semantic model induced from word co-occurrences in natural text with a behaviorally rated model of peoples’ sensory, motor, social, emotional, and cognitive experiences with words. By demonstrating the integrated approach systematically boosts decoding accuracy over using either model in isolation, we reveal the first evidence that both experiential and linguistic elements of meaning are detectable in brain activation elicited in natural reading. We provide confirmatory evidence for the hitherto untested assertion that the text-based model will offer a decoding advantage for sentences containing linguistically oriented “abstract” words. By introducing a new method to decode individual’s brain activation using other peoples’ brain activation, we derive an estimate for the upper bound on individual-level decoding accuracy achievable. By comparing the model-based results to this upper bound we identify sentences for which model performance can be improved upon. These sentences are most associated with abstract concepts, benefit and upper-limb actions which suggests that these semantic domains may be important areas on which to focus future model development. Finally, by decoding all 14 participants fMRI data in parallel using both models, and then integrating all 14 decoding decisions to generate a group-level estimate we achieve high accuracy results surpassing any at individual-level. This high performance level hints that information extracted across multiple peoples’ fMRI data could itself be sufficiently detailed to enhance semantic model representations for artificial intelligence purposes.

Topic Area: Meaning: Lexical Semantics

Back