Poster D33, Wednesday, August 21, 2019, 5:15 – 7:00 pm, Restaurant Hall
Can computers understand word meanings like the human brain does? Assessing the correlation between EEG responses and NLP-generated word similarity
Xing Tian1,2, Linmin Zhang1,2, Lingting Wang2,3, Jinbiao Yang4, Peng Qian5, Xuefei Wang6, Xipeng Qiu6, Zheng Zhang1,7;1NYU Shanghai, 2NYU-ECNU Institute of Brain and Cognitive Science, 3East China Normal University, 4Max Planck Institute for Psycholinguistics, 5MIT, 6Fudan University, 7AWS Shanghai AI Lab
Semantic representation, a crucial window into human cognition, has been studied mostly independently in several disciplines, including cognitive neuroscience and computer science. Within cognitive neuroscience, semantic relatedness can elicit N400 priming effects: target words in unrelated word pairs (e.g., apple-moon) elicit larger responses than those in related word pairs (e.g., star-moon) around 400 ms after target onset. Within computer science, it has been assumed that semantically similar words tend to appear in similar context, and thus semantic similarity can be computed via co-occurrence frequency. Recently, with deep learning methods, natural language processing (NLP) models can learn semantic similarity from large corpus. Can the representational formats established independently in two complex systems – our brain and computers – be related? More specifically, to what extent can NLP models predict humans' N400 priming effects? Do distinct NLP models differ in their predictability? To address these questions, we implemented a typical two-word priming paradigm with EEG recordings. Then we evaluated several representative NLP models and tested which model was the best predictor for the EEG responses. Assessing the correlation between the measurement of semantic similarity in two representation formats can not only shed light on the nature of N400, but also potentially provide a reliable evaluation for NLP models. We collected 32-channel EEG data from 25 participants (Chinese native speakers). Each participant read 240 critical trials (pairs of words) and 120 fillers (pairs of a word and a nonword) and performed lexical decision tasks. For each critical trial, correlation coefficient (r) was computed for each millisecond and each channel between the averaged (over participants) EEG signals and the cosine similarity value by an NLP model (between the prime and the target). Among the three NLP models involved in this study, CBOW (Mikolov et al. 2013) is based on word co-occurrence within local context; GloVe (Pennington et al., 2014) is based on word co-occurrence within both global and local context; CWE, a model similar to CBOW, captures also single-Chinese-character-level information (Chen et al., 2015). For each model, the 32 * 1000 r values (channels * milliseconds) formed a heat map, showing how these r values evolve over time and across channels. We used permutation tests to find statistically significant r values. We found that for each of the three models, there were significant correlations between NLP-generated word similarity and EEG signals elicited between 200 to 300 ms after target onset in most posterior channels. The spatiotemporal information of these elicited EEG signals is consistent with known spatiotemporal information of N400 priming effects, suggesting that overall, these NLP models can indeed predict elicited N400 responses. Moreover, the most robust correlation was found between GloVe model and EEG responses in the channel Oz around 300 ms after target onset (r = 0.173, p = 0.007). We found correlations between N400 responses and NLP-generated word similarity. Our findings revealed a specific time course of semantic processing, linked semantic representation in the human brain and NLP models, and provided an objective and reliable evaluation for NLP models.
Themes: Meaning: Lexical Semantics, Computational Approaches
Method: Electrophysiology (MEG/EEG/ECOG)