Poster C37, Wednesday, August 21, 2019, 10:45 am – 12:30 pm, Restaurant Hall
Language Background and Lexical Representations during Naturalistic Reading: An RSA Analysis of Fixation Related fMRI Data on L1 and L2 Readers of English
Benjamin Schloss1, Friederike Seyfried1, Chun-Ting Hsu1,2, Ping Li Li1;1Pennsylvania State University, 2Kyoto University
During reading, lexical access happens at the early stage of text comprehension, leading to later stages of coherent mental representations such as situation models. When this input is missing or impoverished (small vocabulary size), reading comprehension suffers. Although bilinguals reading in their L2 also show poorer reading comprehension than monolinguals, it is unclear to what degree deficits in their high-level text representations are caused by impoverished low level input versus difficulties in later stages of processing. In the current study, we simultaneously obtained fMRI and eye tracking data from 52 monolingual and 56 bilingual participants (L1 = Chinese); of the bilinguals, 28 living in the US (immersed), 28 living in China (non-immersed) while they read five English texts on five scientific topics in a self-paced reading paradigm. We modeled brain responses for each of 346 content words occurring in the texts from 12 brain regions along the left ventral “what” stream and conducted a representational similarity analysis between each group’s data and ELMo, a deep learning model which uses a recurrent neural network to capture both sentential context as well as general word meaning and has been shown to outperform traditional language models that only capture general word meaning on a variety of natural language processing tasks. Representations of the words were extracted by estimating double gamma (DG) or initial dip (ID) hemodynamic response which coincided with fixation on one of the words. This, in turn, was input to a singular value decomposition (SVD) or independent component analysis (ICA). The final RDM (representational dissimilarity matrices) for each brain region in each group was then calculated by either computing individual level RDMs for each individual and averaging them, or hyperaligning the extracted neural features and averaging the representations, from which the final RDM was created. We observed that the ID model of the HRF provided better fit to the ELMo model for monolinguals compared to bilinguals (Z=-3.32, p=.00045). This is in line with the difference between monolinguals and bilinguals in reading speed. We also observed evidence for a gradient along the posterior-to-anterior axis of the ventral stream (excluding V1 which was an outlier) that explained 13.9 % of the variation (rank correlation=.37) in the observed cosine similarities with the model, but did not reach significance due to insufficient data. Finally, we found partial evidence in support of language background affecting similarity with ELMo. While immersed bilinguals did show consistently higher similarity to the model than non-immersed bilinguals, monolinguals neural representations were systematically more dissimilar than those of both bilingual groups. Further research is needed to understand the mechanisms underlying this effect. Context-dependent models have not been studied in detail with regard to how well they capture representations in L2-learners and in native speakers. It is possible that quicker fixation/reading pace (shorter inter-stimulus intervals) as well as more experience with words (possibly less detailed processing of low-level lexical information) lead to worse performance of these models for monolinguals.
Themes: Meaning: Lexical Semantics, Computational Approaches
Method: Functional Imaging