Presentation
Search Abstracts | Symposia | Slide Sessions | Poster Sessions | Lightning Talks
Tracking the representational dynamics of linguistic composition during sentence comprehension: A proposed neural decoding study
Poster A47 in Poster Session A, Tuesday, October 24, 10:15 am - 12:00 pm CEST, Espace Vieux-Port
This poster is part of the Sandbox Series.
Junyuan Zhao1, Jonathan Brennan1; 1University of Michigan, Ann Arbor
[Introduction] The spatial and temporal correlates of linguistic composition have received much attention in the field (Brennan et al., 2020; Ding et al., 2016; Pylkkänen, 2020; Zaccarella & Friederici, 2015). However, the representational details of composition remained underspecified in neuroscientific terms: What is the relationship between inputs and outputs of this computation (c.f. Fyshe et al., 2019)? In this proposed study, we probe how linguistic representations evolve during phrasal composition using neural decoding (King et al., 2018). Specifically, we test the hypothesis that syntactic information of a verb head is re-activated when it is “merged” into a Verb-Adverb phrase (e.g., at the right bracket in [eat slowly]). [Design] The current study involves (1) training a decoder that exclusively targets syntactic category information and (2) using it to probe how category information is activated across a phrase. First, we train a V-Adv part-of-speech (PoS) temporal decoder on EEG data recorded during rapid serial visual presentation of sentences. Then, the decoder is used to predict PoS on data from a separate testing region of those sentences where we manipulate compositional context. Activation dynamics of the grammatical head (the verb) is quantified as the probability that a V-Adv decoder assigns the corresponding label (“V”). To illustrate, for the word-wise translated Chinese sentence “[run DE quickly] DE team [arrived early]” (“The team that ran fast arrived early”; the DE morpheme either introduces a modifier or a head), a V-Adv decoder will be trained on the sentence-final region and is then applied to predict PoS tags at the critical Adv (“quickly”) in the sentence-initial region. We manipulate two factors in the critical region: Bracket Closing (present: “run quickly”; absent: “run quick”) and Conceptual Association (high: “run quickly”; low: “run profoundly”), resulting in four conditions of 60 experimental sentences each. If PoS information is reactivated phrase-finally, we expect higher verb activation at the critical Adv when there is bracket closing, independent of conceptual association. To evaluate generalizability, we also train a separate decoder on epochs from a phrase plausibility judgment experiment (thus “between-task”). [Analysis] EEG epochs (train and test) will be baseline corrected and then augmented with a 3-epoch averaging technique (Murphy et al., 2022). Augmented epochs enter the temporal decoding pipeline based on a sliding logistic regression classifier over all sensors, yielding a time-series of P(Verb) across the critical region. Cluster-based permutation tests will be performed to identify significant clusters between conditions. [Results and Directions] Pilot analysis (N=2) yielded PoS decoding accuracy of above 70% within the training epochs using cross-validation, and above-chance accuracy (52%) in test epochs. ERP analyses showed a composition-induced negativity at Fz in the 250-300ms time window, which is modulated by Bracket Closing and Conceptual Association (consistent with Neufeld et al., 2016; Parrish & Pylkkänen, 2022). Statistical comparisons will be performed once we reach a target N of 30.
Topic Areas: Syntax and Combinatorial Semantics, Computational Approaches