You are viewing the SNL 2017 Archive Website. For the latest information, see the Current Website.

 
Poster C40, Thursday, November 9, 10:00 – 11:15 am, Harborview and Loch Raven Ballrooms

The effect of multimodal predictability on the N400

Christine Ankener1, Maria Staudte1, Heiner Drenhaus1, Matthew W. Crocker1;1Saarland University

A word’s predictability, derived from its cloze probability, affects the N400-component on the target word itself (e.g. Kutas, 2011). Further, increasing the predictiveness of the linguistic context reduces N400 amplitudes for visually depicted target words in referring expressions (Tourtouri et al, 2015). By combining stable linguistic stimuli with varying visual contexts, we show that predictability of a target word is also sensitive to the probability profile of a context, derived from multimodal information, which is reflected in a modulation of the N400. We presented 368 visual displays paired with 92 different plausible German sentences of the type “The man spills on saturday the water in the kitchen“ in RSVP-style to 28 German native speakers (Ø age: 24,7). While the sentences were the same in each condition, the independent variable was the number of depicted objects that matched the verb constraint. Each display contained 4 clip- arts, out of which either 0, 1, 3 or all 4 were competitors (spill-able objects) for the upcoming target noun. Displays were presented 1000 ms prior to and throughout the whole sentence. As a result of the manipulation, the expectedness for the target noun was predicted to decrease as the number of competitors increased. That is, “the water“ could be predicted with 0% (when no spill-able object was among the 4 clip-arts), 33 % (when 3 out of 4 objects were spill-able), 25% (when all 4 objects were spill-able) or 100% (when only the water was shown among 3 non-spillable objects) certainty. We expected a reduced N400 on the noun as fewer competitors are available so that prediction is more certain. We analyzed the influence of our manipulation (Nr. of competitors) on the N400 amplitude, time locked to the onset of the target noun phrase, including ROIs for electrode site (frontal/central/parietal) as within-subject factors. Repeated measures analysis of variance (ANOVA) using Greenhouse-Geisser correction and follow-up pairwise t-tests with Bonferroni corrected p-values assessed statistical significance. Results revealed the expected graded modulation in the N400 amplitude elicited by the target noun (“water“), as a function of the number of potential visual referents determined on the basis of the verb’s selectional restrictions. Compared to the baseline condition (1) with the lowest N400, the amplitude was significantly larger for the conditions where 3 (p < .05) and 4 (p < .05) objects matched the verb constraint. Further, the N400 was largest in the 0 condition (p < .001) where no object matched the verb, i.e. the noun was least predictable. We attribute this effect to the sensitivity of the comprehender to the probability profiles that result, during situated sentence processing, from applying verb selectional preferences to a co-present visual scene such that uncertainty about upcoming words (nouns) can be reduced as much as possible. Our results are in line with studies showing that situated language processing is related to context-expectancy (e.g. van Berkum, 1999), even when derived by combining information from different modalities.

Topic Area: Meaning: Combinatorial Semantics

Back to Poster Schedule