My Account

Poster A23, Tuesday, August 20, 2019, 10:15 am – 12:00 pm, Restaurant Hall

Predictive Abilities During Visual Narrative Comprehension

Emily Coderre1, Elizabeth O'Donnell1, Emme O'Rourke1, Neil Cohn2;1University of Vermont, 2Tilburg University

Prediction during language processing remains a debated topic, but is thought to free up neural resources and facilitate processing of subsequent material. Prediction is typically measured by examining how the amplitude of the N400 event-related potential (ERP) component is modulated by cloze probability, the expectancy of a specific word given the contextual constraints of a preceding sentence. Highly predictable words in semantically constraining “high cloze” sentences generate reduced N400 amplitudes compared to words in less-constraining “low cloze” sentences. Importantly, non-verbal stimuli also elicit N400s, offering the potential to examine the role of prediction in other modalities, such as visual narratives which convey meaning across image sequences. Visual narrative comprehension relies on similar cognitive mechanisms as language comprehension. However, it remains unknown whether cloze probability modulates visual narrative sequences in the same way as in language. Here, we test this question using a cloze probability paradigm with visual narratives. In a pre-test, 200 healthy adults were shown the starting panels of a comic strip and asked “what comes next?” Cloze ratings for each strip corresponded to the percentage of participants who agreed on the predicted ending (ranging from 0-100%). From these ratings we generated high cloze (>40%, M=60%) and low cloze (<25%, M=20%) conditions. Anomalous strips were also created by replacing critical panels (cloze ratings 0-83%, M=40%) with semantically incongruous panels from different strips. The anomalous condition served as a control and should elicit the highest N400 amplitudes of the three conditions. For the EEG experiment, 22 adults (who did not offer pre-test ratings) viewed 60 strips in each condition during EEG recording using a 128-channel Geodesic Sensor net and NetStation 5. ERPs were time-locked to the onset of the critical panel in each strip. Analyses were conducted at nine scalp sites representing frontal, central, and parietal regions over the left hemisphere, midline, and right hemisphere. Results demonstrated enhanced negativities for anomalous conditions compared to both high and low cloze conditions at fronto-central sites from 500-700 ms. Differences also occurred between all three conditions from 500-600 ms in the expected pattern (anomalous sequences showing the largest amplitudes, followed by low cloze and high cloze). Across all items, cloze ratings correlated significantly with amplitude from 400-900 ms. Correlations were strongest over midline and right centro-parietal scalp and were all positive, indicating that as cloze rating decreased (i.e. as sequences became less predictable), amplitude decreased (i.e. became more negative). These results indicate that predictability has an analogous effect on visual narrative sequences as on sentences. High cloze sequences showed reduced ERP amplitudes compared to low cloze sequences from 500-600 ms. This is analogous to an N400 effect, although occurring slightly later (perhaps due to the more complex visual information inherent in this modality). Amplitude also correlated with cloze ratings such that less predictable sequences generated larger N400 amplitudes. Prediction thus appears to play a similar role in visual narrative comprehension as in language comprehension, providing further evidence that sequential images are processed using similar cognitive and neural mechanisms as language.

Themes: Meaning: Combinatorial Semantics, Reading
Method: Electrophysiology (MEG/EEG/ECOG)

Back