Poster C28, Wednesday, August 21, 2019, 10:45 am – 12:30 pm, Restaurant Hall
Electrophysiology of inferential processing in visual narratives
Neil Cohn1;1Tilburg University
Inference has been a primary focus of studies of discourse of both verbal and visual narratives, like comics or picture stories. Most of this work has emphasized backward-looking processes, where a reader must infer absent information from the subsequent information that is explicitly provided. However, comics have conventional forms where inference is demanded, but a narrative unit is still provided. “Action stars” depict a star-shaped “flash” the size of a full panel that suggests a climactic event, while leaving that event unspecified, meaning that a reader knows that the events are not depicted at that moment, rather than events being omitted entirely (Cohn, 2013). Action stars therefore allow us to ask: what are the neurocognitive correlates of inference generation when a narrative unit explicitly signals omitted information? We thus measured the event-related brain potentials (ERPs) to 60 visual narratives one image at a time to sequences that either depicted an explicit event, or that replaced it with an action star or a “noise” panel of non-representational scrambled lines (20 in each condition). A prior study of 101 participants showed that the climactic event was recognized as missing when it was omitted (mean rate of recognition: .71) thus confirming the inferential quality of the sequences. EEG was recorded from 24 participants using a 32 channel BrainVision ActiCHAMP. At the critical panel, in the 300-500 ms epoch, action stars and noise panels both generated large N400s compared to explicit-events, but did not differ from each other. These N400s reflect the cost of accessing semantic memory given the relative lack of bottom-up cues, but show that participants indeed attempted to find meaning in these images, despite their impoverished representations. These panels then diverged in their processing between 500 and 900ms. Action stars evoked sustained anterior negativities indicative of further interpretive processes (Baggio, 2018), while also eliciting a posterior P600, reflecting the update of this information into a growing mental model. Meanwhile, noise panels, which are not conventional representations, evoked a late frontal positivity (LFP) indicative of their low probability as substitutions for a visual event. This divergence at the critical panel suggests that participants attempted to integrate the symbolic meaning from action stars into a mental model of the visual discourse, while noise panels were simply recognized as incongruous. Nevertheless, as both action stars and noise panels omit event information, the subsequent panels evoked sustained negativities from 500-1100ms to both types of panels relative to those after explicit events. Such negativities suggest working memory processes further interpreting the missing event information (Baggio, 2018). These findings support that inferential processing in visual narratives evokes cascading mechanisms that differ depending on the (un)conventionality of the incoming information, and these mechanisms appear to overlap with those in language processing. Baggio, G. (2018). Meaning in the Brain. Cambridge, MA: MIT Press. Cohn, N. (2013). The visual language of comics: Introduction to the structure and cognition of sequential images. London, UK: Bloomsbury.
Themes: Meaning: Discourse and Pragmatics, Meaning: Combinatorial Semantics
Method: Electrophysiology (MEG/EEG/ECOG)