Poster B3, Tuesday, August 20, 2019, 3:15 – 5:00 pm, Restaurant Hall
Prediction: when, where & how? An investigation into spoken language prediction in naturalistic virtual environments
Eleanor Callaghan1, David Peeters1,2, Peter Hagoort1;1Max Planck Institute for Psycholinguistics, 2Tilburg University
A longstanding question remains as to how language is processed so efficiently and so rapidly. Recent evidence suggests that this fast processing is assisted by the prediction of upcoming linguistic input. Visual word paradigms have been fundamental in providing evidence for prediction in spoken language comprehension, by demonstrating that participants make anticipatory eye movements towards depicted objects before their associated noun is spoken. However, it is questionable whether this type of prediction also occurs in more naturalistic, everyday environments, which are often intrinsically rich and visually complex. It is equally unclear to what extent subtle cues in speech are used to update predictions. Recent work from our Virtual Reality laboratory supports that prediction does indeed occur in naturalistic environments. In a series of four experiments, we observed anticipatory eye movements to objects in rich virtual environments, even when increasing the number of distractor objects in the scene and the proportion of filler sentences included in the experiment (Heyselaar et al., under review). The number of different verbs used in the experiments was rather low, however, thereby questioning the generalizability of the findings. We therefore continued to investigate these findings in a novel virtual reality experiment, in which participants listened to sentences spoken by a virtual agent during a virtual tour of eight scenes (e.g., an office, a living room, a canteen). The agent discussed her relation to each scene while participants’ eye movements were continuously recorded. Spoken stimuli, produced by the agent (incl. lip sync and gaze to the participant), consisted of 128 sentence pairs that contain a subject-verb-object clause. Sentences within each pair were identical apart from the verb, one of which is restrictive (related to a single object in the scene), where the other is unrestrictive (related to multiple objects in the scene). Sentence pairs were separated into two lists that participants were randomly assigned to, so that no participant heard both sentences from a pair. Only 50% of the sentences referred to an object that was present in the scene. The remaining 50% of sentences behaved as filler trials. Preliminary results confirm that, in a critical time window that precedes noun onset, the mean proportion of fixations towards the target object were greater in the restrictive compared to unrestrictive condition. These findings indicate that prediction in language comprehension occurs in naturalistic, real life situations. Ongoing work additionally investigates how disfluencies in speech, for example hesitations, influence predictions. In light of the importance of prediction for efficient language comprehension, we aim to establish which elements of speech are used to predict upcoming utterances in more ecologically valid scenarios.
Themes: Control, Selection, and Executive Processes, Speech Perception
Method: Eye Tracking