Slide Sessions

Session #

Session Title

Time & Location

Session A

Speech Processing

Wednesday, August 27, 4:30 - 5:50 pm
Effectenbeurszaal

Session B

Language Evolution and Brain Structure

Thursday, August 28, 8:30 - 9:50 am
Effectenbeurszaal

Session C

Combinatorial Processing: Syntax, Semantics, Pragmatics

Friday, August 29, 8:30 - 9:50 am
Effectenbeurszaal

Session D

Lexical Processing and Cognitive Control

Friday, August 29, 1:00 – 2:20 pm
Effectenbeurszaal



Slide Session A

Wednesday, August 27, 4:30 - 5:50 pm, Effectenbeurszaal

Speech Processing

Chair: Heather Bortfeld
Speakers: Kristen Berry, Mirjam J.I. de Jonge, Julius Fridriksson, Joao Correia

The When and Where of Multisensory Speech Processing

Kristen Berry1, Valerie Nunez1, Werner Doyle1, Callah Boomhaur1, Lucia Melloni1,3, Daniel Friedman1, Patricia Dugan1, Orrin Devinsky1, Eric Halgren4, Thomas Thesen1,2; 1NYU Comprehensive Epilepsy Center, Department of Neurology, School of Medicine, New York University, 2Department of Radiology, School of Medicine, New York University, 3Department of Neurophysiology, Max Planck Institute for Brain Research, Frankfurt am Main, Hessen, Germany, 4Multimodal Imaging Lab, University of California, San Diego

Converging evidence suggests that audio-visual speech integration occurs “early” in superior temporal regions. However, additional and spatially widespread cortical areas have been identified in audio-visual speech integration, including the inferior frontal gyrus, premotor cortex, and anterior cingulate gyrus. Little is known about the spatio-temporal dynamics of auditory-visual processing and integration across these other areas of cortex. Further, it is unclear if audiovisual speech integration occurs exclusively in speech specific areas or extends to non-speech specific audio and visual regions as well. In this study, we use electrocorticography (ECoG) recordings, which offer improved spatial and temporal resolution over non-invasive human neuroimaging techniques to study auditory-visual speech integration. We recorded intracranial EEG in 13 patients implanted with subdural electrodes for invasive monitoring prior to epilepsy surgery. Patients were presented with words in the form of video images of the lower face and corresponding auditory speech signals. Conditions included auditory (A), visual (V), and audio-visual (AV) speech, as well as sensory control stimuli (Ac, Vc). A non-parametric randomization test with temporal clustering to correct for multiple comparisons was used to compare average high gamma (70-190Hz) power (HGP) band responses for speech specificity (A > Ac; V >Vc) and integration (AV vs. A+V). We recorded from a total of 1627 surface and depth electrodes across the whole brain. Significant responses to experimental stimulation were found in 312 (19%) of the electrodes. Of these, 148 (47%) showed auditory speech-specificity, whereas 34 (11%) showed visual speech-specificity, and 31 (10%) showed integration responses. Of the electrodes showing integration responses, 10 (32%) also showed auditory speech specificity and 16 (52%) showed visual speech specificity. Electrodes were localized to structural MRI scans and grouped based on anatomical parcellations and responsiveness. Both auditory speech-specificity and multisensory integration was highest in posterior superior temporal gyrus (pSTG) where 35% of all electrodes in this region exhibited speech-specificity and 11% showed integration responses. Of the electrodes showing integration in pSTG, a majority (83%) also showed auditory speech-specificity and 50% showed visual speech-specificity. The earliest evidence for auditory speech-specificity occurred in pSTG and supramarginal gyrus (SMG) around 100ms relative to auditory onset. Similarly, the earliest evidence of integration was observed in SMG (104ms) and pSTG (144ms). Late integration responses were found in precentral (244ms) and caudal middle frontal (319ms) regions of the frontal lobe. Integration responses in the caudal middle frontal area occurred exclusively in electrodes selective for visual speech (V>Vc). This study shows that auditory-visual speech integration is a multistage process occurring across widespread cortical areas. Early integration occurs in both speech-specific and non-specific auditory areas of the superior temporal gyrus. This is followed by later integration in frontal areas exclusively involved with visual speech processing and precentral areas involved with both speech-specific and non-specific processing.

Featural underspecification, not acoustic peripherality, predicts neural mismatch response: evidence from French

Mirjam J.I. de Jonge1, Paul Boersma1; 1University of Amsterdam, Netherlands

Different theories predict different levels of perceived contrast between speech sounds based on acoustic or abstract phonological properties. The peripherality hypothesis (PH, Polka & Bohn, 2003) states that listeners are most sensitive to changes towards the periphery of the vowel space. The Featurally Underspecified Lexicon model (FUL, Lahiri & Reetz, 2002) predicts that listeners are most sensitive to changes towards sounds with sparse phonological representations. In this study we investigated which theory best explains listeners’ neural mismatch responses (MMN, Näätänen, 2001) to vowels contrasting in place of articulation (PoA) and height. We recorded the EEG of 24 righthanded native monolingual French participants in a passive oddball task while they were watching a silent movie. Stimuli were 150 ms long synthesised tokens of the vowels [y, u, o, ø] with formants 1 to 3 based on the average male formant values reported in Calliope (1989). A session consisted of 4 blocks, in each block one of the vowels served as standard (85%) and the other 3 vowels served as deviants (5% each). Mastoid-referenced ERPs were computed at Fz from 100 ms before to 400 ms after stimulus onset and the MMN was computed as the deviant minus standard ERP in the 100-250 ms interval after stimulus onset. MMNs of all stimulus vowels were compared in three vowel contrast conditions: PoA Only, Height Only, or Both. Overall MMN magnitude was larger in the PoA Only and Both conditions compared to Height Only, presumably reflecting the larger acoustic differences in these conditions. Additionally, front vowels [y, ø] elicited larger MMNs overall than back vowels [u, o]. There is no evident acoustic origin for this effect but it is predicted by FUL as a consequence of underspecification for [CORONAL] place, while PH predicts the opposite. In PoA Only contrasts, high deviants [y, u] evoke larger MMNs than high-mid deviants [ø, o] as predicted by PH, but in contrasts where vowel height changes (Height Only and Both contrasts) the larger response is elicited by high-mid deviants, as predicted by FUL. A similar trend is observed for PoA: the larger MMN elicited by front vowels is particularly pronounced in PoA and Both contrasts. The occurrence of phonology-based asymmetries precisely in those conditions where the phonological feature in question plays a role shows that the MMN does not just reflect acoustic stimulus properties but also the linguistic analysis that is made in the listener’s brain. The lack of interactions between stimulus PoA and Height is further confirmation that the effects are not driven by individual vowels but indeed by their abstract phonological properties. Altogether, our results indicate that early auditory processing is affected by top-down influence of phonological context, and that formulating phonological representations in terms of underspecified features predicts MMN patterns better than the Peripherality Hypothesis.

Speech entrainment reveals a crucial role of efference copy for fluent speech production

Julius Fridriksson1, Alexandra Basilakos1, Leonardo Bonilha2, Chris Rorden1; 1University of South Carolina, Columbia, SC, 2Medical University of South Carolina, Charleston, SC

Speech production is guided by an internal motor plan and on-line auditory and proprioceptive feedback (efference copy) that enables error detection and fine-tuning of speech output. Patients with Broca’s aphasia present with impaired speech fluency, characterized as halting speech. Recently, Fridriksson and colleagues (2012) demonstrated that patients with Broca’s aphasia can produce fluent speech by mimicking an audio-visual speech model. They referred to this phenomenon as speech entrainment – the patient’s speech is pulled along by the external model. Successful speech entrainment suggests that the external audio-visual speech model provides patients an alternative efference copy that guides speech production. This study examined localized brain damage that predicts patients’ ability to speak with the aid of speech entrainment. Forty-eight patients with left hemisphere stroke underwent behavioral testing and high-resolution MRI. Behavioral testing: 1. Speech entrainment– Patients attempted to mimic audio-visual speech models that included a speaker producing a one-minute script about a generic topic. Each patient attempted to mimic three separate scripts; 2. Picture description– Patients described pictures including rich visual material. For both the speech entrainment task and the picture description task, the dependent factor was qualified as the number of words produced per minute. Additionally, to determine improvement in speech output with the aid of speech entrainment in comparison to free speech (picture description), standard scores were calculated for the dependent factors in both behavioral tasks. Then, Z-scores for speech entrainment were subtracted from Z-scores for picture description to make a third dependent factor that represented the difference in words per minute produced in each of the two tasks. Each patient underwent MRI (3T) with high-resolution T1 and T2. Lesions were demarcated on T2 images, using the T1 for guidance, in native space. The T1 and T2 images were coregistered and the T1 images were normalized using cost-function masking for the lesion. The T2 images and lesions were yoked into standard space using the T1 transformation matrix. Voxel-wise lesion symptom mapping was accomplished using Non-Parametric Mapping (Rorden et al., 2007) with which independent regression analyses were run for the three dependent measures (permutation thresholding to control for multiple comparisons). Statistically significant results were revealed for the speech entrainment and picture description conditions. Damage to the left middle and posterior portions of the superior temporal sulcus (STS) and the posterior portion of the insula was associated with poor speech entrainment ability. In contrast, impaired speech output in the picture description condition was predicted by damage to the anterior segment of the arcuate fasciculus and the anterior and middle portions of the insula. Damage to the portions of the pars opercularis and pars triangularis was associated with improved speech production with the aid of speech entrainment (uncorrected). Findings suggest that patients who are able to speak more fluently with the aid of speech entrainment have damage to Broca’s area. However, patients whose damage involves the STS and surrounding regions cannot mimic audio-visual speech. These results have crucial implications for understanding the role of internal feedback mechanisms (efference copy) in speech production.

Decoding articulatory features from fMRI responses to speech

Joao Correia1, Bernadette Jansma1, Milene Bonte1; 1Department of Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht University, The Netherlands

Sensorimotor integration, linking the neural systems involved in perceiving and producing speech, is crucial for verbal communication. Sensorimotor integration has been proposed to not only play an important role in acquiring and monitoring speech production but also in speech perception, via the transformation of acoustic/auditory to articulatory/phonetic features. The cortical locus of this transformation remains unclear however. In particular, methodological constraints related to experimental design and analysis methods have so far prevented the disentanglement of neural responses to acoustic versus articulatory features of speech. In this fMRI study, we use multivariate pattern analysis (MVPA) combined with a classification procedure that allows discriminating spoken syllables based on their articulatory features independently of the acoustic properties of the individual syllables. Stimuli consisted of 24 consonant-vowel syllables constructed from eight consonants (/b/, /d/, /f/, /p/, /s/, /t/, /v/, /z/) and three vowels (/a/, /i/, /u/) forming two features per articulatory dimension (place of articulation: labial and alveolar; manner of articulation: stop and fricative; voicing: voiceless and voiced). The three different vowels introduced acoustic variability. Ten Dutch subjects were instructed to attend to the syllables presented in a slow event-related fashion (jittered inter-trial-interval, ITI=12-16 s) during a silent scanning gap (time of repetition, TR=2.0 s; time of acquisition, TA=1.3 s). Functional image acquisition was performed on a Siemens TRIO 3 tesla scanner using a multiband-2 sequence (2mm isotropic voxels). After pre-processing and univariate evaluation of the fMRI data we performed multivariate decoding of the articulatory features of the syllables. Decoding of articulatory features relied on the generalization capability across the different articulatory dimensions. For example, we trained a classifier to discriminate between two places of articulation (labial vs. alveolar) for stop consonants and tested whether this training generalizes to fricatives, i.e. decoding place of articulation independent of manner of articulation. The decoding analysis combined a moving surface-based searchlight procedure that selected cortical points based on their spatial proximity and multivariate classification (support vector machines, SVM). Individual results were group aligned based on cortical curvature information (cortex based alignment, CBA), statistically assessed in random-effects (exact permutation testing, p<0.05) and corrected for multiple comparisons using cluster-size thresholds (alpha=5%). Decoding place of articulation and manner of articulation independent of acoustic variation was successful in the left auditory (superior temporal gyrus) and somatosensory (inferior post central gyrus) regions. Place of articulation was also significantly decoded in the left supramarginal gyrus, whereas voicing was not possible to decode. These results suggest that the representation of spoken syllables during speech perception includes the transformation of acoustic input to articulatory codes at several levels of the speech processing network, including areas within the left temporo-parietal-junction proposed to subserve sensorimotor integration during speech perception.



Slide Session B

Thursday, August 28, 8:30 - 9:50 am, Effectenbeurszaal

Language Evolution and Brain Structure

Chair: Sonja Kotz
Speakers: Benjamin Wilson, Frederic Dick, Uri Hasson, Tristan Davenport

Artificial-grammar learning engages evolutionarily conserved regions of frontal cortex in humans and macaques

Benjamin Wilson1,2, Yukiko Kikuchi1,2, Kenny Smith3, William Marslen-Wilson4, Christopher Petkov1,2; 1Institute of Neuroscience, Henry Wellcome Building, Newcastle University, Framlington Place, Newcastle upon Tyne, NE2 4HH, United Kingdom., 2Centre for Behaviour and Evolution, Henry Wellcome Building, Newcastle University, Framlington Place, Newcastle upon Tyne, NE2 4HH, United Kingdom., 3School of Philosophy, Psychology and Language Sciences, University of Edinburgh, Edinburgh, United Kingdom., 4Department of Psychology, University of Cambridge, Cambridge, United Kingdom.

Understanding the evolutionary origins of the neurobiological systems that underpin human language requires us to distinguish between human-unique neurocognitive processes that support language function and domain-general, evolutionarily-conserved processes that are traceable back to our primate ancestors. The human ability to evaluate syntax—the grammatical relations between words in an utterance—is a uniquely human adaptation. However, a key component of this capacity is the ability to learn how sensory elements are appropriately sequenced. Recent behavioural experiments have shown that a range of nonhuman animals, including nonhuman primates, are able to learn the structure of sequences of stimuli generated by artificial grammars and recognise violations of these structures (e.g., Wilson, et al. 2013; Gentner, et al. 2006; Murphy, et al. 2008). However, to determine whether the cognitive processes involved in these tasks are supported by homologous, functionally-conserved brain areas or by different neurobiological substrates, cross-species neuroimaging studies are required. Here, we combined functional Magnetic Resonance Imaging (fMRI) with an artificial grammar learning task to explore the neural distribution of the processes that support the learning of sequences of auditory syllables in Rhesus macaques and human participants. We used an artificial grammar with a forward-branching structure, designed to model certain aspects of the non-deterministic nature of word transitions in natural language (Saffran, et al., 2008), which we have previously demonstrated that monkeys can learn (Wilson, et al. 2013). Both humans and macaques were initially exposed to exemplary sequences of syllables, consistent with the artificial grammar structure. In two fMRI experiments, adult humans (n = 12) and Rhesus macaques (n = 3) were then presented with sequences that were either consistent with the artificial grammar or which violated the structure. In both humans and macaques, region-of-interest analyses revealed that key regions of ventral frontal and opercular cortex were sensitive to violations of the artificial grammar structure (fMRI activity contrast, ‘violation’ > ‘consistent’), but with no significant lateralisation in either humans or monkeys. These areas lie directly ventral to presumed homologues of Broca’s territory (Brodmann Areas (BA) 44/45) and have been reported to be associated with sequence interpretation in human language processing. BA44/45 itself was statistically involved in the macaques but not in humans, suggesting that these structurally homologous regions in humans and in Rhesus macaques are not fully functionally equivalent. Overall, the results suggest that regions in human ventral frontal and opercular cortex have functional counterparts in the monkey brain, providing a critical step in the development of an animal model in which these processes could be studied at the neuronal level. The study further raises the possibility that certain ventral frontal neural systems, which play a significant role in language function in modern humans, originally evolved to support domain-general cognitive processes related to sequence learning and evaluation.

The relationship of auditory language regionalization to myeloarchitectonically and tonotopically-defined auditory areas: axes of stability and variability within and between subjects.

Frederic Dick1, Marty Sereno1,2, Rachel Kelly1, Mark Kenny-Jones1, Martina Callaghan2; 1Birkbeck College, University of London, 2University College London

There are hundreds of fMRI studies on the regionalization of different auditory language tasks. However, it is often unclear how to mesh these findings with what we know about 'primate-general' auditory functional organization, due to lack of clarity about where and what the human homologues of other primates' auditory areas are. How have different aspects of language function colonized the auditory system? Do the observed functional regionalizations have any clear relationship to known indices of auditory cortex organization? How much of 'higher-level' language function preferentially recruits cortex outside of definable auditory areas? Are these extra-auditory 'language' regions associated with any architectonic characteristics? Finally, how variable is the functional regionalization of language (both within participants and across subjects), relative to the variability in basic auditory cortical organization? To begin to address these questions, we scanned healthy adults in multiple structural and functional sessions. We first mapped auditory cortical areas using combined myeloarchitectonic and tonotopic methods (Dick et al., 2012). Cortical myelination was estimated using high-resolution (512um^3) quantitative multiparameter mapping (MPM) at 3T followed by optimized cortical surface reconstruction techniques with adjustment of myelin maps for effects of local cortical curvature and thickness (Lutti et al., 2014). Repeat MPM data were acquired on a subset of participants to evaluate myelin mapping replicability. High-resolution (343um^3) semi-quantitative T1 data were also acquired at 7T on a subset of participants for the same purpose. Finally, a combination of MPM measures were used to create estimates of cortical vasculature that were used in both myelination and functional analyses. Tonotopic maps were derived from multiple phase-encoded runs of band-pass-swept natural sounds or amplitude-modulated white noise. For all participants, tonotopic data were acquired at 3mm^3 resolution at 1.5T with 32-channel head coil. To assess stability of maps, on a subset of participants, tonotopy runs were acquired at 1.5T using 4x acceleration multi-band EPI at 2.3mm isotropic resolution, and at 7T at 1.2mm isotropic resolution. All functional data were also compared with estimates of cortical vasculature derived from MPMs. All participants underwent an additional functional scanning session with standard auditory language tasks, carried out with high-resolution whole-head multi-band-accelerated EPI at 1.5T with 32-channel head coil. All experiments used block design paradigms in multiple runs. To establish the sentence comprehension network and to map general effects of comprehensibility and intelligibility, we compared passive activation to forward, backward, and foreign language sentences. To map regions involved in sentential/syntactic interpretation, we compared active interpretation of sentences varying in syntactic complexity, presented with and without acoustical degradation. Finally, we mapped regions which were modulated by more phonological versus semantic task demands during sentence listening. We found complex patterns of constancy and variability within and across subjects, with remarkable consistency of anatomical and tonotopic fields, more variability in higher-level functional organization, and overlap of tonotopic maps with sentence-level processing. The findings have implications for theories about language evolution and development.

Caudal-rostral and lateral-medial organization of subregions of the supratemporal plane revealed by cortical-thickness covariation patterns

Uri Hasson1, Pascale Tremblay2, Isabelle Deschamps2; 1University of Trento, 2Université Laval

**Introduction: Cortical thickness (CT) covaries across brain areas in a way that suggests grouping according to common functions (Chen et al., 2008; doi: 10.1093/cercor/bhn003). Applying this concept to the study of supratemporal plane (STP) cortex morphometry, we examined whether distinct factors determine covariation of CT within small-scale sub areas in the STP. Finding a markedly clustered organization, we further showed that whole-brain CT covariation patterns differed for adjacent STP regions. **Methods: Structural images from 33 participants (age range = 18 – 40, median = 24) were collected from two MR centers. Images were processed using FreeSurfer and corrected for topological defects. FreeSurfer's initial 6-region segmentation of STP was manually edited to derive 13 subregions per hemisphere, by further dividing the planum temporale into 3 anterior-posterior sections, transverse temporal sulcus and gyrus into medial/lateral sections, superior temporal gyrus into 3 anterior-posterior sections, and the posterior Sylvian fissure into anterior and posterior portions. For each participant, the meant CT was extracted for each subregion resulting in a 33 [participants] by 26 [regions] matrix. This matrix was analyzed via a hierarchical clustering procedure that utilizes bootstrapping to identify statistically significant clusters. **Results: STP subregions organized into 3 main clusters based on CT covariation patterns (Figure https://db.tt/ourJ2aL0): 1) a 4-region “anterior association” cluster including anterior STG and PP bilaterally, 2) a “posterior association” cluster including, bilaterally, all PT subregions and lateral TTG, and right lateral TTS, and 3) a bilateral “root/core” cluster including medial TTG, TTS, and the SF, with SFp bilaterally and SFa bilaterally forming 2 internal significant sub-clusters. The bilateral arrangement found for all 3 clusters shows that the correlations cannot be attributed to method-artifact such as inaccurate image registration across participants, which would artificially induce CT-covariation between adjacent subregions within but not across hemispheres. Repeating this analysis after partialling out Age from each region’s CT values revealed no significant clusters, suggesting that age plays a central role in this organization. Given this clustering, we examined whether adjacent STP regions also show different structural correlations with areas outside STP, which was indeed the case (Figure https://db.tt/svNAuOuN). Whereas medial TTG was strongly correlated with motor regions, lateral TTS showed correlation with left-hemisphere areas strongly implicated in language comprehension: IFG, SMG, STG, and MTG **Summary and Implications: Our findings indicate that CT covariation patterns partition between STP subregions. Notably, the resulting clusters bear out: 1) The bilateral organization of primary and secondary association cortices of STP; 2) The marked differentiation between regions considered core/root auditory regions (medial TTS and TTG as well as SF) and secondary association regions (PP, PT); and 3) The differentiation between PP and PT, both considered auditory association cortices, but repeatedly implicated in different levels of processing. The whole-brain structural connectivity maps support the notion of sharp boundaries between adjacent STP regions in showing markedly different correlation maps for such regions. In summary, the work shows that CT-correlations can be used to study the organization of STP regions and offers a potential tool for quantifying morphometric organization in different populations.

Effects of childhood language deprivation on picture processing: Insights from adolescent first-language learners

Tristan Davenport1, Naja Ferjan Ramirez2, Matthew Leonard3, Rachel Mayberry1, Eric Halgren1; 1University of California, San Diego, 2University of Washington, 3University of California, San Francisco

When language is acquired in early life, words are processed primarily in the left hemisphere (LH) perisylvian network, while picture processing shows a right hemisphere (RH) bias. This study examines whether the same neural organization patterns for picture vs. words holds when language is not acquired in early life, but instead in adolescence. This study addresses the extent to which the neural basis of a nonlinguistic semantic process, picture processing, is affected by language experience. We investigated picture vs. word processing in two deaf adolescents who, due to their profound deafness from birth and lack of access to special services and other deaf individuals throughout early childhood, began to learn American Sign Language, ASL, at age 14 via immersion. Prior to learning ASL, they had acquired no spoken, written, or signed language and communicated only in limited pantomime (Ferjan Ramirez et al, 2013a). Using anatomically constrained magnetoencephalography (aMEG) (Dale et al, 2000) with a picture-sign priming paradigm, we studied the adolescents’ neural responses to ASL signs after 2.5 years of ASL experience (Time 1; Ferjan Ramirez et al, 2013), and again after 4 years of experience (Time 2; Ferjan Ramirez et al, 2013b). At Time 1, the adolescents exhibited lexico-semantic priming effects in the RH superior parietal, anterior occipital, and dorsolateral prefrontal areas. At Time 2, their brain responses to words they knew well became more concentrated in LH perisylvian cortex. Given that their early deprivation was linguistic in nature, the question is whether their brain responses to pictures would be preserved. MEG recordings taken at Time 1 and Time 2 were analyzed for brain activity related to presentation of the picture primes. During each experimental session, each picture was presented twice (in a novel condition and a repeat condition), allowing us to measure an N400 repetition effect. Event-related fields were calculated by averaging within condition (novel vs. repeated) the changes in magnetic field strength time-locked to picture presentation. Examination of waveforms and source localization maps revealed that in both adolescents, the N400 response to pictures was bilateral. However, in Time 2 relative to Time 1, the N400 repetition effect and the size of the N400 response decreased in LH temporal lobe, but not in RH. This suggests that in the absence of language, pictures serve a symbolic function and thus engage LH perisylvian cortex. At the same time, words may initially be processed in a less symbolic fashion by RH perceptual cortex. Then, as language is learned for the first time in adolescence, language gradually occupies the LH perisylvian network and picture processing is concurrently displaced. References: Ferjan Ramirez, N., Leonard, M., Torres, C., Hatrak, M., Halgren, E., & Mayberry, R. (2013). Neural language processing in adolescent first-language learners. Cerebral Cortex. Doi: 10.1.1093/cercor/bht137. Ferjan Ramirez, N., Leonard, M., Torres, C., Halgren, E., & Mayberry, R. (2013b). Neural language processing in adolescent first-language learners: Longitudinal case studies in American Sign Language. Poster presented at the Neurobiology of Language Conference, San Diego



Slide Session C

Friday, August 29, 8:30 - 9:50 am, Effectenbeurszaal

Combinatorial Processing: Syntax, Semantics, Pragmatics

Chair: Jeff Binder
Speakers: Connor Lane, Monika Mellem, Annika Hultén, Evelina Fedorenko

Sensitivity to syntactic complexity in visual cortex of blind adults.

Connor Lane1, Shipra Kanjlia1, Akira Omaki1, Marina Bedny1; 1Johns Hopkins University

Introduction: It is widely held that left fronto-temporal cortical regions are uniquely adapted for processing language. Recent research suggests that early blind individuals activate "visual" areas of the occipital lobe during verbal tasks. Do these atypical neural responses reflect language processing? We tested two predictions 1) Occipital responses to language are domain specific, relative to symbolic math and memory for sequences. 2) Occipital responses to language are sensitive to syntactic complexity of sentences. Methods: We acquired fMRI data from early blind and sighted adults while they performed two sentence comprehension tasks. In the first task, participants listened to high and low complexity sentences and answered yes/no questions about them. The high complexity sentences involved a long-distance “movement” dependency in the form of an object-extracted relative clause. In a sequence memory control condition, participants heard lists of non-words followed by short probe lists made up of some of the original non-words. Participants had to decide whether the non-words in the probe lists were in the same order as they had been in the target lists. In the second task, participants listened to pairs of sentences and decided whether they had the same meaning. Sentences within a pair always contained identical words. In each pair, one sentence was in active voice, the other in passive voice. In half of the pairs, the agent-patient relations were the same across the two sentences. In the other half, there was an agent-patient role switch. In a control condition, participants heard pairs of spoken equations (e.g. X-5=7) and were asked to decide if the value of “X” was the same in the two equations. Equations were either easy (single digit) or hard (double digit). fMRI data were preprocessed in FSL and Freesurfer using standard analysis procedures. After fitting a first level general linear model, we defined functional ROIs in each subject’s language dominant hemisphere using the sentences > equations contrast from the second task at P<.01. ROIs were defined in inferior frontal cortex (sighted and blind participants) and occipital cortex (blind participants). Parameter estimates for orthogonal contrasts were extracted from each ROI (sentences > non-words; high complexity > low complexity sentences; hard > easy equations). Results: In early blind individuals, regions of occipital cortex 1) responded more to spoken sentences than lists of non-words or math equations, 2) responded more to high complexity than low complexity sentences and 3) were insensitive to math difficulty. We observed a similar response profile in the inferior frontal cortex of sighted (and blind) adults (see also Fedorenko et al., 2011; Ben-Shachar et al., 2003). Conclusion: In early blind individuals, regions of “visual” cortex respond selectively to language and are sensitive to syntactic complexity. We hypothesize that in the absence of vision, language invades cortical territory that is typically devoted to vision. These results illustrate a striking functional flexibility in the human brain during development.

Activity in left anterior temporal cortex is modulated by constituent structure of sentences, but only with social/emotional content

Monika Mellem1, Kyle Jasmin1, Cynthia Peng1, Alex Martin1; 1Lab of Brain and Cognition, NIMH/NIH, Bethesda, MD

Both social/emotional processing (e.g., Simmons et al., 2010; Olsen et al., 2007) and building of words into phrases and sentences (i.e., constituent structure; Pallier et al., 2011) have been found to activate anterior areas of the temporal lobe. Often these studies have examined phrase-building processing using sentences of a social and/or emotional nature. This raises the question of whether these phrase-building effects in anterior temporal lobe reflect domain-general effects (for all types of content) or are preferably for social and/or emotional sentences and phrases. To investigate this question we modulated syntactic complexity and content type in a 3x4 design. Subjects were presented with trials consisting of scrambled words, 3-word constituent phrases, or 6-word sentences of 4 content types: Social-Emotional, Social, Object, and Jabberwocky (designed similarly to Pallier et al., 2011). A trial consisted of six words, each presented for 300 ms (total trial time was 1800 ms): six single words (1-word condition), two 3-word phrases (3-word condition), or one 6-word sentence (6-word condition). Areas sensitive to increasing complexity should show increasing activity across the 1-, 3- and 6-word trials. Stimuli were matched for total word length, frequency, and concreteness across content types and complexity levels. Additionally, trials were created from separate sentences, so phrases and sentences were never repeated. Jabberwocky trials were created from the real word trials with all open class words replaced by pseudowords of the same length and similar morphology. Forty trials of each condition were presented in a fast event-related design optimally randomized and jittered with the program Optseq2. Subjects were told to silently read the words and respond to the occasional trial instructing them to press a button. Data was acquired on a 7 Tesla Siemens scanner with 1.6 mm isotropic voxels and a TR of 2 seconds. After standard preprocessing and general linear modeling in AFNI, a 2-way ANOVA revealed main effects of Content and Complexity as well as their interaction. Preliminary analyses (n = 9) revealed that the left anterior superior temporal sulcus and gyrus showed a main effect of Complexity (p<0.01), and this activity was limited to the Social-Emotional and Social conditions (Content X Complexity interaction; p<0.01). Both main effects also overlapped in the left fusiform gyrus (Content: p<0.05; Complexity: p<0.05). But this area preferred objects (Object > Social-Emotional; p<0.05) and showed a Complexity effect for the Object conditions (Object 6-word > Object 1-word; p<0.09). In contrast, the triangularis portion of left inferior frontal gyrus (LIFG) was not modulated by content and showed only a main effect of Complexity (p<0.01). Thus, whereas LIFG is involved in domain-general syntactic processing, this process is preferentially linked to social and social-emotional stimuli in left anterior temporal cortex. These dissociations challenge the prevailing claims in the field that the anterior temporal lobe is a general phrase-building area. Instead, phrase-building seems to happen within areas that process domain-specific knowledge.

Effects of sentence progression in event-related and rhythmic neural activity measured with MEG

Annika Hultén1,2, Jan-Mathijs Schoffelen1,2, Julia Uddén1,2, Nietzsche Lam1,2, Peter Hagoort1,2; 1Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands, 2Radboud University Nijmegen, Donders Institute for Brain. Cognition and Behaviour. Donders Centre for Cognitive Neuroimaging, Nijmegen, The Netherlands

Natural language operates mostly at the level of sentences and beyond (i.e. discourse) and it is therefore imperative that language is also studied at this level and that the cognitive mechanisms underlying it are understood. One cognitive mechanism that is paramount for sentence processing is the ability to unify incoming words with the sentence context. In order to explore the unification process at the neural level, we used magnetoencephalography (MEG) to track the neural dynamics of sentences as they unfold. We quantified both the event-related and oscillatory responses to visually presented words that were either part of a sentence or a word list (scrambled sentence). Data was acquired from 102 participants and analysed from -0.2 to 1.0 s from the onset of each word. We performed source reconstruction with minimum norm estimation (MNE) for the event-related fields and a frequency domain beamformer for the oscillatory responses. In order to track the effect of sentence progression we modelled the event-related and oscillatory responses as a linear function of the ordinal word position and then contrasted the sentence with the word list condition. In the event-related responses we observed a parametric decrease of amplitude (0.4-0.6 s) as the sentence progressed in the inferior frontal area, the middle and superior temporal cortex as well as in the ventral sensorimotor cortex. In the oscillatory responses, a relative bilateral power decrease was observed both in the theta (5 Hz) and beta (16 Hz) bands in occipital, parietal and posterior temporal regions. In other words, in the event-related and the oscillatory domain, classic language areas react to increasing unification constraints towards the end of the sentence by decreased activation for each incoming word. This is also in line with the prediction of the Memory, Unification and Control model for sentence processing (Hagoort, 2005). In the oscillatory domain, the additional effects in non-classical language regions such as the parietal and superior frontal cortex, suggest involvement of executive and/or attentional functions and/or working memory. These are cognitive functions that are part, but not specific to sentence processing and that by being on-going in nature are also less likely to be strictly time-locked. An incremental power increase was observed in the left anterior temporal cortex in both the beta and theta band as well as in the superior frontal cortex (midline) in the beta band. This effect may be a reflection of the information build-up that occurs towards the end of the sentence. Alternatively processing of the word list condition evokes a type processing which is not present to the same degree in sentences. Overall, the results suggest that event-related and oscillatory activity provide complementary measures of brain activation as they did not capture identical spatial or functional properties of the on-going sentence processing. Importantly, the neural dynamics of sentence unification can with both measures be seen to unfold over the course of a sentence.

The cognitive and neural basis of pragmatic processing: A case study of jokes

Evelina Fedorenko1, Jeanne Gallée2, Zuzanna Balewski3; 1MGH, 2Wellesley College, 3MIT

Neuropsychological investigations of brain-damaged individuals have long suggested that the right hemisphere (RH) plays an important role in processing non-literal aspects of language (e.g., Critchley, 1962; Eisenson, 1962; Zaidel, 1985; Myers, 1999; Lehman Blake, 2005). For example, patients with RH damage experience difficulties with processing conversational inferences (e.g., Kaplan et al., 1990) – including indirect requests (e.g., Weylman et al., 1989; Stemmer, 1994) and commands (e.g., Foldi, 1987) – humor and sarcasm (e.g., Gardner, 1975), and information conveyed by prosody (e.g., Heilman et al., 1975). However, the RH contains several distinct systems that could be contributing to these deficits, including i) the RH homologues of the high-level language processing regions (e.g., Binder et al., 1997); ii) the RH subset of the system that supports social cognition / Theory of Mind (e.g., .Saxe & Kanwisher, 2003); and iii) the RH subset of the domain-general fronto-parietal cognitive control system implicated broadly in goal-directed behavior (e.g., Duncan, 2010). It is therefore difficult to determine which of these RH systems is the key contributor to our pragmatic abilities. This is especially true given that many pragmatic phenomena are complex and thus possibly require some combination of linguistic, social, and generic problem-solving abilities. We here address this question using the functional localization approach in fMRI. In each participant (n=12), we functionally identified three sets of brain regions using independent “localizer” tasks: i) the language regions, ii) the regions that support Theory of Mind (ToM), and iii) the cognitive control regions. We then examined the responses of these sets of regions in the RH to jokes and their literal controls matched for various lexical-level factors known to affect comprehension, as in (1), using the materials from Coulson & Williams (2005). This approach provides a powerful way to probe the relative contributions of these three systems to pragmatic processing, while avoiding the common problem of reverse inference in fMRI (Poldrack, 2006, 2011), and yielding high sensitivity and functional resolution (Nieto-Castañon & Fedorenko, 2012). (1) She went on a fourteen-day diet, but she only lost two weeks/ounces. The ToM regions, including the most functionally selective RH ToM region (rTPJ; Saxe & Powell, 2006), responded more strongly during jokes than during the literal control conditions (ps<0.05). In contrast, although the language and the cognitive control regions responded robustly to both jokes and literal controls relative to a low-level baseline, they did not strongly differentiate between these two conditions: a few language and cognitive control regions showed weak preferences for jokes, but if anything, these preferences were stronger in the LH regions. We therefore conclude that the RH deficits in pragmatic processing as well as the left-visual-hemifield or right-ear processing advantages for non-literal aspects of language (Coulson & Lovett, 2004; Coulson & Williams, 2005; Coulson & Wu, 2005) plausibly arise from damage to the ToM circuits rather than the RH homologues of the language regions or the RH subset of the cognitive control system.



Slide Session D

Friday, August 29, 1:00 – 2:20 pm, Effectenbeurszaal

Lexical Processing and Cognitive Control

Chair: Fred Dick
Speakers: Sara Pillay, Olaf Dimigen, Alexis Hervais-Adelman, Megan Zirnstein

Category-related semantic impairment: A "chronometric" voxel-based lesion-symptom mapping study

Sara Pillay1,2, Colin Humphries1, Diane Book1, William Gross1, Jeffrey Binder1; 1Medical College of Wisconsin, 2RFUMS

Category-related semantic impairments are sometimes observed in patients with focal brain damage, but the underlying mechanisms are not yet understood. Deficits involving knowledge about living things have typically been associated with anterior temporal lobe lesions, and deficits involving tools and other artifacts have been associated with posterior temporal and parietal lesions, but the reliability of these lesion correlations is unknown. Functional imaging studies have yielded inconsistent results except for an association between processing of manipulable object concepts and activation of the left posterior middle temporal gyrus (MTG). We sought to clarify the neural basis of a set of category-related processing impairments using a novel chronometric voxel-based lesion symptom mapping (VLSM) approach in 39 patients with left hemisphere stroke. All patients were right-handed, native English speakers and at least 6 months from stroke onset. Patients performed a word-picture matching task in which they matched a spoken or written word to one of four semantically related pictures taken from the same category. Trials were presented using a computer touch-screen system that recorded reaction time (RT) on each trial. There were 160 trials (80 spoken, 80 written), including 28 Animal, 32 Plant, 24 Body Part, 16 Musical Instrument, 28 Tool (non-musical manipulable objects), and 32 miscellaneous artifact trials. Mean accuracy was 92.7%. Mean RT was computed, for correct trials only, for each category in each patient. VLSMs were conducted for each category using RT for that category as the dependent variable, and mean RT across all other trials as a covariate to control for nonspecific task demands and other sources of between-subject variance. Slowing of RT specific to Plant trials was correlated with damage in the anterior STG and MTG. Slowing specific to Body Part trials was correlated with damage in the angular gyrus. Slowing specific to Musical Instrument trials was correlated with damage to primary sensory-motor cortex in the central sulcus. Slowing specific to Tool trials was correlated with damage to the posterior STG and MTG. No lesion correlates of slowing specific to Animal trials were identified. We hypothesize that the results may reflect differential contributions of embodied sensory and motor knowledge across categories. For example, body part recognition may depend largely on posterior parietal representations of the "body schema," whereas plant recognition depends on visual, olfactory, and gustatory knowledge that converges in the ATL. A surprising result was the correlation between musical instrument recognition and primary sensory-motor cortex integrity, suggesting that recognition of musical instruments may involve low-level motor simulations. RT-based VLSM is a novel and sensitive method for detecting specific processing impairments in the context of relatively high task accuracy.

The impact of parafoveal preprocessing on natural word recognition: Evidence from fixation-related potentials and RSVP with flankers

Olaf Dimigen1, Benthe Kornrumpf1, Florian Niefind1, Michael Dambacher2, Reinhold Kliegl3, Werner Sommer1; 1Humboldt University at Berlin, Germany, 2University of Potsdam, Germany, 3University of Konstanz, Germany

Brain-electric correlates of visual word recognition are traditionally recorded during word-by-word presentation (RSVP), a procedure that eliminates important aspects of the normal reading process. Natural reading not only involves self-paced saccadic eye movements, but allows for the parafoveal preprocessing of not-yet-fixated words. At the behavioral level, these benefits of preview are evident in the form of shorter fixation durations on words that were parafoveally visible (rather than masked) during preceding fixations. Goal of this presentation is to summarize several key results from a 6-year research program in which we have combined EEG recordings with simultaneous high-resolution eye tracking in order to study word recognition under natural conditions with and without parafoveal preview. In all experiments, fixation-related brain potentials (FRPs) were recorded during active, left-to-right reading, while the availability of parafoveal information was manipulated using the gaze-contingent Boundary Technique. All experiments also included a control condition in which the same materials were shown with RSVP (at a reading-like SOA of 280 ms), either with or without the concurrent presentation of parafoveal flanker words. In a first set of experiments, we investigated the timing of N400 word predictability effects in contextually constraining sentences. In a second set of experiments, a simplified list reading paradigm was employed to systematically manipulate the amount and type of parafoveal information, for example by masking a varying number of letters of the upcoming word. We report four basic findings: (1) If parafoveal processing is artificially precluded, topography, size, and time course of N400 effects are strikingly similar in RSVP and natural reading. This similarity also holds for other psycholinguistic EEG effects, such as the effect of a word’s lexical frequency. (2) However, under realistic conditions that include a parafoveal preview, the time course of effects can be dramatically shifted. In particular, centroparietal N400 effects arise no later than 120-160 ms after the first direct fixation of an unpredictable word, indicating a comparatively early access to word meaning. (3) The availability of parafoveal information not only changes the timing of effects, but modulates the brain-electric response to words in absolute terms. In particular, the amplitude of the occipitotemporal N1 component is inversely related to the amount of useful information (number of visible word-initial letters) obtained during the preceding fixation – and therefore markedly attenuated during natural reading with a full preview. (4) Qualitatively similar effects of preview are also obtained when flanker words are presented during RSVP. However, our results suggest that this procedure strongly underestimates the size of preview effects, presumably because RSVP does not require the pre-saccadic attention shift that is part of normal oculomotor preparation. Taken together, our results are compatible with the notion that the parafoveal extraction of sub-lexical features (e.g., partial orthographic priming) facilitates the subsequent recognition of words, allowing for a rapid access to a word’s meaning once it is directly fixated. The fact that most words are already partially processed by the time they enter foveal vision should be considered if findings from EEG studies are generalized to natural reading.

A longitudinal fMRI investigation of simultaneous interpretation training

Alexis Hervais-Adelman1,2, Barbara Moser-Mercer2, Narly Golestani1; 1Brain and Language Lab, Department of Clinical Neurosciences, University of Geneva, 2Department of Interpretation, Faculty of Translation and Interpretation, University of Geneva

Learning to become a simultaneous interpreter depends upon acquiring a high degree of control over language management skills, attention and working memory, in order to enable the interpreter to rapidly produce accurate translation in a variety of contexts. Using fMRI we carried out a longitudinal investigation of functional plasticity arising from simultaneous interpretation training. 22 simultaneous interpretation trainees (10 female, 3 left-handed, mean age: 25 years) and 20 multilingual controls (11 female, mean age: 24 years, 5 left-handed) were scanned at two time-points, 14 months apart. Between scans, trainee interpreters received training in simultaneous interpretation, while controls studied unrelated disciplines. All participants reported a high level of proficiency in a minimum of three languages, including English or French. A sparse fMRI paradigm was employed (TA=2s, TR=9s, 3mm*3mm voxels, 3.6mm interslice gap). Participants were presented with sentences, over headphones, during the quiet 7s intervals between scans. On-screen instructions cued participants to listen passively (“PL”), to shadow (“SH”, simultaneously repeat the sentence) or to interpret (“SI” simultaneously translate the sentence). Condition order was randomized. Stimuli were quartets of thematically-linked sentences in English or French (participants chose their preferred language). Sentences from every quartet were presented sequentially and within the same condition. Each quartet was followed by a null event. In order to reveal training-related changes in brain responses to SI, the univariate contrasts SH – PL and SI – PL were compared pre- and post-training, between the groups. A significant (p(uncorrected)<0.005) time-by-group-by-condition interaction was observed notably in the right caudate nucleus, as well as in the left inferior frontal gyrus (LIFG), left precentral gyrus (LPCG) and left superior cerebellum. Post hoc pairwise tests revealed that effect in the caudate nucleus and in the cerebellum was driven by a decrease in BOLD response during interpretation in the trained but not the control group, suggesting training-related change. However, in LIFG and LPCG, change in BOLD response during SI in trained participants was not significant. We also carried out a multivariate pattern classification analysis. A linear SVM classifier was trained to classify the group (simultaneous interpretation trainees or controls) to which belonged difference maps (Scan 2 – Scan 1) of SI activation. Balanced classification accuracy was 68.9%, significantly above chance (p=0.022), indicating an effect of training on the brain network recruited during SI. The right caudate and left cerebellum exhibit expertise-related functional changes, and are less engaged during simultaneous interpretation after training. This is consistent with existing studies of training-induced functional plasticity, and with the notion that expertise-contingent automatisation of task-performance reduces demands on cerebral resources. Expertise requires the ability to rapidly and dynamically respond to various inputs within the trained domain, and depends on the flexible deployment of acquired action repertoires. The caudate nucleus is a plausible candidate for such a managerial role, being involved in action pattern selection and refinement, in predictive systems and in the control of goal-directed action. Our findings add further weight to the idea that the exercise of multilingual language control shapes brain networks involved in executive control.

The Dual Roles of Cognitive Control and Verbal Fluency in Prediction Generation and Recovery: Evidence from Monolinguals and Bilinguals

Megan Zirnstein1, Janet G. van Hell1,2, Judith F. Kroll1; 1Pennsylvania State University, 2Radboud University, Nijmegen

The ability to accurately predict upcoming information, in particular semantic features of words, has been shown to be highly beneficial for readers, often leading to lower amplitude N400 event-related potential (ERP) effects (Federmeier, 2007; van Berkum 2008). This suggests that successful prediction reduces subsequent processing load, and may therefore free up cognitive resources for other, more demanding tasks. This is especially relevant when considering bilinguals who are attempting to read or communicate via their second language (L2), which has been shown to induce greater cognitive demands than that in the L1 (e.g., Hasegawa et al., 2002). Research has also shown that monolingual readers tend to produce a delayed frontal positivity in response to unexpected but plausible words in contexts where prediction is likely to occur (Federmeier et al., 2007). This effect may be the result of difficulty with inhibiting a previously formed prediction and/or in mediating the conflict between two competing representations (i.e., the prediction and the unexpected word). If this is true, then cognitive control ability (in particular inhibitory control) should predict the magnitude of this effect, possibly more strongly for bilinguals reading in the L2. We therefore tested English monolinguals (Experiment 1) and Chinese-English bilinguals (Experiment 2) in an online sentence reading task while their EEG was recorded. ERPs were time-locked to expected or unexpected target words that were embedded in high or low semantically constraining contexts. Participants also completed a battery of behavioral tasks, including a measure of inhibitory control ability (the AX-CPT; Cohen et al., 1999) and a verbal production fluency task (in both the L1 and L2 for bilinguals; e.g., Luo et al., 2010). Results from Experiment 1 showed that, for monolingual readers, having lower inhibitory control ability resulted in the greatest unexpected word cost (i.e., when unexpected words occurred in highly semantically constraining contexts), resulting in a greater frontal positivity in comparison to expected targets. High control monolinguals, in contrast, showed no such effect. In Experiment 2, these findings were replicated for bilinguals reading in their L2 (English). However, control ability also interacted with L1-L2 verbal production fluency dominance. Bilingual readers who were higher in cognitive control ability showed no difficulty with unexpected target words. Bilinguals who were lower in inhibitory control only produced a frontal positivity if they were relatively high in L1 fluency, whereas those with low L1 and L2 fluency instead produced a significant N400 effect. These data suggest that the repercussions for prediction recovery are evident in both monolinguals and bilinguals, and that these ERP costs may be the result of difficulty with inhibition and/or conflict resolution. In addition, it appears that prediction generation, especially in the L2, may require a high level of production ability, without which individuals rely on more bottom-up strategies when reading for comprehension.