Slide Sessions

Session #

Session Title

Time & Location

Session A

Slide Session A

Wednesday, November 8, 1:10 – 2:30 pm
Chesapeake Ballroom

Session B

Slide Session B

Friday, November 10, 11:20 am – 12:40 pm
Chesapeake Ballroom

Slide Session A

Wednesday, November 8, 1:10 – 2:30 pm, Chesapeake Ballroom

Speakers: Kiefer Forseth, Neal Fox, Esti Blanco-Elorrieta, Lotte Schoot

Predictive Neural Instruments of Early Auditory Cortex

Kiefer Forseth1, Gregory Hickok3, Nitin Tandon1,2; 1Vivan L Smith Department of Neurosurgery, University of Texas Medical School at Houston, Houston, TX, USA, 2Memorial Hermann Hospital, Texas Medical Center, Houston, TX, USA, 3Department of Cognitive Sciences, University of California, Irvine, CA, USA

Neural computations in the brain are not merely a passive, stimulus-driven response – rather, cortical networks could be expected to anticipate patterns of sensory events. Acoustic rhythms contain the requisite information for such the prediction of future events.  While the importance of rhythm in auditory perception seems intuitively clear, the neural mechanisms of auditory entrainment and the interactions of entrained cortex with incoming stimuli are not fully understood.   Intracranial electrodes (n = 3247, 15 patients), implanted as part of a stereotactic electrocorticographic evaluation (sEEG) for epilepsy, furnish the full spectrum of neural oscillations at millimeter spatial and millisecond temporal resolution and provide an ideal methodology by which to study neural prediction of rhythms and of implicit rhythms in human language perception, in early auditory cortex. We used an innovative stimulus with a period of amplitude modulated white noise followed by a period of constant amplitude white noise. In half of the trials and at variable delay, a pure tone was presented in the second period (coincident and partly masked by the white noise). Patients were asked to report the presence of the tone in each trial: a single-interval two-alternative forced-choice task. Analysis of the behavioral data showed a modulation of perceptual accuracy by the phase of the entraining rhythmic stimulus.   Data from depth electrodes placed along the dorsal superior temporal gyrus in these 15 individuals revealed a consistent posterior to anterior gradient of selectivity for distinct elements of the acoustic stimulus: onset, entrainment, and offset. These responses were characterized by consistent alignment of gamma (60 - 120 Hz), beta (15 - 30 Hz), and low frequency (1 - 15 Hz) power relative to the stimulus, as well as rhythmic low frequency phase reset. Interestingly, when we repeated the experiment using non-penetrating high density subdural grid electrodes (n = 3 patients), the ECoG responses were muted, suggesting that these unique features are coded along the depths of the planum temporale.   In a second experiment on the same individuals we found that the quasi-rhythmic amplitude envelope of speech specifically engaged delta, theta, and gamma oscillations – perhaps packaging the acoustic signal into discrete units. These neural representations of rhythm may constitute an adapted computational solution – cascaded neural oscillators – to enable predictive coding and timing. We demonstrate that the descriptive relationship between neural oscillations and the amplitude envelope of rhythmic acoustic signals extends to a predictive relationship that modulates subsequent sensory selectivity. The identification of such specific neural mechanisms that may guide the development of computational models of anticipatory speech processing for use in neural prosthetics.

Transforming continuous temporal cues to a categorical spatial code in human speech cortex

Neal Fox1, Matthias Sjerps1,2,3, Matthew Leonard1, Edward Chang1; 1University of California, San Francisco, 2University of California, Berkeley, 3Radbound University

During speech perception, listeners extract acoustic cues from a continuous sensory signal to map it onto behaviorally relevant phonetic categories. Many such cues are encoded within the fine temporal structure of speech. For example, voice-onset time (VOT), the interval between a stop consonant’s release and the onset of voicing, distinguishes voiced (e.g., /b/, short VOT) from voiceless (e.g., /p/, long VOT) stops in English. Despite the ubiquity of time-dependent cues like VOT in the world’s languages, the neurophysiological mechanisms that allow listeners to distinguish sounds that differ along temporal dimensions remain unclear. To investigate this question, we recorded neural activity directly from the cortex of nine human subjects while they listened to and categorized syllables along a VOT continuum from /ba/ (0ms VOT) to /pa/ (50ms VOT). We found that spatially distinct neural populations respond preferentially to one category (either /b/ or /p/). In both populations, responses are sensitive to VOT differences within the preferred, but not the non-preferred, category. This graded VOT encoding rapidly evolves to reflect the ultimate (categorical) behavioral response function, showing that categorical perception of VOT emerges across time in auditory cortex. Additionally, /b/-selective responses are lagged depending on VOT, while /p/-selective responses are time-locked to the burst, suggesting differential sensitivity to spectral cues indicative of burst vs. voicing. To probe what computations might give rise to these response properties, we implemented a neural network model that simulates neuronal populations as leaky integrators tuned to detect either coincident or temporally-lagged burst and voicing cues. The same temporal dynamics and encoding patterns observed in real neural data emerged in the computational model, suggesting that local tuning for distinct spectral cues at precise lags may underlie temporal cue integration in auditory cortex. Finally, we also recorded neural responses to naturally-produced sentences containing multiple speech sounds differing in VOT (e.g., /d/ vs. /t/, /g/ vs. /k/). Results demonstrated that neuronal tuning for this temporal cue generalized across speech sounds containing different spectral cues. Our results provide direct evidence that continuous temporal information is transformed into a categorical spatial code by discrete, phonetically-tuned neural populations in human auditory cortex.

Turning a language “off” is cognitively effortful, but turning a language “on” is not: MEG evidence from bimodal language switching

Esti Blanco-Elorrieta1,4, Karen Emmorey2, Liina Pylkkanen1,3,4; 1Department of Psychology, New York University New York, NY 10003, USA, 2School of Speech, Language and Hearing Sciences, San Diego State University San Diego, CA 92181, USA, 3Departments of Linguistics, New York University New York, NY 10003, USA, 4NYUAD Institute, Abu Dhabi, United Arab Emirates

Introduction. The ability to switch languages is a unique aspect of bilingualism. While this phenomenon has been the object of much research (e.g, Blanco-Elorrieta & Pylkkänen, 2016; Crinion et al., 2006; Meuter & Allport, 1999), crucial questions regarding the mechanisms of language control have remained unanswered because the bilinguals in these studies used two spoken languages (“unimodal” bilinguals). For these bilinguals, language switching involves suppression of the non-target language (turning “off” a language) while simultaneously activating the target language (turning “on” a language). In this experiment we asked whether these two actions are directed by the same set of control processes or whether there is a fundamental difference between the “off” and “on” procedures involved in switching? Methods. 21 native American Sign Language (ASL) – English bilinguals performed a picture naming language-switching task in which they switched between producing English, ASL, or both languages simultaneously (code-blending) in an unpredictable fashion, following a language cue presented 300 ms before the to-be-named picture. This design allowed us to tease apart the processes involved in turning a language “on” (when going from ASL or English into a code-blend (CB)) or turning a language “off” (when going from a CB to ASL or English). Univariate analyses focused on prefrontal and cingulate cortices (PFC/ACC) previously implicated for language switching (Abutalebi & Green, 2007), left inferior prefrontal cortex (LIPC) previously related to lexical retrieval and language production (Thompson-Schill et al., 1997) and the left temporal lobe, implicated in lexical access. Multivariate decoding analyses aimed at discerning underlying language representations and potential proactive language activation were conducted in sensor-space. Results. Turning a language “off” led to increased engagement of the ACC and PFC, while turning a language “on” did not differ from non-switch trials. This effect was observed ~100 ms after the presentation of the cue and ~100 ms after the to-be-named picture. Decoding analyses accurately classified turning a language on vs. off starting at 110 ms after picture presentation. Activity in the left temporal lobe increased during ASL sign production (either alone or in a code-blend) compared to producing English words alone. Finally, we successfully decoded the to-be-produced language starting 260 ms after language cue presentation (40 before picture presentation). Conclusion. Our results show that it is turning a language “off” and not turning another language “on” that is cognitively effortful. Further, although some flavor of language control has to mediate both processes, their neural underpinnings are distinct and start to diverge ~100 ms after a to-be-named picture is presented. The results from the language decoding analysis show that bilinguals can successfully utilize proactive control to prepare for the upcoming language before lexical retrieval processes start. However, given that our experiment included a condition in which both languages were simultaneously produced, it is unclear to what extent this proactive control is utilized to apply inhibition to the non-target language (Thierry & Wu, 2017). It is also possible that proactive control is used for (re)activation of the target language or to direct attention to the correct lexicon.

Spatiotemporal dissociations for fulfilling and violating predictions at multiple levels of representation: A multimodal approach

Lotte Schoot1,2, Lin Wang1,2, Nate Delaney-Busch1,2, Eddie Wlotko2,3, Edward Alexander1,2, Minjae Kim1,2, Lena Warnke1,2, Arim Choi Perrachione1,2, Sheraz Kahn1, Matti Hamalainen1, Gina Kuperberg1,2; 1Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, USA, 2Department of Psychology, Tufts University, USA, 3Moss Rehabilitation Research Institute, USA

INTRODUCTION: There is growing evidence that we use linguistic context to predict at multiple levels of representation. Using multimodal imaging (ERP, MEG, fMRI), we asked whether and when distinct neuroanatomical networks are engaged to inputs that fulfill or violate strong contextual predictions generated at the level of specific lexical items and/or semantic-thematic structure. METHODS: 32 subjects participated in fMRI and ERP/MEG sessions. They read and judged the acceptability of three-sentence scenarios that varied in their contextual predictability (High Constraint HC, Low Constraint LC) and in whether critical nouns in the third sentence fulfilled or violated contextual predictions and/or the selection restrictions (SRs) of their preceding verbs (examples below). The first two sentences appeared in full; the third sentence appeared word-by-word. Stimuli were counterbalanced across conditions, within and across fMRI and EEG/MEG sessions. RESULTS: (1) ERPs showed selectively reduced activity on the N400 (300-500ms) to predictable critical nouns (HC_pred), relative to all other conditions, reflecting semantic facilitation. In fMRI, all these contrasts revealed reduced activity throughout the left anterior temporal lobe (ATL) and within the left inferior frontal gyrus (IFG). MEG source localization showed that only the left ATL was modulated within the N400 time window. (2) Nouns that violated high constraint contexts (HC_lexviol) selectively evoked an anteriorly-distributed positivity ERP between 600-1000ms, relative to all other conditions. In fMRI, all these contrasts revealed activity not only within the left ATL and IFG, but also within the left-lateralized posterior superior/middle temporal gyrus (S/MTG), premotor cortex (PC) and inferior parietal lobule. MEG source localization within the later time window revealed enhanced activity within the left post-S/MTG and PC. (3) Words that violated the SRs of their preceding verbs (HC_SRviol) selectively evoked a larger posteriorly-distributed positivity/P600 ERP between 600-1000ms, relative to all other conditions. In fMRI, all these contrasts revealed modulation within the left ant-DLPFC and right motor cortex. MEG source localization within the later time window revealed modulation within the left ant-DLPFC. DISCUSSION: These findings provide strong evidence that the brain engages partially distinct networks, distinguished both in their timing and neuroanatomical localization, in response to inputs that fulfill versus violate strong predictions. Incoming words that fulfill strong lexico-semantic predictions are associated with reduced activity within left ATL between 300-500ms. Inputs that violate strong lexical predictions lead to the additional engagement of left IFG and post-STG, perhaps reflecting prolonged attempts to retrieve unpredicted lexico-semantic items, and infer the event dictated by the bottom-up input. Inputs that violate semantic-thematic predictions, however, lead to the engagement of a distinct region – the ant-DLPFC, perhaps reflecting prolonged efforts to relate the properties of the verbs and arguments, thereby inferring novel event structures dictated by the bottom-up input. EXAMPLE STIMULI The lifeguards received a report of sharks near the beach. Their immediate concern was to prevent any incidents in the sea. Hence they cautioned the… swimmers (HC_pred); trainees (HC_lexviol); drawer (HC_SRviol)… Eric and Grant received the news late in the day. They decided it was better to act sooner than later. Hence, they cautioned the…trainees (LC_unpred).

Slide Session B

Friday, November 10, 11:20 am – 12:40 pm, Chesapeake Ballroom

Speakers: Cortney Howard, Benjamin Gagl, Thomas M.H. Hope, Elissa L. Newport

The role of conflict and feedback in action error monitoring and correction: evidence from conduite d’approche

Cortney Howard1, Tamer Soliman1, Louisa Smith2, Laurel Buxbaum1; 1Moss Rehabilitation Research Institute, 2University of Colorado, Boulder

Monitoring and correction of speech errors has traditionally been explained as a function of the perceptual (comprehension) system acting on overt utterances and/or intended meaning (Levelt, 1989). More recently, motor-control-influenced accounts have proposed that monitoring and correction are based on predicted error in the production system; evidence comes from dissociations between comprehension and monitoring abilities in aphasic speakers (e.g., Pickering & Garrod, 2013). Finally, a recent account suggests that conflict between alternative representations provides a signal that monitoring and correction are required (Nozari et al., 2011). The study of conduite d’approche (CD) behavior in aphasia —successive phonological approximations to target utterances—is of both practical and theoretical relevance to this debate. Practically, CD presents an important opportunity to study error correction. Theoretically, successful CD is seen in patients with impaired phonology but intact semantics, indicating that potential responses are constrained by the match between intended and produced meaning. Studying the determinants of error correction in CD may thus shed light on the interplay of meaning systems and production systems. CD in action has also been observed, but not studied experimentally. Recently, we documented higher rates of CD when limb apraxics produced gesture pantomimes to objects associated with several conflicting actions (e.g., calculator) than to objects associated with a single predominant action (e.g., hammer) (Watson & Buxbaum, 2014). The present study extended this work by testing competing predictions derived from theories of monitoring and correction in language. Perceptual self-monitoring accounts suggest that the integrity of action comprehension and the availability of visual feedback should predict CD and successful correction. Production-based monitoring accounts predict no relationship with comprehension, but rather that the integrity of the production system will predict successful correction. The conflict detection account predicts that stimuli eliciting multiple potential responses should increase CD. To test these predictions, 12 left hemisphere stroke survivors pantomimed the use of 40 tools after passively viewing them. Action conflict and visual feedback of the limb were manipulated within-subjects. Separate action recognition and gesture production tasks were also administered. Data were analyzed with mixed-effect logistic regressions. CD was more frequent under high than low conflict conditions (p<.001). Visual feedback increased the probability of CD (p < .05) and successful error correction (p<.001). Successful correction was less frequent in high conflict conditions (p<.001). Gesture production did not predict CD or successful correction (p’s > .2). High action comprehension scores predicted successful correction, but only when visual feedback was available (p < .05). In support of perceptual accounts, patients with relatively intact action comprehension successfully corrected errors when visual feedback was available. Conflict detection accounts were also partially supported: Conditions increasing conflict resulted in more CD, suggesting that the presence of conflict may trigger self-monitoring behaviors in action, as in language. The status of the production system did not predict CD or successful corrections, suggesting potential differences from the patterns observed in the language domain. Of future interest will be parallel examination of self-monitoring across language and action domains in the same patients.

Visual word recognition relies on a sensory prediction error signal

Benjamin Gagl1,2, Jona Sassenhagen1, Sophia Haan1, Fabio Richlan3, Christian J. Fiebach1,2; 1Department of Psychology, Goethe University Frankfurt, Frankfurt am Main, Germany, 2Center for Individual Development and Adaptive Education of Children at Risk (IDeA), Frankfurt am Main, Germany, 3Centre for Cognitive Neuroscience, University of Salzburg, Salzburg, Austria

How do we process visual information in visual word recognition? Here we propose a visual optimization algorithm that "explains away" redundant information on the basis of visual word knowledge. The present model implements, in accordance with the predictive coding theory, a subtraction computation where the sensory information (i.e. visual information of a stimulus) is reduced by a prediction. In the absence of contextual constraints, this prediction is built on our knowledge about the redundancies in a script that arises from experience with that script, and includes those general visual features that are shared by most words (i.e. that contribute little to nothing to the unique identification of words). We propose that during perception of written words, this redundant portion of the visual percept is subtracted from the visual input, thereby substantially reducing the amount of information to be processed to the unique, i.e., non-redundant portion of the percept (i.e., the predictions error). We realized this template-based prediction by estimating a pixel-wise mean from overlaid images that present all words from a lexicon (e.g. SUBTLEX database), and calculated a PE representation for each word by subtracting the mean template from each individual word, thereby achieving an information reduction of ~41%. Interestingly, the PE reflects orthographic familiarity in a pure form, since the PE is associated to orthographic word-characteristics (e.g. Orthographic neighborhood) without a relation to higher-level lexical concepts (e.g. Word frequency). In addition, we found in four behavioral datasets (lexical decision tasks from English, French, Dutch and German; all Ns > 53) that reaction times (RT) were positively associated with PE (faster RTs for words with low PE and vice versa). In an fMRI dataset (silent reading; N=39) we found the same PE effect localized to occipital fusiform gyrus and lateral occipital gyrus in both hemispheres. Using EEG (silent reading; N=31), we observed that the PE predicted the amplitude of the ERP around 170 ms. In a final evaluation we investigated the readability of handwritings and found that handwritings with a low PE were rated more readable. Combined, these findings suggest that in single-word reading, i.e., in the absence of contextual constraint, visual information is optimized by a template-based word-knowledge prediction providing an optimal representation for efficient lexical access.

Predicting language outcomes after stroke: is structural connectomics necessary?

Thomas M.H. Hope1, Alex P. Leff1, Cathy J. Price1; 1University College London

INTRODUCTION: For decades, researchers have sought to understand whether and when stroke survivors with aphasia will recover their speech and language abilities. There is broad agreement that lesion location information should play some role in these predictions, but there is still no consensus on the best or right way to encode lesion-symptom associations. Here, we address the emerging emphasis on the structural connectome in this work – specifically the claim that disrupted connectivity conveys important, unique, prognostic information for stroke survivors with aphasia. METHODS: Our sample included 956 stroke patients extracted from the PLORAS database, which associates structural MRI from stroke patients with language assessment scores from the Comprehensive Aphasia Test (CAT) and basic demographic data. Patients were excluded only when their lesions were too diffuse or small (<1cm3) to be detected by the Automatic Lesion Identification toolbox, which we used to encode their lesions as binary images in standard space. Lesion location was encoded using the 116 cortical regions defined by the Automatic Anatomical Labelling atlas. We examined models driven by both ‘lesion load’ in these regions (i.e. the proportion of each region destroyed by each patient’s lesion), and by the disconnection of the white matter connections between them. The latter was calculated via the Network Modification toolbox, as the mean disconnection implied by each lesion, of structural connectomes defined for 73 neurologically normal controls. Prognostic models were built using Gaussian Process Model regression for the 7 ‘summary language scores’ defined by the CAT, which assess: the comprehension of (a) spoken and (b) written language, (c) auditory repetition, (d) naming, (e) spoken picture description, (f) reading and (g) writing. The models’ predictive performance was assessed using 10-fold cross-validation, and models were compared via Bayes Factors (BF) calculated from Akaike Information Criteria. RESULTS: The connectivity disruption models were able to predict all of the language outcomes: correlations between predicted and empirical scores were all > 0.66 (max. = 0.75, for naming). However, and contrary to past results, models based purely on cortical lesion load were equally good (min. r = 0.64 for writing; max. r = 0.74 for naming; BF < 1.1 and > 0.99 for every model comparison). Using principle components analysis on a composite of the lesion load and connectivity disruption data, we found that every component which, individually, explained at least 1% of the total variance also loaded significantly onto both types of data. This suggests that the two data types convey shared prognostic variance, which explains why we could not distinguish them here. CONCLUSION: Structural connectivity may play a critical role in the neurobiology of language. But this is no guarantee that connectivity disruption data – which is difficult to measure, particularly in the damaged brain – will be clinically useful. Our results demonstrate that these data do convey prognostic information, but also that this information is shared with more traditional variables based on lesion load. When predicting language outcomes after stroke, structural connectivity analyses do not appear to be necessary.

Developmental plasticity and language reorganization after perinatal stroke

Elissa L. Newport1,2, Barbara Landau3, Anna Greenwald1,2, Catherine E. Chambers1, Peter E. Turkeltaub1,2, Alexander W. Dromerick1,2, Madison M. Berl4, Jessica Carpenter4, William D. Gaillard4; 1Georgetown University Medical Center, 2MedStar National Rehabilitation Network, 3Johns Hopkins University, 4Children’s National Medical Network

A prominent theme in the literature on brain injury and recovery has been early developmental plasticity. This has been a particular focus in work on language, but overarching principles and constraints remain unclear. In healthy adults, language is virtually always lateralized to the left hemisphere (LH) (Broca, 1865; Wernicke, 1874). Some researchers have suggested that the LH is privileged for language and that recovery after early brain injury necessarily entails LH perilesional cortex (Fair et al. 2010; Raja et al. 2010; Vargha-Khadem et al. 1985). In contrast, other researchers have suggested that, after perinatal LH stroke, children can develop language in the homotopic regions of the RH (Booth et al. 2000; Gaillard et al. 2007; Lenneberg 1967; Lidzba et al. 2006; Rasmussen & Milner, 1977; Staudt et al, 2002; Stiles et al, 2012). A third hypothesis is that there is enormous flexibility in early development and a wide range of brain areas can take on language after early injury or input alterations (Bates et al 2001; Bedny et al 2011). Here we re-examine these alternatives by assessing the long-term outcomes for language and its neural basis in teenagers who had a perinatal stroke destroying most or all of the normal LH language network, with no other accompanying disease; and, for comparison, their healthy siblings and a matched group of teenagers who had a RH perinatal stroke. We administered to each participant a battery of language tests and 2 fMRI language tasks performed in a 3T scanner. Our main fMRI task asks which brain areas show activation during sentence comprehension (listening to sentences like ‘A large gray animal is an elephant’ compared to the same items played backwards); in healthy children and adults this task activates LH frontal and temporal cortex. A second fMRI task assesses activation for covert naming (ordinarily LH frontal). Behavioral tasks include measures of simple and complex syntax and morphology from the CELF-5, TROG-2, and tasks designed in our lab. Data have been collected from 12 teens who had a LH perinatal stroke to the middle cerebral artery, their healthy siblings, and 8 teens who had a RH perinatal stroke; further testing is ongoing. Participants all show normal levels of performance on all language tasks. While IQ and executive function show deficits after perinatal stroke, core language abilities are intact (including complex syntax and morphology). When listening to sentences or covert naming in the scanner, healthy siblings and those with RH stroke activate traditional LH language regions. Those with LH perinatal stroke show activation in precisely homotopic regions of the RH (RH frontal and temporal cortex). No other pattern of functional reorganization appears. Our results show that there are strong and specific constraints on developmental plasticity for language; in particular, that RH areas homotopic to the normal LH language network are capable of supporting language after very early stroke. We suggest that the neural distribution of language in early development, which is more bilateral than language in adults, may underlie and support language reorganization after stroke.