Presentations

Abstract

One of the main challenges during online comprehension is relating an utterance to the wider communicative context, for example in terms of who or what an utterance refers to or whether the utterance is true or informative. In my talk, I will give a brief overview of my work, hoping to elucidate what can be learned from electrophysiology about the sources of information that guide incremental pragmatic comprehension

Abstract

The ability to use linguistic representations to refer to the world is a vital mechanism that gives human language its communicative power. In particular, the anaphoric use of words to refer to previously mentioned concepts (antecedents) is what allows dialogue to be coherent and meaningful. Psycholinguistic theory posits that anaphor comprehension involves reactivating an episodic memory representation of the antecedent [1-2]. Whereas this implies the involvement of memory structures, the neural processes for reference resolution are largely unknown. Here, we report time-frequency analysis of four EEG experiments [3-6], revealing the increased coupling of functional neural systems associated with coherent referring expressions compared to referentially ambiguous expressions. We performed time-frequency analysis on data from four experiments in which referentially ambiguous expressions elicited a sustained negativity in the ERP waveform compared to coherent expressions. In Experiment 1, 32 participants read 120 correct Dutch sentences with coherent or ambiguous pronouns. In Experiment 2, 31 participants listened to 90 naturally spoken Dutch mini-stories containing coherent or ambiguous NP anaphora. In Experiment 3, 22 participants each read 60 Spanish sentences with a coherent or ambiguous ellipsis determiner. In Experiment 4, 19 participants each read 180 grammatically correct English sentences containing coherent or ambiguous pronouns. Analysis was performed with Fieldtrip [7], separately for low frequency (2-30 Hz) and high frequency (25-90 Hz) activity. Power-changes per trial were computed as a relative change from a pre-CW baseline interval, average power changes were computed per subject for coherent and ambiguous conditions separately. Statistical tests used cluster-based random permutation [8]. Despite varying in modality, language and type of expression, all experiments showed larger gamma-band power around 80 Hz for coherence compared to ambiguity, within a similar time range. No differences were observed in low frequencies. In high-density EEG Experiment 4, an additional short-duration gamma-increase was observed around 40 Hz, around 300-500 ms after pronoun-onset, which was localised using Beamformer analysis [9] to left posterior parietal cortex (PPC). The 80 Hz power increase around 600-1200 ms after word onset was localised to left inferior frontal-temporal cortex. We argue that the observed gamma-band power increases reflect successful referential binding and resolution, linking incoming information to previously encountered concepts and integrates that information into the unfolding discourse representation. Specifically, we argue that this involves antecedent reactivation in the PPC episodic memory network [10-11], interacting with unification processes in the frontal-temporal language network [12]. Based on these results, and on results of patient [13] and fMRI [14] research on pronoun comprehension, we propose an initial neurobiological account of reference, by bridging the psycholinguistics of anaphora with the neurobiology of language and of episodic memory. [1] Dell et al., 1983 [2] Gerrig & McKoon, 1998 [3] Nieuwland & Van Berkum, 2006 [4] Nieuwland et al., 2007a [5] Martin et al., 2012 [6] Nieuwland, 2014 [7] Oostenveld et al., 2011 [8] Maris & Oostenveld, 2007 [9] Gross et al., 2001 [10] Shannon & Buckner, 2004 [11] Wager et al., 2005 [12] Hagoort & Indefrey, 2014 [13] Kurczek et al., 2013 [14] Nieuwland et al., 2007b

Abstract

The ability to use words to refer to the world is a vital mechanism that gives human language its communicative power. In particular, the use of words to refer to previously mentioned concepts (anaphora) is what allows dialogue to be coherent and meaningful. Psycholinguistic theory posits that anaphor comprehension involves reactivating a memory representation of the antecedent. Whereas this implies the involvement of episodic memory, the neural processes for reference resolution are largely unknown. Here, we report time-frequency analysis of four EEG experiments to reveal the increased coupling of functional neural systems associated with referring expressions that can be straightforwardly understood compared to those that cannot (referential coherence or ambiguity). Despite varying in modality, language and type of referential expression, all experiments showed larger gamma-band power for coherence compared to ambiguity. In high-density EEG Experiment 4, Beamformer analysis localised this increase to the posterior parietal cortex around 300-500 ms after onset of the anaphor and to frontal-temporal cortex around 500-1000 ms. We argue that the observed gamma-band power increases reflect successful referential binding and resolution, which links incoming information to previously encountered concepts through an interaction between the episodic memory network and the frontal-temporal language network.

Abstract

Current accounts of sentence comprehension invoke the notion of retrieval interference as a primary determinant of difficulty during processing [1-2]. Specifically, similarity between constituents (e.g., NP feature-overlap) has been argued to interfere when people resolve subject-verb or anaphoric dependencies [3-7]. We ask whether similarity-based interference effects arise as a function of multiple NPs in the discourse that overlap in gender and/or number. We take a novel approach by examining interference effects at the second NP rather than downstream after “maintaining” multiple NPs [6- 8], using ERPs to establish quantitative and qualitative processing consequences. We used the empty category PRO to introduce two NPs, only the second NP could be PRO controller (e.g., “While [PRO] talking to the waitresses, the man/men/woman/women inspected the menu”). If feature overlap affects processing of the second NP, most interference should occur under gender- and number-matching NPs. Because this interference crosses the subject-object distinction, we predicted that interference would elicit a P600 effect, the effect most reliably associated with syntactic processing difficulties [9]. Methods: During EEG recording, 24 participants read 160 grammatical sentences (40 per condition) in a 2(gender: match, mismatch) x 2(number: match, mismatch) factorial design where the first clause introduced the object-NP and had PRO as subject, and the matrix clause introduced the controller of PRO. Subject and object NPs could overlap in gender and/ or number. We fully counterbalanced 160 male/female singular/plural gender-definitional nouns as object NPs, and as critical NP always ‘woman/man/girl/boy’ (or plural form). Sentences were mixed with 156 fillers and presented word by word (300 ms duration, 200 ms blank), followed by intermittent comprehension questions (85% response accuracy). Results: Across all electrodes, a significant gender by number interaction was observed (500-800 ms window [9]; F(1,23)=6.02, p<.05), due to a robust P600 effect of number-mismatch in the gender- match conditions (M=-1.18, F(1,23)=8.04, p=.01), that did not occur in the gender-mismatch conditions (M=-.17, F(1,23)=.18, ns). No distributional effects were observed. Conclusions: The P600 effect for double-match NPs suggests that interference was driven by similarity contingent upon matching gender and number. Our results testify to the strength of gender-cues during incremental processing, consistent with memory-based accounts of discourse comprehension [2-7]. The results suggest that when features maximally overlap, the subject NP may be momentarily considered as an anaphor for the more distinctive (i.e., first-mentioned and semantically richer) object NP. Alternatively, the P600 may reflect increased discourse complexity stemming from similar NPs [10]. Our results imply a central role for interference during comprehension, even of simple grammatical sentences. References: [1] Lewis, Vasishth, Van Dyke, 2006; [2] McElree, Foraker, & Dyer, 2003; [3] Gerrig & O’Brien, 2005; [4] Gordon, Hendrick, & Johnson, 2004; [5] McKoon & Ratcliff, 1998; [6] Van Dyke & McElree, 2006; [7] Gordon, Hendrick, Johnson, & Lee, 2006; [8] Wager & Phillips, 2013; [9] Osterhout & Holcomb, 1992; [10] Kaan & Swaab, 2003

Abstract

Current accounts of sentence comprehension invoke the notion of retrieval interference as a primary determinant of difficulty during processing [1-2]. Specifically, similarity between constituents (e.g., NP feature-overlap) has been argued to interfere when people resolve subject-verb or anaphoric dependencies [3-7]. We ask whether similarity-based interference effects arise as a function of multiple NPs in the discourse that overlap in gender and/or number. We take a novel approach by examining interference effects at the second NP rather than downstream after “maintaining” multiple NPs [6-8], using ERPs to establish quantitative and qualitative processing consequences. We used the empty category PRO to introduce two NPs, only the second NP could be PRO controller (e.g., “While [PRO] talking to the waitresses, the man/men/woman/women inspected the menu”). If feature overlap affects processing of the second NP, most interference should occur under gender- and number-matching NPs. Because this interference crosses the subject-object distinction, we predicted that interference would elicit a P600 effect, the effect most reliably associated with syntactic processing difficulties [9]. Methods: During EEG recording, 24 participants read 160 grammatical sentences (40 per condition) in a 2(gender: match, mismatch) x 2(number: match, mismatch) factorial design where the first clause introduced the object-NP and had PRO as subject, and the matrix clause introduced the controller of PRO. Subject and object NPs could overlap in gender and/or number. We fully counterbalanced 160 male/female singular/plural gender-definitional nouns as object NPs, and as critical NP always ‘woman/man/girl/boy’ (or plural form). Sentences were mixed with 156 fillers and presented word by word (300 ms duration, 200 ms blank), followed by intermittent comprehension questions (85% response accuracy). Results: Across all electrodes, a significant gender by number interaction was observed (500-800 ms window [9]; F(1,23)=6.02, p<.05), due to a robust P600 effect of number-mismatch in the gender-match conditions (M=-1.18, F(1,23)=8.04, p=.01), that did not occur in the gender-mismatch conditions (M=-.17, F(1,23)=.18, ns). No distributional effects were observed. Conclusions: The P600 effect for double-match NPs suggests that interference was driven by similarity contingent upon matching gender and number. Our results testify to the strength of gender-cues during incremental processing, consistent with memory-based accounts of discourse comprehension [2-7]. The results suggest that when features maximally overlap, the subject NP may be momentarily considered as an anaphor for the more distinctive (i.e., first-mentioned and semantically richer) object NP. Alternatively, the P600 may reflect increased discourse complexity stemming from similar NPs [10]. Our results imply a central role for interference during comprehension, even of simple grammatical sentences. References: [1] Lewis, Vasishth, Van Dyke, 2006; [2] McElree, Foraker, & Dyer, 2003; [3] Gerrig & O’Brien, 2005; [4] Gordon, Hendrick, & Johnson, 2004; [5] McKoon & Ratcliff, 1998; [6] Van Dyke & McElree, 2006; [7] Gordon, Hendrick, Johnson, & Lee, 2006; [8] Wager & Phillips, 2013; [9] Osterhout & Holcomb, 1992; [10] Kaan & Swaab, 2003

Abstract

Research on subject-verb agreement during comprehension suggests a ‘grammaticality asymmetry’ in similarity-based retrieval interference. Whereas processing costs incurred by ungrammatical subject- verb agreement are reduced in the presence of a grammatically illicit attractor noun that matches the verb in number, attractor nouns have not been found to affect the processing of grammatical sentences [1]. However, most existing studies have only included singular verbs in the grammatical conditions, and the lack of retrieval interference in such cases could be a result of the fact that singular is an unmarked feature [2]. In the current study, we tested for similarity-based interference for both singular and plural verbs in fully grammatical sentences. If plural is a marked feature, we expect to find evidence of retrieval interference for plural verbs but not singular verbs when multiple items in memory match the number of the verb. We predicted that retrieval interference would elicit a P600 effect [3-4], the effect commonly associated with syntactic processing difficulties. Methods: Participants read 120 grammatical sentences (30 per condition) belonging to a 2(subject noun: plural, singular) x 2(attractor noun: plural, singular) factorial design in which the critical verb (have/had/were/was) always agreed in number with the subject noun. PS: “The keys to the cabinet were getting very rusty”, PP: “The keys to the cabinets were getting very rusty”, SS: “The key to the cabinet was getting very rusty”, SP: “The key to the cabinets was getting very rusty”. Sentences were mixed with 280 fillers and presented word by word (300 ms duration, 200 ms blank). Intermittent yes/no comprehension questions were answered with 92% accuracy. EEG data was recorded from sixty-four channels and segmented into epochs from 200 ms before to 1000 ms verb onset. Data was baselined to 0-200 ms post-stimulus to eliminate spurious effects from pre-critical word differences (see also [3-4]). Results: Using average amplitude per condition across 16 centrally distributed EEG electrodes, repeated measures ANOVAs in the 500-700 ms time window showed an effect of attractor that was reliably different for plural and singular verbs (F(1,35)=4.8, p<.05), with a robust P600 effect elicited by plural verbs (PP minus PS voltage difference, M=.64, F(1,35) = 5.7, p < .05) but none for singular verbs (SS minus SP, M= -.22, F(1,35)=.44, ns). Conclusions: The observed P600 effect for grammatically correct, plural verbs in context of a plural attractor noun suggests that retrieval interference arises as a by-product of grammatical processing, and constitutes evidence against a grammaticality asymmetry in interference effects. References: [1] Wagers, M. W., Lau, E. F., & Phillips, C. (2009). Journal of Memory and Language [2] Bock, K., & Eberhard, K. M. (1993). Language and Cognitive Processes [3] Kaan, E. (2002). Journal of Psycholinguistic Research [4] Tanner, D., Nicol, J., Herschensohn, J., & Osterhout, L. (2012). Proceedings of the 36th Annual Boston University Conference on Language Development (pp. 594-606).

Abstract

Differences in native and nonnative sentence processing may surface most clearly around parameters that are not shared between L1 and L2. We investigated whether differences between Spanish-Basque bilinguals exist in processing related to the particular constraints of the ergative-absolutive case system of Basque, which is not present in Spanish, but not in processing related to number agreement which occurs in both languages. In an event-related FMRI experiment, we tested this hypothesis by examining the cortical networks recruited for reading in Spanish-Basque bilinguals. Highly proficient nonnative and native speakers of Basque read sentences containing violations of ergative case assignment or violations of number agreement as well as correct sentences (e.g., “Gizonak lehiatilan jaso ditu sarrerek/sarrera/sarrerak goizean”, respectively, approximate translation: “The man at the box office has received the tickets-erg/ticket/tickets in the morning”) while performing an acceptability judgment task. Preliminary results (6 nonnative and 16 native speakers) showed that ergative case violations and number violations similarly elicited activation increases compared to correct sentences in the right inferior parietal lobule and the precuneus while number violations elicited additional activation increases in middle and inferior frontal cortex, consistent with reports for morphosyntactic agreement errors. Compared to native speakers, nonnative speakers engaged the medial prefrontal cortex more strongly while processing ergative case violations and number violations, suggesting that they engaged additional cognitive resources to arrive at the same behavioral outcome. These latter effects, however, did not seem to differ between the ergative case and number violations. Thus, our preliminary results support the hypothesis that while morphosyntactic processing is quantitatively different in the two groups, native and nonnative speakers do not show qualitatively different responses when processing morphosyntactic features that are specific of the L2.

Abstract

In an event-related fMRI study, we investigated to what extent semantic and
referential aspects of language comprehension recruit common or dis-
tinct neural ensembles. We compared BOLD responses to sentences containing semantically anomalous or coherent words, and to sentences containing referentially ambiguous pronouns (e.g., “Ronald told Frank that he...”), referentially failing pronouns (e.g., “Rose told Emily that he...”) or coherent pronouns. Semantic anomaly elicited activation increases in lateral prefrontal brain regions associated with semantic pro-
cessing. Referential failure elicited
activation increases in brain regions
associated with morphosyntactic processing, and additional activations
associated with elaborative inferenc
ing if readers took failing pronouns
to refer to unmentioned entities. Referential ambiguity selectively
recruited medial prefrontal regions,
suggesting that readers engaged in
problem-solving to select a unique
referent from the discourse model.
Furthermore, our results showed that semantic anomaly and referential
ambiguity recruit overlapping neural ensembles in opposite directions,
possibly reflecting the dynamic re
cruitment of semantic and episodic
processing to resolve semantically or referentially problematic situations.
These findings suggest that neurocogni
tive accounts of language compre-
hension will have to address not just how we parse a sentence and com-
bine individual word meanings, bu
t also how we determine who’s who
and what’s what during sentence and discourse comprehension

Abstract

In this event-related brain potentials (E RP) study, we explored the possibility to selectively track referential ambiguity during spoken discourse comprehension. Earlier ERP research has shown that referentially am biguous nouns (e.g., “the girl” in a two-girl context) elicit a frontal, sustained negative sh ift relative to unambiguous control words. In the current study, we examined whether this ERP effect reflec ts ‘deep’ situation model ambiguity or ‘superficial’ textbase am biguity. We contrast ed these different interpretations by investigating whether a discourse-level semantic manipulation that prevents referential ambiguity also averts t he elicitation of a referentially induced ERP effect. We compared ERPs el icited by nouns that were re ferentially non-ambiguous but were associated with two discourse entities (e .g., “the girl” with two girls introduced in the context, but one of which has died or le ft the scene), with referentially ambiguous and non-ambiguous control words. While temporally referentially ambiguous nouns elicited a frontal negative shift compared to control words, the ‘double bo und’ but referentially non-ambiguous nouns did not. These results suggest that it is possible to selectively track referential ambiguity with ERPs at the level that is most relev ant to discourse comprehension, the situation model

Abstract

In general, language comprehension is surprisingly reliable. Listeners very rapidly extract meaning from the unfolding speech signal, on a word-by-word basis, and usually successfully. Research on ‘semantic illusions’ however suggests that under certain conditions, people fail to notice that the linguistic input simply doesn’t make sense. In the current event-related brain potentials (ERP) study we examined whether listeners would spontaneously detect an anomaly in which a human character central to the story at hand (e.g. “a tourist”) was suddenly replaced by an inanimate object (e.g. “a suitcase”). Because this replacement introduced a very powerful coherence break, we expected listeners to immediately notice the anomaly and generate the standard ERP effect associated with incoherent language, the N400 effect. However, instead of the standard N400 effect, anomalous words elicited a differential ERP effect from about 500-700 ms onwards. The absence of an N400 effect indicates that subjects did not immediately notice the anomaly, and that for a few hundred milliseconds the comprehension system has converged on an apparently coherent but factually incorrect interpretation. The presence of the later ERP effect indicates that subjects were processing for comprehension and did ultimately detect the anomaly. Therefore, we take our results to reflect a temporary semantic illusion. Our results also show that even attentive listeners sometimes fail to notice a radical change in the nature of a story character, and therefore demonstrate a case of short-lived ‘change deafness’ in language comprehension.

Abstract

n two ERP-experiments we examined whether discourse context could overrule a local semantic violation. In both experiments, subjects listened to stories in which a person was engaged in conversation with an inanimate object. In experiment 1, story-initial animacy violations reflected in an N400 effect were completely neutralized further down the story. In experiment 2, canonical but story-irrelevant inanimate predicates assigned to the inanimate object elicited an N400 effect, compared to contextually appropriate animate predicates.