Presentations

Abstract

One of the most daunting tasks of a listener is to map a continuous auditory stream onto known speech sound categories and lexical items. A major issue with this mapping problem is the variability in the acoustic realizations of sound categories, both within and across speakers. Past research has suggested listeners may use various sources of information, such as lexical knowledge or visual cues (e.g., lip-reading) to recalibrate these speech categories to the current speaker. Previous studies have focused on audiovisual recalibration of consonant categories. The present study explores whether vowel categorization, which is known to show less sharply defined category boundaries, also benefit from visual cues.
Participants were exposed to videos of a speaker pronouncing one out of two vowels (Dutch vowels /e/ and /ø/), paired with audio that was ambiguous between the two vowels. The most ambiguous vowel token was determined on an individual basis by a categorization task at the beginning of the experiment. In one group of participants, this auditory token was paired with a video of an /e/ articulation, in the other group with an /ø/ video. After exposure to these videos, it was found in an audio-only categorization task that participants had adapted their categorization behavior as a function of the video exposure. The group that was exposed to /e/ videos showed a reduction of /ø/ classifications, suggesting they had recalibrated their vowel categories based on the available visual information. These results show that listeners indeed use visual information to recalibrate vowel categories, which is in line with previous work on audiovisual recalibration in consonant categories, and lexically-guided recalibration in both vowels and consonants.
In addition, a secondary aim of the current study was to explore individual variability in audiovisual recalibration. Phoneme categories vary not only in terms of boundary location, but also in terms of boundary sharpness, or how strictly categories are distinguished. The present study explores whether this sharpness is associated with the amount of audiovisual recalibration. The results tentatively support that a fuzzy boundary is associated with stronger recalibration, suggesting that listeners’ category sharpness may be related to the weight they assign to visual information in audiovisual speech perception. If listeners with fuzzy boundaries assign more weight to visual cues, given that vowel categories have less sharp boundaries than consonants, there ought to be audiovisual recalibration for vowels as well. This is exactly what was found in the current study.

Abstract

In line with two-stage models of memory, it has been proposed that memory traces for newly learned words are
initially dependent on medial temporal structures and acquire neocortical, more lexical representations during the first
night’s sleep after training (Davis & Gaskell, 2009). Only after sleep-dependent consolidation are novel words fully
integrated into the lexicon and are therefore able to enter into lexical competition with phonologically overlapping
existing words. This effect, observable as a slowing down of responses to existing words with a novel competitor, has
been demonstrated using various tasks including lexical decision, pause detection, semantic judgement, and wordspotting.

Abstract

Declarative memory appears to involve two separate systems, with more episodically oriented memories coded in a hippocampal network, and more non-episodic or semantic memories coded in a neocortical network. Previous works (e.g. Dumay & Gaskell, 2007) have shown a role of sleep in the lexicalization of novel words. In line with the two-stage model of memory proposed by McClelland and colleagues (1995), the memory traces for novel words are initially dependent on hippocampal structures. However, a shift towards neocortical representations occurs during the first night’s sleep after training. This shift, or integration of newly learned words into the lexicon (lexicalization) can be observed behaviourally as lexical competition, where novel words slow down recognition of phonologically overlapping known words. To extend understanding of how newly learned words are incorporated into the semantic system, we conducted an fMRI study to elucidate the neural processes underlying sleep-dependent lexicalization, with the additional aim of investigating multimodal information integration in word learning. As a first step towards studying the acquisition of multimodal word meanings, we familiarized subjects with the phonological form of 40 novel words, of which 20 were associated with pictures of novel objects (“picture-associated words”) and 20 were not (“form-only words”). Immediately after training (Day1) and on the following day (Day2), we recorded the BOLD response to auditorily presented “trained novel words”,” untrained novel words” and “existing words”, and administered a lexical competition task to test the effect of novel words on phonologically overlapping existing words. Behavioural data showed enhanced performance in recognition and recall of novel words after sleep, with a greater benefit for picture-associated words. However, lexical competition on Day2 was greater for the form-only words. The fMRI data showed more involvement of the hippocampal network for picture-associated words than for form-only words. In contrast, form-only words activated the semantic memory network already on Day1, whereas this was more apparent on Day2 for picture-associated words. This implies that the consolidation/lexicalization process differs depending on the degree of involvement of the two memory systems, with a greater involvement of the hippocampal system for picture-associated words. Stronger episodic memory traces might slow down the overnight shift of the novel picture-associated words to the lexical network relative to the faster integration into this network of the form-only words.