Publications

Abstract

Semantic knowledge is a defining property of human cognition, profoundly influenced by cultural experiences. In this study, we investigated whether literacy enhances lexical-semantic processing independently of schooling. Three groups of neurotypical adults - unschooled illiterates, unschooled ex-illiterates, and schooled literates - from the same residential and socioeconomic background in Portugal were tested on serial rapid automatized naming (RAN) and on discrete naming of everyday objects (concrete concepts) and basic color patches (abstract concepts). The performance of readers, whether schooled literate or unschooled exilliterate, was not affected by stimulus category, whereas illiterates were much slower on color than object naming, irrespective of task. This naming advantage promoted by literacy was not significantly mediated by vocabulary size. We conclude that literacy per se, regardless of schooling, contributes to faster naming of depicted concepts, particularly those of more abstract categories. Our findings provide further evidence that literacy influences cognition beyond the mere accumulation of knowledge: Literacy enhances the quality and efficiency of lexical-semantic representations and processing.

Abstract

We introduce a new method for measuring prediction and language-vision interactions: tracking the trajectories of hand-reaching movements in Virtual Reality (VR) environments. Spatiotemporal trajectory tracking of hand-reaching movements in VR offers an ecologically valid yet controlled medium for conducting experiments in an environment that mirrors characteristics of real-world behaviors. Importantly, it enables tracking the continuous dynamics of processing on a single-trial level. In an exploratory experiment L2 speakers heard predictive or non-predictive sentences (e.g., “The barber cuts the hair” vs. “The coach remembers the hair”). Participants’ task was to move their hands as quickly and as accurately as possible towards the object most relevant to the sentence. We measured reaction times (RTs) and hand-reaching trajectories as indicators of predictive behavior. There was a main effect of predictability: Predictable items were touched faster than unpredictable ones. Importantly, uncertainty was captured using spatiotemporal survival analysis by prolonged fluctuations in upward and downward vertical hand movements before making a final move to target or distractor. Self-correction of prediction errors was revealed by participants switching the direction of hand-reaching movements mid-trial. We conclude that the Virtual Hand Paradigm enables measuring the onset and dynamics of predictive behavior in real time in single and averaged trial data and captures (un)certainty about target objects and the self-correction of prediction error online in ‘close to real-world’ experimental settings. The new method has great potential to provide additional insights about time-course and intermediate states of processing, provisional interpretations and partial target commitments that go beyond other state-of-the-art methods.

Abstract

We examined the morphosyntactic prediction ability of child heritage speakers and the role of reading skills and language experience in predictive processing. Using visual world eye-tracking, we focused on predictive use of case-marking cues in Turkish with monolingual (N=49, Mage=83 months) and heritage children, who were early bilinguals of Turkish and Dutch (N=30, Mage=90 months). We found quantitative differences in magnitude of the prediction ability of monolingual and heritage children; however, their overall prediction ability was on par. The heritage speakers’ prediction ability was facilitated by their reading skills in Dutch, but not in Turkish as well as by their heritage language exposure, but not by engagement in literacy activities. These findings emphasize the facilitatory role of reading skills and spoken language experience in predictive processing. This study is the first to show that in a developing bilingual mind, effects of reading-on-prediction can take place across modalities and across languages.

Abstract

Dealing with text is crucial in modern societies. However, not everyone acquires sufficient literacy skills during school education. This systematic review summarizes and synthesizes research on adults with low literacy skills (ALLS) in alphabetic writing systems, includes results from behavioral and neurobiological studies, and compares these findings with those on developmental dyslexia given that this developmental disorder is one possible explanation for low literacy skills in adulthood. Twenty-seven studies focusing on the cognitive profile of ALLS met the three predefined criteria of reading level, age, and education. Results showed that ALLS performed worse than literate adults in various tasks at skill and information processing level, and exhibited structural and functional differences at the neurobiological level. The cognitive profile of ALLS was closer to that of primary school children than of literate adults. However, relative to children, ALLS’ literacy skills relied less on phonological and more on orthographic strategies. A narrative comparison of results with meta-analyses on developmental dyslexia showed large, though not complete, overlap in the cognitive profiles. The present results helps to better understand the literacy skills and reading-related cognitive functions of ALLS and may support the development of tailored interventions directed to the specific cognitive difficulties ALLS have.

Abstract

Contemporary accounts of anticipatory language processing assume that individuals predict upcoming information at multiple levels of representation. Research investigating language-mediated anticipatory eye gaze typically assumes that linguistic input restricts the domain of subsequent reference (visual target objects). Here, we explored the converse case: Can visual input restrict the dynamics of anticipatory language processing? To this end, we recorded participants’ eye movements as they listened to sentences in which an object was predictable based on the verb’s selectional restrictions (“The man peels a banana”). While listening, participants looked at different types of displays: The target object (banana) was either present or it was absent. On target-absent trials, the displays featured objects that had a similar visual shape as the target object (canoe) or objects that were semantically related to the concepts invoked by the target (monkey). Each trial was presented in a long preview version, where participants saw the displays for approximately 1.78 seconds before the verb was heard (pre-verb condition), and a short preview version, where participants saw the display approximately 1 second after the verb had been heard (post-verb condition), 750 ms prior to the spoken target onset. Participants anticipated the target objects in both conditions. Importantly, robust evidence for predictive looks to objects related to the (absent) target objects in visual shape and semantics was found in the post-verb but not in the pre-verb condition. These results suggest that visual information can restrict language-mediated anticipatory gaze and delineate theoretical accounts of predictive processing in the visual world.

Abstract

Previous studies have shown that during comprehension readers activate words beyond the unfolding sentence. An open question concerns the mechanisms underlying this behavior. One proposal is that readers mentally simulate the described event and activate related words that might be referred to as the discourse further unfolds. Another proposal is that activation between words spreads in an automatic, associative fashion. The empirical support for these proposals is mixed. Therefore, theoretical accounts differ with regard to how much weight they place on the contributions of these sources to sentence comprehension. In the present study, we attempted to assess the contributions of event simulation and lexical associations to discourse reading, using event-related brain potentials (ERPs). Participants read target words, which were preceded by associatively related words either appearing in a coherent discourse event (Experiment 1) or in sentences that did not form a coherent discourse event (Experiment 2). Contextually unexpected target words that were associatively related to the described events elicited a reduced N400 amplitude compared to contextually unexpected target words that were unrelated to the events (Experiment 1). In Experiment 2, a similar but reduced effect was observed. These findings support the notion that during discourse reading event simulation and simple word associations jointly contribute to language comprehension by activating words that are beyond contextually congruent sentence continuations.

Abstract

A main challenge for theories of embodied cognition is to understand the task dependency of embodied language processing. One possibility is that perceptual representations (e.g., typical colour of objects mentioned in spoken sentences) are not activated routinely but the influence of perceptual representation emerges only when context strongly supports their involvement in language. To explore this question, we tested the effects of colour representations during language processing in three visual- world eye-tracking experiments. On critical trials, participants listened to sentence- embedded words associated with a prototypical colour (e.g., ‘...spinach...’) while they inspected a visual display with four printed words (Experiment 1), coloured or greyscale line drawings (Experiment 2) and a ‘blank screen’ after a preview of coloured or greyscale line drawings (Experiment 3). Visual context always presented a word/object (e.g., frog) associated with the same prototypical colour (e.g. green) as the spoken target word and three distractors. When hearing spinach participants did not prefer the written word frog compared to other distractor words (Experiment 1). In Experiment 2, colour competitors attracted more overt attention compared to average distractors, but only for the coloured condition and not for greyscale trials. Finally, when the display was removed at the onset of the sentence, and in contrast to the previous blank-screen experiments with semantic competitors, there was no evidence of colour competition in the eye-tracking record (Experiment 3). These results fit best with the notion that the main role of perceptual representations in language processing is to contextualize language in the immediate environment.

Abstract

Composing sentence meaning is easier for predictable words than for unpredictable words. Are predictable words genuinely predicted, or simply more plausible and therefore easier to integrate with sentence context? We addressed this persistent and fundamental question using data from a recent, large-scale (N = 334) replication study, by investigating the effects of word predictability and sentence plausibility on the N400, the brain’s electrophysiological index of semantic processing. A spatiotemporally fine-grained mixed-effects multiple regression analysis revealed overlapping effects of predictability and plausibility on the N400, albeit with distinct spatiotemporal profiles. Our results challenge the view that the predictability-dependent N400 reflects the effects of either prediction or integration, and suggest that semantic facilitation of predictable words arises from a cascade of processes that activate and integrate word meaning with context into a sentence-level meaning.

Abstract

Experiments 1 and 2 examined the time-course of retrieval of phonological, visual-shape and semantic knowledge as Dutch participants listened to sentences and looked at displays of four pictures. Given a sentence with beker, `beaker', for example, the display contained phonological (a beaver, bever), shape (a bobbin, klos), and semantic (a fork, vork) competitors. When the display appeared at sentence onset, fixations to phonological competitors preceded fixations to shape and semantic competitors. When display onset was 200 ms before (e.g.) beker, fixations were directed to shape and then semantic competitors, but not phonological competitors. In Experiments 3 and 4, displays contained the printed names of the previously-pictured entities; only phonological competitors were fixated preferentially. These findings suggest that retrieval of phonological, shape and semantic knowledge in the spoken-word and picture-recognition systems is cascaded, and that visual attention shifts are co-determined by the time-course of retrieval of all three knowledge types and by the nature of the information in the visual environment.

Abstract

Visual attention can be directed immediately, as a spoken word unfolds, towards conceptually related but nonassociated objects, even if they mismatch on other dimensions that would normally determine which objects in the scene were appropriate referents for the unfolding word (Huettig & Altmann, 2005). Here we demonstrate that the mapping between language and concurrent visual objects can also be mediated by visual-shape relations. On hearing "snake", participants directed overt attention immediately, within a visual display depicting four objects, to a picture of an electric cable, although participants had viewed the visual display with four objects for approximately 5 s before hearing the target word - sufficient time to recognize the objects for what they were. The time spent fixating the cable correlated significantly with ratings of the visual similarity between snakes in general and this particular cable. Importantly, with sentences contextually biased towards the concept snake, participants looked at the snake well before the onset of "snake", but they did not look at the visually similar cable until hearing "snake". Finally, we demonstrate that such activation can, under certain circumstances (e.g., during the processing of dominant meanings of homonyms), constrain the direction of visual attention even when it is clearly contextually inappropriate. We conclude that language-mediated attention can be guided by a visual match between spoken words and visual objects, but that such a match is based on lexical input and may not be modulated by contextual appropriateness.