Presentations

Abstract

Coordination is at the heart of human conversation. In order to interact with one another through talk, we must coordinate at many levels, first and foremost at the level of our mental states, intentions and conversational contributions. In this talk, I will present findings on the pragmatics of multi-modal communication from both production and comprehension studies. In terms of production, I will talk about (1) how co-speech gestures are used in the coordination of meaning allowing interactants to arrive at a shared understanding of the things they talk about, as well as on (2) how gesture and gaze are employed in the coordination of speaking turns in conversation, with special reference to the psycholinguistic and cognitive challenges that turn-taking poses. In terms of comprehension, I will focus on the interplay of ostensive (social gaze) and semantic (gesture) signals in the context of intention perception and language processing. My talk will bring different sets of findings together to argue for richer research paradigms that capture more of the complexities and sociality of face-to-face conversational interaction. Advancing the field of multi-modal communication research in this direction will allow us to more fully understand the psycholinguistic processes that underlie human language use and language comprehension.

Abstract

Human language has long been considered a unimodal
activity, with the body being considered a mere vehicle
for expressing acoustic linguistic meaning. But theories of
language evolution point towards a close link between
vocal and visual communication early on in history,
pinpointing gesture as the origin of human language.
Some consider this link between gesture and
communicative vocalisations as having been temporary,
with conventionalized linguistic code eventually replacing
early bodily signaling. Others argue for this link being
permanent, positing that even fully-fledged human
language is a multi-modal phenomenon, with visual
signals forming integral components of utterances in faceto-
face conversation. My research provides evidence for
the latter. Based on this research, I will provide insights
into some of the factors and principles governing multimodal
language use in adult interaction. My talk consists
of three parts: First, I will present empirical findings
showing that movements we produce with our body are
indeed integral to spoken language and closely linked to
communicative intentions underlying speaking. Second, I
will show that bodily signals, first and foremost manual
gestures, play an active role in the coordination of
meaning during face-to-face interaction, including
fundamental processes like the grounding of referential
utterances. Third, I will present recent findings on the role
of bodily communicative acts in the psycholinguistically
challenging context of turn-taking during conversation.
Together, the data I present form the basis of a framework
aiming to capture multi-modal language use and
processing situated in face-to-face interaction, the
environment in which language first emerged, is acquired
and used most.

Abstract

Conversation is the core niche of human multi-modal language use and it is characterized by a system of taking turns. This organization poses a particular psycholinguistic challenge for its participants: considering the gap between two speaking turns averages around just 200 ms (Stivers et al., 2009) but the production of single word utterances takes a minimum of 600 ms alone (Indefrey & Levelt, 2004), language production and comprehension must largely run in parallel; while listening to an on-going turn, a next speaker has to predict the upcoming content and end of that turn to start preparing their own and launch it on time (Levinson, 2013). Recently, research has begun to investigate the cognitive processes underpinning turn-taking (see Holler et al., 2015 for an overview), but this research has focused on the spoken modality. The present study investigates the role co-speech gestures may play in this process. We analysed a corpus of 7 casual face-to-face conversations between English speakers for all question-response sequences (N=281), the gestures that accompanied the identified set of questions, and the timing of these gestures with respect to the speaking turns they accompanied. Moreover, we measured the length of all inter-turn gaps in our set. Our main research question was whether the length of the gap between turns varied systematically as a consequence of questions being accompanied by gesture. Our results revealed that this is indeed the case: Questions with a gestural component were responded to significantly faster than questions without a gestural component. This finding holds when we consider head and hand gestures separately, when we control for points of possible completion in the verbal utterance prior to turn end, and when we control for complexity associated with question type. Furthermore, our findings revealed that within the group of questions accompanied by gestures, those questions whose gestures retracted prior to turn end were responded to faster than questions whose gestures retracted following turn end. This study provides evidence that gestures accompanying spoken questions in conversation facilitate the coordination of turns. While experimental studies have demonstrated beneficial effects of gestures on language processing, this is the first evidence that gestures may benefit processing even in the rich, cognitively challenging context of conversational interaction. That is, gestures appear to play an important psycholinguistic function during immersed, in situ language processing. Experimental work is currently exploring at which level (semantic, pragmatic, perceptual) the facilitative effects we found are operating. The findings not only suggest psycholinguistic processing benefits but also expand on previous turn-taking models that restrict the function of gesture to turn-yielding/-keeping cues (Duncan, 1972) as well as on turn-taking models focusing primarily on the verbal modality (Sacks et al., 1974).

Abstract

In communication, speech is often accompanied by co-speech gestures, which embody a link between language and action. Language impairments in Parkinson’s disease (PD) are particularly pronounced for action-related words in comparison to nouns. People with PD produce fewer gestures from a first-person perspective when they describe others’ actions (Humphries et al., 2016), which may reflect a difficulty in simulation. We extended this to investigate the gestural depiction of other types of action information such as “manner” (how an action is performed) and “path” (the trajectory of a moving figure in space). We also explored whether the level of motion required to perform an action influences the way that people with PD use gestures to depict those actions. 37 people with PD and 35 age-matched controls viewed a cartoon which included low motion actions (e.g. hiding, knocking) and high motion actions (e.g. running, climbing), and described it to an addressee. We analysed the co-speech gestures they spontaneously produced while doing so. Overall gesture rate was similar in both groups, but people with PD produced action-gestures at a significantly lower rate than controls in both motion conditions. Also, people with PD produced significantly fewer manner and first-person action gestures than controls in the high motion condition (but not the low motion condition). Our findings suggest that motor impairments in PD contribute to the way in which actions, especially high motion actions, are depicted gesturally. Thus, people with Parkinson’s may have particular difficulty cognitively representing high motion actions

Abstract

Human face-to-face communication is a multi-modal activity. Recent research has shown that, during comprehension, recipients integrate information from speech with that contained in co-speech gestures (e.g., Kelly et al., 2010). The current studies take this research one step further by investigating the influence of another modality, namely eye gaze, on speech and gesture comprehension, to advance our understanding of language processing in more situated contexts. In spite of the large body of literature on processing of eye gaze, very few studies have investigated its processing in the context of communication (but see, e.g., Staudte & Crocker, 2011 for an exception). In two studies we simulated a triadic communication context in which a speaker alternated their gaze between our participant and another (alleged) participant. Participants thus viewed speech-only or speech + gesture utterances either in the role of addressee (direct gaze) or in the role of unaddressed recipient (averted gaze). In Study 1, participants (N = 32) viewed video-clips of a speaker producing speech-only (e.g. “she trained the horse”) or speech+gesture utterances conveying complementary information (e.g. “she trained the horse”+WHIPPING gesture). Participants were asked to judge whether a word displayed on screen after each video-clip matched what the speaker said or not. In half of the cases, the word matched a previously uttered word, requiring a “yes” answer. In all other cases, the word matched the meaning of the gesture the actor had performed, thus requiring a ‘no’ answer.

Abstract

Objective: To examine how co-speech gestures depicting actions are a ected in Parkinson’s disease (PD), and to explore how gestures might be related to measures of verbal fluency and action naming. Background: PD a ects not only motor abilities, but also language and communication. Language is more impaired for words relating to motor content; e.g., patients take longer to name actions with a high compared to a low motor content. Co-speech gestures embody a form of action which is tightly linked to language and which represent meaningful information that forms a unified whole together with that contained in speech. However, co-speech gestures have rarely been investigated in PD. Recent data showed that gestural precision was reduced in PD patients when describing actions, suggesting that the mental representations of actions underlying their co-speech gestures have become less specific. We investigated this phenomenon for a wider range of actions than the original study, and also explored the possible relationship between verbal fluency/naming deficits and gestures. Method: Sixteen PD patients and 13 IQ-matched healthy controls were video recorded describing pictures and video clips of actions, such as running and knitting. Participants also completed measures of verbal fluency (generating as many words as possible in one minute for certain phonological and semantic categories) and action naming. Results: Analysis is in progress. We are comparing the rate of co-speech gesture production as well as the precision of action-related co-speech gestures between PD patients and controls. We will also examine the relationship between gestures and scores on tasks of verbal fluency and action naming. Conclusions: Investigating co-speech gestures associated with actions has implications for understanding both communication and action representation in Parkinson’s.