Presentations

Abstract

Human communication requires interlocutors to mutually understand each other. Previous research has suggested inter-brain synchrony as an important feature of social interaction, since it has been observed during joint attention, speech interactions and cooperative tasks. Nonetheless, it is still unknown whether inter-brain synchrony is actually related to successful face-to-face communication. Here, we use dual-EEG to study if inter-brain synchrony is modulated during episodes of successful and unsuccessful communication in clear and noisy communication settings. Dyads performed a tangram-based referential communication task with and without background noise, while both their EEG and audiovisual behavior was recorded. Other-initiated repairs were annotated in the audiovisual data and were used as indexes of unsuccessful and successful communication. More specifically, we compared inter-brain synchrony during episodes of miscommunication (repair initiations) and episodes of mutual understanding (repair solutions and acceptance phases) in the clear and the noise condition. We expect that when communication is successful, inter-brain synchrony will be stronger than when communication is unsuccessful, and we expect that these patterns will be most pronounced in the noise condition. Results are currently being analyzed and will be presented and discussed with respect to the inter-brain neural signatures underlying the process of mutual understanding in face-to-face conversation.

Abstract

Human face-to-face communication is a multi-modal activity. Recent research has shown that, during comprehension, recipients integrate information from speech with that contained in co-speech gestures (e.g., Kelly et al., 2010). The current studies take this research one step further by investigating the influence of another modality, namely eye gaze, on speech and gesture comprehension, to advance our understanding of language processing in more situated contexts. In spite of the large body of literature on processing of eye gaze, very few studies have investigated its processing in the context of communication (but see, e.g., Staudte & Crocker, 2011 for an exception). In two studies we simulated a triadic communication context in which a speaker alternated their gaze between our participant and another (alleged) participant. Participants thus viewed speech-only or speech + gesture utterances either in the role of addressee (direct gaze) or in the role of unaddressed recipient (averted gaze). In Study 1, participants (N = 32) viewed video-clips of a speaker producing speech-only (e.g. “she trained the horse”) or speech+gesture utterances conveying complementary information (e.g. “she trained the horse”+WHIPPING gesture). Participants were asked to judge whether a word displayed on screen after each video-clip matched what the speaker said or not. In half of the cases, the word matched a previously uttered word, requiring a “yes” answer. In all other cases, the word matched the meaning of the gesture the actor had performed, thus requiring a ‘no’ answer.

Abstract

Objective: To examine how co-speech gestures depicting actions are a ected in Parkinson’s disease (PD), and to explore how gestures might be related to measures of verbal fluency and action naming. Background: PD a ects not only motor abilities, but also language and communication. Language is more impaired for words relating to motor content; e.g., patients take longer to name actions with a high compared to a low motor content. Co-speech gestures embody a form of action which is tightly linked to language and which represent meaningful information that forms a unified whole together with that contained in speech. However, co-speech gestures have rarely been investigated in PD. Recent data showed that gestural precision was reduced in PD patients when describing actions, suggesting that the mental representations of actions underlying their co-speech gestures have become less specific. We investigated this phenomenon for a wider range of actions than the original study, and also explored the possible relationship between verbal fluency/naming deficits and gestures. Method: Sixteen PD patients and 13 IQ-matched healthy controls were video recorded describing pictures and video clips of actions, such as running and knitting. Participants also completed measures of verbal fluency (generating as many words as possible in one minute for certain phonological and semantic categories) and action naming. Results: Analysis is in progress. We are comparing the rate of co-speech gesture production as well as the precision of action-related co-speech gestures between PD patients and controls. We will also examine the relationship between gestures and scores on tasks of verbal fluency and action naming. Conclusions: Investigating co-speech gestures associated with actions has implications for understanding both communication and action representation in Parkinson’s.