Publications

Abstract

Affective behaviors enable social robots to not only establish better connections with humans but also serve as a tool for the robots to express their internal states. It has been well established that emotions are important to signal understanding in Human-Robot Interaction (HRI). This work aims to harness the power of Large Language Models (LLM) and proposes an approach to control the affective behavior of robots. By interpreting emotion appraisal as an Emotion Recognition in Conversation (ERC) tasks, we used GPT-3.5 to predict the emotion of a robot’s turn in real-time, using the dialogue history of the ongoing conversation. The robot signaled the predicted emotion using facial expressions. The model was evaluated in a within-subjects user study (N = 47) where the model-driven emotion generation was compared against conditions where the robot did not display any emotions and where it displayed incongruent emotions. The participants interacted with the robot by playing a card sorting game that was specifically designed to evoke emotions. The results indicated that the emotions were reliably generated by the LLM and the participants were able to perceive the robot’s emotions. It was found that the robot expressing congruent model-driven facial emotion expressions were perceived to be significantly more human-like, emotionally appropriate, and elicit a more positive impression. Participants also scored significantly better in the card sorting game when the robot displayed congruent facial expressions. From a technical perspective, the study shows that LLMs can be used to control the affective behavior of robots reliably in real-time. Additionally, our results could be used in devising novel human-robot interactions, making robots more effective in roles where emotional interaction is important, such as therapy, companionship, or customer service.

Abstract

The current study explored the role of sentential inference in connecting lexical/grammatical knowledge and overall text comprehension in foreign language learning. Using structural equation modeling (SEM), causal relationships were examined between four latent variables: lexical knowledge, grammatical knowledge, sentential inference, and text comprehension. The study analyzed 281 Chinese university students learning Japanese as a second language and compared two causal models: (1) the partially-mediated model, which suggests that lexical knowledge, grammatical knowledge, and sentential inference concurrently influence text comprehension, and (2) the wholly-mediated model, which posits that both lexical and grammatical knowledge impact sentential inference, which then further affects text comprehension. The SEM comparison analysis supported the wholly-mediated model, showing sequential causal relationships from lexical knowledge to sentential inference and then to text comprehension, without significant contribution from grammatical knowledge. The results indicate that sentential inference serves as a crucial bridge between lexical knowledge and text comprehension.

Abstract

This study is the first to report chronometric evidence on Tongan language production. It has been speculated that the mora plays an important role during Tongan phonological encoding. A mora follows the (C)V form, so /a/ and /ka/ (but not /k/) denote a mora in Tongan. Using a picture-word naming paradigm, Tongan native speakers named pictures containing superimposed non-word distractors. This task has been used before in Japanese, Korean, and Vietnamese to investigate the initially selected unit during phonological encoding (IPU). Compared to control distractors, both onset and mora overlapping distractors resulted in faster naming latencies. Several alternative explanations for the pattern of results - proficiency in English, knowledge of Latin script, and downstream effects - are discussed. However, we conclude that Tongan phonological encoding likely natively uses the phoneme, and not the mora, as the IPU..

Abstract

Does the way a word is written influence its spoken production? Previous studies suggest that orthography is involved only when the orthographic representation is highly relevant during speaking (e.g., in reading-aloud tasks). To address this issue, we carried out two experiments using the blocked cyclic picture-naming paradigm. In both experiments, participants were asked to name pictures repeatedly in orthographically homogeneous or heterogeneous blocks. In the naming task, the written form was not shown; however, the radical of the first character overlapped between the four pictures in this block type. A facilitative orthographic effect was found when picture names shared part of their written forms, compared with the heterogeneous condition. This facilitative effect was independent of the position of orthographic overlap (i.e., the left, the lower, or the outer part of the character). These findings strongly suggest that orthography can influence speaking even when it is not highly relevant (i.e., during picture naming) and the orthographic effect is less likely to be attributed to strategic preparation.

Abstract

This study sought to identify tumor characteristics that associate with regional lymph node metastases in squamous cell carcinomas originating in the upper gingiva.

Abstract

In English, Dutch, and other Germanic languages the initial phonological unit used in word production has been shown to be the phoneme; conversely, others have revealed that in Chinese this is the atonal syllable and in Japanese the mora. The current paper is, to our knowledge, the first to report chronometric data on Vietnamese phonological encoding. Vietnamese, a tonal language, is of interest as, despite its Austroasiatic roots, it has clear similarities with Chinese through extended contact over a prolonged period. Four experiments (i.e., masked priming, phonological Stroop, picture naming with written distractors, picture naming with auditory distractors) have been conducted to investigate Vietnamese phonological encoding. Results show that in all four experiments both onset effects as well as whole syllable effects emerge. This indicates that the fundamental phonological encoding unit during Vietnamese language production is the phoneme despite its apparent similarities to Chinese. This result might have emerged due to tone assignment being a qualitatively different process in Vietnamese compared to Chinese.

Abstract

For native (L1) English readers, masked presentations of past-tense verb primes (e.g., fell and looked) produce faster lexical decision latencies to their present-tense targets (e.g., FALL and LOOK) than orthographically related (e.g., fill and loose) or unrelated (e.g., master and bank) primes. This facilitation observed with morphologically related prime-target pairs (morphological priming) is generally taken as evidence for strong connections based on morphological relationships in the L1 lexicon. It is unclear, however, if similar, morphologically based, connections develop in non-native (L2) lexicons. Several earlier studies with L2 English readers have reported mixed results. The present experiments examine whether past-tense verb primes (both regular and irregular verbs) significantly facilitate target lexical decisions for Japanese-English bilinguals beyond any facilitation provided by prime-target orthographic similarity. Overall, past-tense verb primes facilitated lexical decisions to their present-tense targets relative to both orthographically related and unrelated primes. Replicating previous masked priming experiments with L2 readers, orthographically related primes also facilitated target recognition relative to unrelated primes, confirming that orthographic similarity facilitates L2 target recognition. The additional facilitation from past-tense verb primes beyond that provided by orthographic primes suggests that, in the L2 English lexicon, connections based on morphological relationships develop in a way that is similar to how they develop in the L1 English lexicon even though the connections and processing of lower level, lexical/orthographic information may differ. Further analyses involving L2 proficiency revealed that as L2 proficiency increased, orthographic facilitation was reduced, indicating that there is a decrease in the fuzziness in orthographic representations in the L2 lexicon with increased proficiency.

Abstract

This study addressed the question as to whether grammatical properties of a first language are transferred to a second language. Dutch-English bilinguals classified Dutch words in white print according to their grammatical gender and colored words (i.e. Dutch common and neuter words, and their English translations) according to their color. Both the classifications were made with the same hand (congruent trials) or different hands (incongruent trials). Performance was more erroneous and the error-elated negativity was enhanced on incongruent compared with congruent trials. This effect was independent of the language in which words were presented. These results provide evidence for the fact thatbilinguals may transfer grammatical characteristics oftheir first language to a second language, even when such characteristics are absent in the grammar of the latter.

Abstract

The current study investigated the effects of phonologically related context pictures on the naming latencies of target words in Japanese and Chinese. Reading bare words in alphabetic languages has been shown to be rather immune to effects of context stimuli, even when these stimuli are presented in advance of the target word (e. g., Glaser & Dungelhoff, 1984; Roelofs, 2003). However, recently, semantic context effects of distractor pictures on the naming latencies of Japanese kanji (but not Chinese hanzi) words have been observed (Verdonschot, La Heij, & Schiller, 2010). In the present study, we further investigated this issue using phonologically related (i.e., homophonic) context pictures when naming target words in either Chinese or Japanese. We found that pronouncing bare nouns in Japanese is sensitive to phonologically related context pictures, whereas this is not the case in Chinese. The difference between these two languages is attributed to processing costs caused by multiple pronunciations for Japanese kanji.

Abstract

Theories of language production generally describe the segment as the basic unit in phonological encoding (e.g., Dell, 1988; Levelt, Roelofs, & Meyer, 1999). However, there is also evidence that such a unit might be language specific. Chen, Chen, and Dell (2002), for instance, found no effect of single segments when using a preparation paradigm. To shed more light on the functional unit of phonological encoding in Japanese, a language often described as being mora based, we report the results of 4 experiments using word reading tasks and masked priming. Experiment 1 demonstrated using Japanese kana script that primes, which overlapped in the whole mora with target words, sped up word reading latencies but not when just the onset overlapped. Experiments 2 and 3 investigated a possible role of script by using combinations of romaji (Romanized Japanese) and hiragana; again, facilitation effects were found only when the whole mora and not the onset segment overlapped. Experiment 4 distinguished mora priming from syllable priming and revealed that the mora priming effects obtained in the first 3 experiments are also obtained when a mora is part of a syllable. Again, no priming effect was found for single segments. Our findings suggest that the mora and not the segment (phoneme) is the basic functional phonological unit in Japanese language production planning.

Abstract

This thesis investigates the processing of words written in Japanese kanji and Chinese hànzì, i.e. logographic scripts. Special attention is given to the fact that the majority of Japanese kanji have multiple pronunciations (generally depending on the combination a kanji forms with other characters). First, using masked priming, it is established that upon presentation of a Japanese kanji multiple pronunciations are activated. In subsequent experiments using word naming with context pictures it is concluded that both Chinese hànzì and Japanese kanji are read out loud via a direct route from orthography to phonology. However, only Japanese kanji become susceptible to semantic or phonological context effects as a result of a cost due to the processing of multiple pronunciations. Finally, zooming in on the size of the articulatory planning unit in Japanese it is concluded that the mora as a phonological unit best complies with the observed data pattern and not the phoneme or the syllable