Issues in multimodal nonverbal communication and emotion in embodied (conversational) agents

Virtual worlds are getting inhabited by virtual humans. Sometimes they act as (autonomous) embodied conversational agents; sometimes they represent human visitors and reflect (real-time) actions performed by the human visitors or users of the environment. It is not always necessary to represent a full body or any body at all. However, there are also many applications where it can be useful to have an embodied agents that does not only allow verbal interaction, but also interacts through nonverbal means, including a display of emotion. In this paper we survey the main modalities to show emotion in an embodied conversational agent. We argue that for many situations it is more important to have subtle ways of adding nonverbal cues to agent-human interaction than to be able to express some discrete full-blown emotions. We illustrate this with two of our research projects, one on gaze behavior of an embodied conversational agent and one on showing intensities and blends of emotions in facial expressions of an embodied agent.