AuGe
With our AuGe (Automatic Gesture) bands, users do not have to worry about selecting appropriate conversational gestures for their chat messages. The AuGe bands add gestures automatically.
You can get your own AuGe band at the following Second Life landmark!
Installation
The AuGe bands are available in our virtual Starbucks coffee shop inside the replica of the National Center of Sciences building in Tokyo.
- Click on the red doughnut box and agree to the disclaimer.
- Accept the AuGe band object.
- The object will then appear in your inventory: just right click it and select “wear” (the object will then be attached to the right forearm of your avatar).
If you type a message in the chat channel, the system will select an appropriate gesture and animate your avatar. Currently, at least one gesture is displayed for each chat message. In standard case, this is a formless hand/arm movement, also called “beat” (gesture).
Text-to-Speech (TTS): A TTS engine has to be installed separately. We currently use the product from E.V.A. Voicechat Enhancement.
If you want to see the AuGe bands in action, please watch our YouTube video, which is a parody of the “Kill Bill” movie:
Research Background
Gestures are an integral part of human non-verbal communicative behavior: they can convey meaning, stress certain aspects of the utterance, and add to a more expressive and natural conversation. With our AuGe system, users of Second Life may communicate more naturally and human-like, without having to select gestures manually.
The AuGe architecture consists of two parts:
- The Second Life object.
- The Gesture Generation System (GGS) running on our server.
The interface between those two is implemented as a Java Servlet. If the user types a new message in the chat, our band recognizes it and sends it to the GGS, which then starts the generation process.
The gesture generation in our system operates on the utterance level (single chat input), for which certain rules are defined. Based on contextual and linguistic information of the text message, gesture behavior for the avatar is suggested. In addition to the layer of rules, we apply a second layer based on keywords and phrase spotting for more specific gesture selection. A third layer uses iterations to align the suggested gestures to each other.
Eventually a single string containing the gesture and timing information is sent back to the AuGe Band, which starts the animations in Second Life. Those are either embedded in the AuGe band object of the avatar or Second Life internal animations.
The AuGe System was developed by Werner Breitfuss, a member of Ishizuka Lab at the University of Tokyo, in a joint research project with Prendinger Lab at the National Institute of Informatics. We would like to acknowledge the Stanford NLP Group for providing the Stanford Parser under GNU GPL, which is used by our system for annotating syntactic and morphological information.
For more information on the underlying gesture generation system, please check our publication:
Werner Breitfuss, Helmut Prendinger, and Mitsuru Ishizuka. Automatic generation of gaze and gestures for dialogues between embodied conversational agents. Int’l J of Semantic Computing, Vol. 2, No. 1, July 2008, pp 71-90 [pdf]
Other relevant papers can be found on the homepages of Helmut Prendinger and Ishizuka Lab.
