Gesture-based interfaces

With digital convergence, gesture-based interfaces are becoming a reality, wherein hand or other gestures by the user drive software instead of a mouse and keyboard. A popular example in film was the interface in Minority Report, but real systems like that are now being prototyped. The Wii is an example of a practical system using gesture-based input.


Some systems also use force feedback or haptics to give the user tactile feedback as he or she navigates around the interface.


It is only a matter of time before animation tools get gesture based interfaces, which would give an oddly reflexive element to animating - you animate gestures in software with other gestures. The tool below is an example of existing technology, and demonstrates the proof of concept, though there is still some way to go before the technology is cheap, portable and intuitive enough to use for the animator.




I've worked on some research projects that involved gesture based interfaces myself, such as the one below which was a 3D environment navigation application driven by a user wearing special gloves.



More information at: http://infolab.usc.edu/projects/geodec/

Gesture, Culture and Mirror Neurons

Image taken from http://iacoboni.bmap.ucla.edu/language_gesture.html#gest-mot

One of my earlier posts listed instances of communities that spontaneously came up with micro-languages based on gesture because of their inability to communicate via speech.

But how closely are gesture and culture related?

I recently came across an interesting study by Istvan Molnar-Szakacs and Dr. Marco Iacoboni that examines the relationship between gesture and culture using a specific type of brain cell known as a "mirror neuron". According to the article, these neurons "fire not only when an individual performs a particular action but also when he or she watches another individual perform that same action. Neuroscientists believe this "mirroring" is the mechanism by which we can read the minds of others and empathize with them."

By measuring brain activity, the scientists found was that American subjects showed higher responsiveness to either American, Nicaraguan or completely meaningless gestures when performed by an American as opposed to the same gestures when performed by a Nicaraguan. In other words, we are programmed to respond more to our own cultural or ethnic "in-group", leading the scientists to surmise that "our brain mirrors people, not actions".

Since the perceived culture of the performer affects our interpretation of gestures, there is a cultural component to gesture. The strength of this connection is still unclear from the article I read, but would determine the impact on animation as it becomes an increasingly cross cultural medium.

Summary Of Dance and Robot Choreographer, Margot Apostolos

Highlights from Margot Apostolos's presentation on Dance and Robot Choreography. Examples of Margot's Robot Choreography are displayed and dissected by Margot as they relate to her experiences in Dance and how they relate to animation. Margot presents her experiences in conveying robot movements into smooth, more graceful and dance-like portrayals, focusing on her efforts of constructing key-frames and in-between keys in her programming to allow the robots to move along curved paths of motion.......a very important idea in conveying fluid natural motion in character animation.

Deconstructing Character Choreography and Thoughts On How Motion Capture Is Receive In The Dance Community

Margot discusses some interesting thoughts on communicating gesture and motion in her dance students. She hopes her students will plan to choreograph their own performances by de-constructing their dance choreography to express their "Movements as Sentences, and your Steps as Words" to convey, successfully, the message of their character gestures and motions through dance performance.

Margot also discusses her experiences with working with Motion Capture technology to create an interacting dance performance between motion captured character performers and live action dancers performing together and how her experiences have shown that many dancers feel hesitant or reluctant to share the stage with a screen projected motion capture character, performing the same dance choreography.

Panel Discussion About Disecting Dance Motions And Gestures From The Point Of Views Of A Dance Choreographer, Traditional 2-D Animators, And A 3-D C.G

In this video the discussion is presented to a panel of Character Animation specialists, A Dance and Robot Choreographer (Margot Apostolos), Two Disney Animators (Andreas Deja and Tom Sito), and a Sony Imageworks 3-D Animator (Maks Naporowski). The discussion is based on the question of how dance motions and gestures are conveyed in choreographed dance to elicit a successful movement of emotion, and if these same guidelines or principles of successful motion are used to critique the success of an animated movement in the choreography presented through the character animation performance.

Automated Anticipation?

I came across a paper entitled "Anticipation Effect Generation for Character Animation". Basically, these researchers were looking at automated ways to add anticipation to existing animation (presumably 3d animation). It's an interesting notion, and seems to tackle the subtleties of animation from the opposite perspective as motion capture. This technique aims to make it easier to improve keyframe animation. Most animators would object to this approach on the grounds that anticipation is subjective and not easily derived automatically. The researchers found the same thing - they could not automate the duration of the anticipation and needed human intervention.

Here's the abstract:
"According to the principles of traditional 2D animation techniques, anticipation makes an animation convincing and expressive. In this paper, we present a method to generate anticipation effects for an existing animation. The proposed method is based on the visual characteristics of anticipation, that is, “Before we go one way, first we go the other way [1].” We first analyze the rotation of each joint and the movement of the center of mass during a given action, where the anticipation effects are added. Reversing the directions of rotation and translation, we can obtain an initially guessed anticipatory pose. By means of a nonlinear optimization technique, we can obtain a consequent anticipatory pose to place the center of mass at a proper location. Finally, we can generate the anticipation effects by compositing the anticipatory pose with a given action, while considering the continuity at junction and preserving the high frequency components of the given action. Experimental results show that the proposed method can produce the anticipatory pose successfully and quickly, and generate convincing and expressive anticipation effects."

The entire paper can be found at:
http://www.springerlink.com/content/h42451j2j38l5216/ and the pdf is at
http://www.springerlink.com/content/h42451j2j38l5216/fulltext.pdf

(You may need to be on USC campus to be able to access the links)

Laban Movement Analysis

Our seminar speaker, Margot Apostolos, mentioned Laban in her discussion of gesture and dance. Along these lines, I took a closer look at Laban Movement Analysis (LMA). What's interesting about Laban Movement Analysis is that unlike our earlier posts on classifying gesture into iconic gestures, emblems, etc., this approach relies more on understanding movement as a continually changing, but orderly and intentionally sequenced series of gesture elements. In particular, this approach looks not only at the elements of the gesture (i.e. Shape) but also the way in which the elements are put together (i.e. Effort).

Thus, two gestures that very similar in structure (an example on Wikipedia is punching someone in anger versus reaching for a glass) can be differentiated by the intensity or Effort of the gesture.

In contrast to simply categorizing gestures, I think that this form of gesture analysis also leaves plenty of room for emotion as a driving tool for gesture, which is something that both the animators and dancer at our seminar presentation stressed.

References:
Synthesis and Acquisition of Laban Movement Analysis Qualitative Parameters for Communicative Gestures
Liwei Zhao, Norman I. Badler
http://repository.upenn.edu/cgi/viewcontent.cgi?article=1118&context=cis_reports

Wikipedia
http://en.wikipedia.org/wiki/Laban_Movement_Analysis

Margo Apostolos Bio

Margo is both a dancer and a roboticist (!). Seems like she will be a really interesting resource for trying to understand how we perceive and interpret gesture.

I found a neat article that she wrote about robots choreography: http://www.jstor.org/view/0024094x/ap050073/05a00110/0
Since the article is on JStor for which you need a subscription, you may need to be on campus to access the above link (USC provides free access).

Here's her bio from the USC website:

Dr. Apostolos has authored and presented numerous articles on her research and design in Robot Choreography. In addition to her doctoral and post-doctoral studies at Stanford University, she earned an M.A. in Dance from Northwestern University. She has served as visiting professor in the Department of Psychology at Princeton University and has taught in Chicago, San Francisco, at Stanford University, Southern Illinois University and California Polytechnic State University-San Luis Obispo. A recipient of the prestigious NASA/ASEE Summer Faculty Fellowship, Dr. Apostolos worked for NASA at Jet Propulsion Laboratory/Caltech as a research scientist in the area of space telerobotics. At the Annenberg Center for Communication, Dr. Apostolos conducted research on facial expressions and human-computer interactions. Her work in this area continues in collaboration with the USC neuroscience program. She was named a Southern California Studies Center faculty fellow, where she is surveying theatre and dance in the Southern California region. Currently, Stanford University is providing funding for Dr. Apostolos’ work in the area of “Dance for Sports” in collaboration with the Stanford Athletic Department. This work was presented to the International Olympic Committee in Sydney, Australia, for the 2000 Games and in preparation for the Salt Lake City Games in 2002. Dr. Apostolos developed the Dance Minor program in Theatre, directs the dance concert each semester, and coordinates the Open Gate Dance Program. In 2004, she presented at the Athens IOC meeting on Dance for Sports and at the World Congress of Dance. This past spring, she was instrumental in bringing internationally-renown director/choreographer Mark Morris to campus for a workshop that integrated motion capture technology and robotics with modern dance. Most recently, she co-founded the Cedars-Sinai/USC Dance Medicine Center, a specialized treatment center – the first of its kind in Los Angeles – offering preventive care and treatment specifically designed for professional and recreational dancers.

emblems- example

Ice Age
(click here)

Emblems

Wikipedia says,

The words emblem and symbol often appear interchangeably in day-to-day conversation without causing undue confusion. A distinction between the two may seem unnecessarily fastidious. Nevertheless, an emblem is a pattern that is used to represent an idea, or an individual. An emblem crystallizes in concrete, visual terms some abstraction: a deity, a tribe or nation, a virtue or a vice. An emblem is an object or a representation of an object.

Emblem gesture presents message. We get emblems by learning. Like we have hard time to read the original Shakespeare's play because of the old words, our everyday words are slowly changing. Because of that, people can get different emblems by their culture, generation, and so on. So it's not easy to find understandable emblems to everyone in the world.

examples of emblems;

More on Gesture and the Brain: The Mind's Big Bang

I watched the second video that Kathy recommended to us.

A quick summary:
According to the video, although modern humans evolved 100,000 years ago, starting about 50,000 years ago there were dramatic changes in the technology and culture of humans that eventually led to our dominance of the planet. The video links the "Mind's Big Bang" to man's tendency towards increasingly complex communication and social interaction strategies. Thus, this period witnessed the birth of art and culture, including primitive beads and cave painting.

One of the examples that the video puts forth is the case of deaf children in Managua, Nicaragua. These children spontaneously came up with a complex series of gestures to communicate with each other. The children also resisted attempts to make them learn ASL.

More details can be found at:
http://en.wikipedia.org/wiki/Nicaraguan_Sign_Language

The development is pretty remarkable and it's relevant to our research because it establishes that gesture is not necessarily only a secondary element of communication (after speech), but can take over the entire function of transmitting information among individuals in complex interactions. In turn, this observation reinforces the widespread animation practice of animating poses and body movement first, before moving on to facial expressions, as well the importance of silhouettes. Also, the video talks about "meems" or cultural transmission through imitation, and it appears that the Nicaraguan Sign Language is an example of this phenomenon. Consequently, one would expect that there is a strong culturally-specific aspect to gestures. In support of this notion, there's other cases where similar micro-languages based on gesture have arisen:

• Al-Sayyid Bedouin Sign Language
• Adamorobe Sign Language
  • 
• Yucatec Maya Sign Language
• Kata Kolok
• Martha's Vineyard Sign Language

Gesture types

Gestures, the movement of arms and hands, are different from other body language in that they tend to have a far greater association with speech and language. Whilst the rest of the body indicates more general emotional state, gestures can have specific linguistic content.

Gestures have three phases: preparation, stroke and retraction. The real message is in the stroke, whilst the preparation and retraction elements consist of moving the arms to and from the rest position, to and from the start and end of the stroke.

Emblems

Emblems are specific gestures with specific meaning that are consciously used and consciously understood. They are used as substitutes for words and are close to sign language than everyday body language.

For example, holding up the hand with all fingers closed in except the index and second finger, which are spread apart, can mean 'V for victory' or 'peace' (if the palm is away from the body) or a rather rude dismissal if the palm is towards the body.

Iconic gestures

Iconic gestures or illustrators are closely related to speech, illustrating what is being said, painting with the hands, for example when a person illustrates a physical item by using the hands to show how big or small it is. Iconic gestures are different from other gestures in that they are used to show physical, concrete items.

Iconic gestures are useful as they add detail to the mental image that the person is trying convey. They also show the first person or second person viewpoint that the person is taking.

The timing of iconic gestures in synchronization with speech can show you whether they are unconscious or are being deliberately added for conscious effect. In an unconscious usage, the preparation for the gesture will start before the words are said, whilst in conscious usage there is a small lag between words and gesture (which can make the speaker appear manipulative).

Metaphoric gestures

When using metaphoric gestures, a concept is being explained. Gestures are in three-dimensional space and are used to shape and idea being explained, either with specific shapes such as finger pinches and physical shaping, or more general waving of hands that symbolizes the complexity of what is being explained.

Regulators

Regulators are used to control turn-taking in conversation, for example in the way that as a person completes what they are saying, they may drop their arms, whilst a person wanting to speak may raise an arm as if to grasp the way forward.

Affect displays

Gestures can also be used to display emotion, from tightening of a fist to the many forms of self-touching and holding the self. Covering or rubbing eyes, ears or mouth can say 'I do not want to see/hear/say this'. Holding hands or the whole body can indicate anxiety as the person literally holds themself. Self-preening can show a desire to be liked and can indicate desire of another.

Beat gestures

Beat gestures are just that, rhythmic beating of a finger, hand or arm. They can be as short as a single beat or as long as needed to make a particular point.

Beating and repetition plays to primitive feelings of basic patterning, and can vary in sense according to the context. A beat is a staccato strike that creates emphasis and grabs attention. A short and single beat can mark an important point in a conversation, whilst repeated beats can hammer home a critical concept.

(http://www.changeminds.org/explanations/behaviors/body_language/gesture_type.htm)

What is Gesture?

The term 'gesture' itself is pretty loaded, and there are a wide variety of definitions that I've seen. For our purposes, however, perhaps it's more interesting to look at the different types of gestures.

This paper - http://ls12-www.cs.uni-dortmund.de/~fink/lectures/SS06/human-robot-interaction/GestureBasedInteractionPartOne.pdf - has a taxonomy that might prove useful. According to the paper, gestures can be classified according in a tree-like structure. We're mainly interested in Communicative gestures, and here's my paraphrasing of what the article says:
1. Mimetic gestures mimic an action e.g. dribbling a ball
2. Deictic gestures e.g pointing to something within a space
3. Referential gestures refer to the environment e.g. a circular motion that describes a steering wheel
4. Modalizing gestures change the meaning of what's being said e.g. a shrug of the shoulders

I'm not sure as to how exactly this will tie into our work, but I think it's important to have this sort of definition/classification as a guide throughout the semester. In addition, I think we could potentially do some simple animation to illustrate these types of gestures.

Gesture and the Brain

Here's some links I found that we can investigate further:

http://www.colgate.edu/DesktopDefault1.aspx?tabid=684&pgID=3400&fID=3213

http://www.leaonline.com/doi/abs/10.1207/S15326942dn2201_1

http://www.columbia.edu/~rmk7/PDF/K&H.pdf

http://www.magazine.ucla.edu/exclusives/brain-hand-gestures/

Some of this research talks about the similarities and differences between language and gesture, which is also related to the videos that Kathy would like us to watch. One of the articles also talks about gesture and culture using a neurological approach.