Its great to talk big when it comes to the next generation of narrative games and films.
Now let’s talk smart.

1. FACS and Reality

   A system originally developed for Behavioural Psychologists for the categorisation of expressions The facial action Coding System, or FACS (Ekman & Freisen, 1978) has become  one of the bed-rocks of facial animation software.  However as a tool for linking expression to emotion it is flawed,. In this blog I examine what academics have established FACS can and cannot do, and the effect a reliance on FACS can have in high fidelity graphic environments in the endeavour to create behaviour faithful to the human experience.  I then contrast this with the actor/director approach, and why even the smallest dramatic moment will benefit from their skills.

   An Introduction to FACS

   The Facial Action Coding System or FACS (Ekman & Freisen, 1978) is a method of identifying specific facial muscles, and how they relate to emotional facial expressions.  ‘Fundamental’, or ‘basic’ expressions are defined by the combination of particular groups of muscles, Action Units (AU). The expressions are correlated with the following emotions.

   Anger, Contempt, Disgust, Fear , Happiness, Sadness, Surprise.

   The link between a particular FACS defined expression and the attribution of its correlating emotion by an observer has been studied in controlled and field studies, and across cultures. Although the accuracy of judgements of spontaneous facial expression is lower than posed expressions, it is above chance level, (Matsumoto & Willingham, 2009) and this is the measure that satisfies scientific studies.

   However, there are weaknesses, that are important to recognise if this tool is to be used in the re-creation or enhancement of facial behaviours in animation.

   1. Accuracy of FACS emotion attribution (giving an expression an emotion) is tested in a variety of ways. Objective judgements can be affected by the context given for the facial expression.  In one study, a context described to someone as as of “a person peering at a distant message and having difficulty making a decision about it”, then shown an expression categorised by FACS as Anger, was usually labelled as “an expression of puzzlement”.  A FACS expression, is a molecular categorisation, and as such, without wider context or molar consideration (For definitions see below), is often not enough to go on to make an accurate inference of emotion. (Carroll & Russell, 1996)
   2. The link between subjective experience of emotion, and the FACS correlated facial expression, has not been scientifically established.  A simple example is to consider the wide use of smiling in a range of different contexts.  Smiles can occur whilst describing a sad personal event.  The emotion behind the smile may be anything but happiness. (see Garcia-Higuera & Fernandez-Dols, 1997)
   3. The reason behind facial expressions is also hotly debated, and often drifts into either hard wired or socially motivated. It appears that both have authority, and it seems likely that facial expressions are performed for a variety of reasons.  Expressions change if you’re alone, with someone else, a friend, or in a particular social context (Chovil, 1991)), and can vary according to the intensity of the stimulus too.  Expressions occur as part of the interplay between all these mediating factors.
   4. In the real world expressions are part of a chain of behaviours; never isolated.  In the Matsumoto study cited below, observers were asked to judge a range of expressions from winners and losers at an Olympic judo context.  The first part of the study proved that there was accurate attribution of emotion in isolated clear FACS expressions.  But when observers were asked to judge a longer timeframe, which included a mix of clear FACS and more ambiguous expressions, and from that decide whether they were looking at a winner of a loser, accurate judgement of individual FACS expressions decreased.

   “We believe that this difference in the findings occurred because a number of athletes provided multiple expressions to the stimuli, and athletes’ expressions changed over time.  Some winners who smiled initially subsequently cried or showed no emotion; some losers who showed negative emotions initially showed none or even smiled subsequently.”
(Matsumoto & Willingham, 2009)

   FACS and keyframe animation

   For a tool designed for classification of expression in the realm of academe, FACS serves an animator rather well, particularly when graphics are simplified and photo-realism is not the desired result. In fact, one can make parallels between the  performance and behaviour ‘levels’ in different styles of acting – Restoration, Silent Movie, to HD extreme close-ups, with performance levels in different animation styles.  But that’s another blog, folks!

   However, the closer you approach the infamous ‘uncanny valley’ the more demands are placed on the quality of the facial animation, and all the attendant tells – breathing, head movement, tongue animation and sync, skin tone, eyes etc.  As up-close as it can be, the spell-breaking effect of a weird facial behaviour seems to be more distressing than other bloopers – say walking through the edge of a table. In comparison, the latter is annoying, the former can be truly off-putting.

   Extremely high quality facial images can be constructed, and improved technology (such as  Solvers) will make the current process more efficient. However, until performance capture-to-final product can be as quick and as cheap as film, I propose that improvements can be made with existing technology, and a more discriminating use of FACS, together with old fashioned visual story-telling approaches.

   FACS, although a brilliant tool, should in no way be relied upon in for emotion attribution for the following reasons:

   • Without extensive time at his or her disposal, an animator cannot compete with the idiosyncratic mix, ambiguities, and surprises offered by a lived performance (Actors).
   • The untidiness of constantly shifting, sometimes ambiguous, inter-related behaviours is where character depth and believability exists.
   • A director expert in visual storytelling can offer an edit that makes the most of all the data – the world, the location, the scene, the characters, and the story.

   Discretion of which tool to use and when, is important for any artist, in game, theatre or film. Sure, you can have the best solver, but you’ll still need to tweak (using FACS), and must use the discretion of a director when doing so, and allow ambiguity to be shown. You might have the best animators, but you might not have the time to create the nuanced or surprising behaviours from scratch -  that is the actors speciality.

   The Actor, the Director and the Animator…

   Good animators are highly skilled at inferring emotion, either from narrative context alone, or from an actors expression.  They are required to make objective decisions on what emotional expression is consistent, together with the level required, and removing expression ambiguity when necessary.  It might appear that the demands placed on animators seem analogous to those placed on actors. But there are fundamental differences.

   Good actors are experts in the interpretation and subjective experience of dramatic situations.  They are required to suspend their own disbelief and behave without self-monitoring, as the character in the fictional situation.  Their performance must be consistent with the character, the situation, physical and social context; in short all those factors that mediate behaviour. This means that surprising, but situationally consistent behaviours can occur that would not correlate with a FACS attributed emotion, and are more likely to come from subjective experience, than objective judgement.  Simple examples:  Crying with joy, smiling with loathing, releasing anger at success – for example…  But also ambiguities of AU mixes that would not even be covered by standardised FACS blends.

   A little bit of acting methodology…

   Stanislavsky based acting techniques share the founding principles of modern psychology:  Goal Oriented Action (James, W. 1890) Various teachers (Lewis, R. 1980,1986: Merlin, B., 2002, 2005; M, Hagen, U. 1973, 1991) use a list of questions that help an actor to sort out the different influences that work upon the character, from the clues that exist in the text, filling in the rest with research, experiment and imagination.

   1. THE TEN QUESTIONS:
Who am I?  (Identity)
   2. Where am I? (Physical Context)
   3. What time is it?  (Temporal Context)
   4. Where have I come from (Previous Circumstances)
   5. What do I want? (Objective)
   6. Why do I want it? (Motive)
   7. What will happen if I don’t get it? (Stakes)
   8. What do I do to get what I want? (Action/Activity)
   9. What is stopping me from getting it? (Obstacle)
   10. What do I expect will happen? (Expectations)

   I do not purport that this is the only acting method, and for myself I subscribe to the “whatever works” philosophy.  But as an interesting contrast to the process of decoding and encoding behaviour it provides a good example.

   You’ll notice that the answers to these inter-related questions can move between the particular to the situational, and vice versa.  “Particular” is not to be confused with “molecular”.  After all, a good animator may well ask the same questions, and come to similar conclusions, but method actors never ask “What’s my expression?”.   It is the setting up of the reality that counts, and the emotions/expressions look after themselves.  Nor do any of the questions include – “If I behave this way in scene X, what effect does it have on Character Y at the end of Act 2, and the story as a whole…?” That’s the directors job.  Why?  Because acting is a subjective experience.  It may require some objective choices along the way, but the actual acting is something that you do without thinking about what you’re doing.  If you’re watching yourself, it can result in obvious fakery  for instance – indicating your feelings with overdone “Expressions”, or becoming blank and uninvolved. What has this to do with FACS and animation?

   Indicating feelings with overdone expressions?  Blank and uninvolved looks? Ever seen these in cutscenes and the in-game “face”?  If the answer is “come-to-think-of-it, Yes” then read on.  If not, and you can empirically prove otherwise, please correct me.  Otherwise we’ll just have to put it down to a question of differing taste, and part here amicably.  After all, there are people who rate soap-opera style acting, over minimalist film acting, and vice versa.

   Outsourcing to specialists…  Yep, Actors and Directors.

   Animators are faced with the same challenges of any actor, but their job requires molecular management – something that works against the creation of molar behaviour.  Currently there are limitations on just how detailed you can go, that are dependent on console capabilities, instantly ageing technology, and development cost/time.  But even small touches from an expert director, or that idiosyncratic source behaviour from an actor, can make a positive difference.  The beginning of a car race, has a story to tell – the line up of the cars, the revving of the engines, the starters lights. Without a single actor in sight, you are already making directorial choices over frame, focus, sound,  and movement.

   If big AAA games take workshops on directing, employ professional directors and actors, then why not you? Get the best you can get.  It doesn’t have to be Scorcese.

   Don’t be afraid to ‘earn’ going close

   In some of the AAA games, a great deal of editorial wriggling goes on to avoid having to go too close, or expose costly facial detail.  The darkness obscuring the face; the back-of-head shots; the over-use of wide shots; the stony-faced male-protagonist; even over-focussing on the speaker rather than the listener:  These are techniques that at their best can add to atmosphere and story telling, whilst also being efficient.  Brilliant!  You’ll never hear me criticise quality that improves efficiency!  However, I think these techniques can be overused, and are excluding the possibility of better handling of a greater range of material, and improving dramatic content further.

   FACS naturally lends itself to the idea that its ‘safe’ to work with FACS-clear expressions, and that clarity of expression is what you pitch for.  But it is the constant shift and mix that constitutes the human experience, and added context that carries the emotional meaning.

   There’s an old bit of acting tenet that says you should earn your silences.  This is to avoid the slow pace of overindulgence.  I feel that the next step for the AAA Empathy Game, is to earn those close ups, and build scenes around the greatest dramatic range, investing in targeted close-ups, and mid-shots and that requires a tight integration of directors and actors skills with each game development department.

   No single technique or tool has all the answers to the challenges posed by increasing photo-realism, and demand for improved dramatic content.  Trusting what each skill can provide will improve visual storytelling that much more, without breaking the bank.

   Footnotes:

   Molecular categories consist of specific actions – the movement exclusive set of muscle movements.
   Molar categories combine different kinds of behaviour in generic classes defined by outcome.

   References:

   Carroll, J.M., & Russell, J.A. (1996). Do facial expressions signal specific emotions?  Judging emotions from faces in context. Journal of Personality and Social Psychology, 70: 205-218
   Chovil, N. (1991). Social Determinants of facial Displays. Journal of Noverbal behaviour, 15:141-154
   Ekman, P., & Freisen, W. V. (1978).  Facial Action Coding System (FACS): A technique for the measurement of facial expression. Paolo Alto, CA: Consulting Psychologists Press.
   Garcia-Higuera, J. A. & Fernandez-Dols, J. M. (1996). Unpublished raw data cited in Fernandez-Dols, J. M. Facial Expression and Emotion, The social context of Nonverbal behaviour (1999) Ed Pierre Philipot, Robert S.Feldman, Erik J Coats.  CAmbridge university Press, Paris.
   Hagen, U. (1973). Respect for acting. N.Y: Wiley.
   Hagen, U. (1991). A challenge for the actor. NY: Scribner.
   James, W. (1890). The principles of psychology (Vols. 1-2). New York: Holt.
   Matsumoto, , D., Olide, A., Schug, J., Willingham, B., Callan, M. (2009) Cross-Cultural Judgements of Spontaneous Facial Expressions of Emotion. Journal of Nonverbal Behaviour 33:213 – 238
   Lewis, R. (1980). Advice to the players. NY: Theatre Communications Group, Inc.
   Lewis, R. (1986). Method or madness? NY: Samuel French Inc.
   Merlin, B. (2002). Beyond Stanislavski. The psycho-physical approach to actor training (2nd
ed.) London: Nick Hern Books.
   Merlin, B. (2005). Konstantin Stanislavski (2nd ed) NY: Routledge.

   Posted: October 1, 2010 at 9:29 am