Left: These graphs indicate weighting of dimensions that most affect the perceptions of emotions in the image set; in this case, the graphs represent the dimensions for a.) sadness and b.) anger.
Right: Examples of shape-based recognition using a shape detection algorithm.

Understanding the different categories of normal facial expressions of human emotion is essential for scientists and doctors in order to gain insights into human cognition and affect, as well as for the design of computational models and perceptual interfaces. Past research on facial expressions of emotion – as far back as Aristotle – focused on the study of six basic categories of emotion: happiness, surprise, anger, sadness, fear and disgust. However, many more facial expressions of emotion exist and are used regularly by humans.

A research group led by Aleix Martinez, Ph.D., associate professor of Electrical and Computer Engineering at The Ohio State University, is investigating several important groups of expressions, known as compound-emotion categories. Compound emotions are those that can be constructed by combining basic component categories to create new ones. For instance, happily surprised and angrily surprised are two distinct compound-emotion categories.

Martinez and his colleagues leveraged computational systems of the Ohio Supercomputer Center to analyze 5,000 facial-expression images of 230 subjects, categorizing the images by which facial muscles the participants used. The Facial Action Coding System (FACS) codifies facial expressions by action units, or muscles or groups of muscles that go into making facial expressions – such as lip parts (for showing disgust), showing teeth (for expressing happiness), mouth stretch (for fear), or eyelid tightening (for anger).

“Our current work defines 21 distinct emotion categories,” Martinez explained. “An FACS analysis shows the production of these 21 categories is different from, but consistent with, the subordinate categories they represent. For example, a happily surprised expression combines muscle movements observed in happiness and surprised.”

The long-term goal of the research is to understand how emotions and other facial cues – such as identity and grammar in sign languages – are encoded and interpreted from images and, thus, how these facial movements encode the communication signal.

“The main applications are in the understanding of what goes wrong in these computations in patients with certain conditions, such as depression, post-traumatic stress disorder or autism, as well as in the implementation of computational systems that can emulate this capacity,” said Martinez.

The results of this proposal also will be useful to design computer algorithms that can do recognition of emotions. Face recognition is of primary importance in many areas of computational intelligence – ranging from human-computer interaction to content-based retrieval.


Project Lead: Aleix Martinez, Ph.D., The Ohio State University
Research Title: Face recognition: Computational modeling
Funding Source: National Institutes of Health
Website: http://www2.ece.ohio-state.edu/~aleix/