Automatic 2.5-D Facial Landmarking and Emotion Annotation for Social Interaction Assistance

Xi Zhao, Jianhua Zou, Huibin Li, Emmanuel Dellandrea, Ioannis A. Kakadiaris, Liming Chen

Research output: Contribution to journalArticlepeer-review

25 Scopus citations


People with low vision, Alzheimer's disease, and autism spectrum disorder experience difficulties in perceiving or interpreting facial expression of emotion in their social lives. Though automatic facial expression recognition (FER) methods on 2-D videos have been extensively investigated, their performance was constrained by challenges in head pose and lighting conditions. The shape information in 3-D facial data can reduce or even overcome these challenges. However, high expenses of 3-D cameras prevent their widespread use. Fortunately, 2.5-D facial data from emerging portable RGB-D cameras provide a good balance for this dilemma. In this paper, we propose an automatic emotion annotation solution on 2.5-D facial data collected from RGB-D cameras. The solution consists of a facial landmarking method and a FER method. Specifically, we propose building a deformable partial face model and fit the model to a 2.5-D face for localizing facial landmarks automatically. In FER, a novel action unit (AU) space-based FER method has been proposed. Facial features are extracted using landmarks and further represented as coordinates in the AU space, which are classified into facial expressions. Evaluated on three publicly accessible facial databases, namely EURECOM, FRGC, and Bosphorus databases, the proposed facial landmarking and expression recognition methods have achieved satisfactory results. Possible real-world applications using our algorithms have also been discussed.

Original languageEnglish (US)
Article number7225131
Pages (from-to)2042-2055
Number of pages14
JournalIEEE Transactions on Cybernetics
Issue number9
StatePublished - Sep 2016


  • Assistive technology (AT)
  • automatic emotion annotation
  • facial expression recognition (FER)
  • facial landmark localization
  • portable RGB-D cameras

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Information Systems
  • Human-Computer Interaction
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Automatic 2.5-D Facial Landmarking and Emotion Annotation for Social Interaction Assistance'. Together they form a unique fingerprint.

Cite this