Identifying Human Behaviors Using Synchronized Audio-Visual Cues

Michalis Vrigkas, Christophoros Nikou, Ioannis A. Kakadiaris

Research output: Contribution to journalArticlepeer-review

10 Scopus citations


In this paper, a human behavior recognition method using multimodal features is presented. We focus on modeling individual and social behaviors of a subject (e.g., friendly/aggressive or hugging/kissing behaviors) with a hidden conditional random field (HCRF) in a supervised framework. Each video is represented by a vector of spatio-temporal visual features (STIP, head orientation and proxemic features) along with audio features (MFCCs). We propose a feature pruning method for removing irrelevant and redundant features based on the spatio-temporal neighborhood of each feature in a video sequence. The proposed framework assumes that human movements are highly correlated with sound emissions. For this reason, canonical correlation analysis (CCA) is employed to find correlation between the audio and video features prior to fusion. The experimental results, performed in two human behavior recognition datasets including political speeches and human interactions from TV shows, attest the advantages of the proposed method compared with several baseline and alternative human behavior recognition methods.

Original languageEnglish (US)
Pages (from-to)54-66
Number of pages13
JournalIEEE Transactions on Affective Computing
Issue number1
StatePublished - Jan 1 2017


  • Hidden conditional random fields
  • audio-visual synchronization
  • canonical correlation analysis
  • human behavior recognition
  • multimodal fusion

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction


Dive into the research topics of 'Identifying Human Behaviors Using Synchronized Audio-Visual Cues'. Together they form a unique fingerprint.

Cite this