Joint prototype and metric learning for image set classification: Application to video face identification

Mengjun Leng, Panagiotis Moutafis, Ioannis A. Kakadiaris

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


In this paper, we address the problem of image set classification, where each set contains a different number of images acquired from the same subject. In most of the existing literature, each image set is modeled using all its available samples. As a result, the corresponding time and storage costs are high. To address this problem, we propose a joint prototype and metric learning approach. The prototypes are learned to represent each gallery image set using fewer samples without affecting the recognition performance. A Mahalanobis metric is learned simultaneously to measure the similarity between sets more accurately. In particular, each gallery set is represented as a regularized affine hull spanned by the learned prototypes. The set-to-set distance is optimized via updating the prototypes and the Mahalanobis metric in an alternating manner. To highlight the importance of representing image sets using fewer samples, we analyzed the corresponding test time complexity with respect to the number of images used per set. Experimental results using YouTube Celebrity, YouTube Faces, and ETH-80 datasets illustrate the efficiency on the task of video face recognition, and object categorization.

Original languageEnglish (US)
Pages (from-to)204-213
Number of pages10
JournalImage and Vision Computing
StatePublished - Feb 1 2017


  • Image set classification
  • Metric learning
  • Prototype learning
  • Video face recognition

ASJC Scopus subject areas

  • Signal Processing
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Joint prototype and metric learning for image set classification: Application to video face identification'. Together they form a unique fingerprint.

Cite this