Multimodal autoencoder: A deep learning approach to filling in missing sensor data and enabling better mood prediction

Natasha Jaques, Sara Taylor, Akane Sano, Rosalind Picard

Research output: Chapter in Book/Report/Conference proceedingConference contribution

88 Scopus citations

Abstract

To accomplish forecasting of mood in real-world situations, affective computing systems need to collect and learn from multimodal data collected over weeks or months of daily use. Such systems are likely to encounter frequent data loss, e.g. when a phone loses location access, or when a sensor is recharging. Lost data can handicap classifiers trained with all modalities present in the data. This paper describes a new technique for handling missing multimodal data using a specialized denoising autoencoder: The Multimodal Autoencoder (MMAE). Empirical results from over 200 participants and 5500 days of data demonstrate that the MMAE is able to predict the feature values from multiple missing modalities more accurately than reconstruction methods such as principal components analysis (PCA). We discuss several practical benefits of the MMAE's encoding and show that it can provide robust mood prediction even when up to three quarters of the data sources are lost.

Original languageEnglish (US)
Title of host publication2017 7th International Conference on Affective Computing and Intelligent Interaction, ACII 2017
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages202-208
Number of pages7
ISBN (Electronic)9781538605639
DOIs
StatePublished - Jul 2 2017
Event7th International Conference on Affective Computing and Intelligent Interaction, ACII 2017 - San Antonio, United States
Duration: Oct 23 2017Oct 26 2017

Publication series

Name2017 7th International Conference on Affective Computing and Intelligent Interaction, ACII 2017
Volume2018-January

Conference

Conference7th International Conference on Affective Computing and Intelligent Interaction, ACII 2017
Country/TerritoryUnited States
CitySan Antonio
Period10/23/1710/26/17

ASJC Scopus subject areas

  • Behavioral Neuroscience
  • Social Psychology
  • Artificial Intelligence
  • Human-Computer Interaction

Fingerprint

Dive into the research topics of 'Multimodal autoencoder: A deep learning approach to filling in missing sensor data and enabling better mood prediction'. Together they form a unique fingerprint.

Cite this