TY - GEN
T1 - Multimodal autoencoder
T2 - 7th International Conference on Affective Computing and Intelligent Interaction, ACII 2017
AU - Jaques, Natasha
AU - Taylor, Sara
AU - Sano, Akane
AU - Picard, Rosalind
N1 - Funding Information:
We would like to thank Dr. Charles Czeisler, Dr. Elizabeth Klerman, Conor O’Brien and other SNAPSHOT project members for their help in running the SNAPSHOT study. This work was supported by the MIT Media Lab Consortium, NIH Grant R01GM105018, Samsung Electronics, and Canada’s NSERC program.
Publisher Copyright:
© 2017 IEEE.
PY - 2017/7/2
Y1 - 2017/7/2
N2 - To accomplish forecasting of mood in real-world situations, affective computing systems need to collect and learn from multimodal data collected over weeks or months of daily use. Such systems are likely to encounter frequent data loss, e.g. when a phone loses location access, or when a sensor is recharging. Lost data can handicap classifiers trained with all modalities present in the data. This paper describes a new technique for handling missing multimodal data using a specialized denoising autoencoder: The Multimodal Autoencoder (MMAE). Empirical results from over 200 participants and 5500 days of data demonstrate that the MMAE is able to predict the feature values from multiple missing modalities more accurately than reconstruction methods such as principal components analysis (PCA). We discuss several practical benefits of the MMAE's encoding and show that it can provide robust mood prediction even when up to three quarters of the data sources are lost.
AB - To accomplish forecasting of mood in real-world situations, affective computing systems need to collect and learn from multimodal data collected over weeks or months of daily use. Such systems are likely to encounter frequent data loss, e.g. when a phone loses location access, or when a sensor is recharging. Lost data can handicap classifiers trained with all modalities present in the data. This paper describes a new technique for handling missing multimodal data using a specialized denoising autoencoder: The Multimodal Autoencoder (MMAE). Empirical results from over 200 participants and 5500 days of data demonstrate that the MMAE is able to predict the feature values from multiple missing modalities more accurately than reconstruction methods such as principal components analysis (PCA). We discuss several practical benefits of the MMAE's encoding and show that it can provide robust mood prediction even when up to three quarters of the data sources are lost.
UR - http://www.scopus.com/inward/record.url?scp=85047360710&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85047360710&partnerID=8YFLogxK
U2 - 10.1109/ACII.2017.8273601
DO - 10.1109/ACII.2017.8273601
M3 - Conference contribution
AN - SCOPUS:85047360710
T3 - 2017 7th International Conference on Affective Computing and Intelligent Interaction, ACII 2017
SP - 202
EP - 208
BT - 2017 7th International Conference on Affective Computing and Intelligent Interaction, ACII 2017
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 23 October 2017 through 26 October 2017
ER -