Early versus Late Modality Fusion of Deep Wearable Sensor Features for Personalized Prediction of Tomorrow's Mood, Health, and Stress

Boning Li, Akane Sano

Research output: Chapter in Book/Report/Conference proceedingConference contribution

10 Scopus citations

Abstract

Predicting mood, health, and stress can sound an early alarm against mental illness. Multi-modal data from wearable sensors provide rigorous and rich insights into one's internal states. Recently, deep learning-based features on continuous high-resolution sensor data have outperformed statistical features in several ubiquitous and affective computing applications including sleep detection and depression diagnosis. Motivated by this, we investigate multi-modal data fusion strategies featuring deep representation learning of skin conductance, skin temperature, and acceleration data to predict self-reported mood, health, and stress scores (0 - 100) of college students (N = 239). Our cross-validated results from the early fusion framework exhibit a significantly higher (p < 0.05) prediction precision over the late fusion for unseen users. Therefore, our findings call attention to the benefits of fusing physiological data modalities at a low level and corroborate the predictive efficacy of the deeply learned features.Clinical relevance - This establishes that with automatically extracted features from multiple sensor modalities, choosing the proper scheme of fusion can reduce the errors of predicting new users' future wellbeing by as much as 13.2%.

Original languageEnglish (US)
Title of host publication42nd Annual International Conferences of the IEEE Engineering in Medicine and Biology Society
Subtitle of host publicationEnabling Innovative Technologies for Global Healthcare, EMBC 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages5896-5899
Number of pages4
ISBN (Electronic)9781728119908
DOIs
StatePublished - Jul 2020
Event42nd Annual International Conferences of the IEEE Engineering in Medicine and Biology Society, EMBC 2020 - Montreal, Canada
Duration: Jul 20 2020Jul 24 2020

Publication series

NameProceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS
Volume2020-July
ISSN (Print)1557-170X

Conference

Conference42nd Annual International Conferences of the IEEE Engineering in Medicine and Biology Society, EMBC 2020
Country/TerritoryCanada
CityMontreal
Period7/20/207/24/20

ASJC Scopus subject areas

  • Signal Processing
  • Biomedical Engineering
  • Computer Vision and Pattern Recognition
  • Health Informatics

Fingerprint

Dive into the research topics of 'Early versus Late Modality Fusion of Deep Wearable Sensor Features for Personalized Prediction of Tomorrow's Mood, Health, and Stress'. Together they form a unique fingerprint.

Cite this