Multimodal Ambulatory Sleep Detection Using LSTM Recurrent Neural Networks

Akane Sano, Weixuan Chen, Daniel Lopez-Martinez, Sara Taylor, Rosalind W. Picard

Research output: Contribution to journalArticlepeer-review

31 Scopus citations


Unobtrusive and accurate ambulatory methods are needed to monitor long-term sleep patterns for improving health. Previously developed ambulatory sleep detection methods rely either in whole or in part on self-reported diary data as ground truth, which is a problem, since people often do not fill them out accurately. This paper presents an algorithm that uses multimodal data from smartphones and wearable technologies to detect sleep/wake state and sleep onset/offset using a type of recurrent neural network with long-short-term memory (LSTM) cells for synthesizing temporal information. We collected 5580 days of multimodal data from 186 participants and compared the new method for sleep/wake classification and sleep onset/offset detection to, first, nontemporal machine learning methods and, second, a state-of-the-art actigraphy software. The new LSTM method achieved a sleep/wake classification accuracy of 96.5%, and sleep onset/offset detection F1 scores of 0.86 and 0.84, respectively, with mean absolute errors of 5.0 and 5.5 min, res-pectively, when compared with sleep/wake state and sleep onset/offset assessed using actigraphy and sleep diaries. The LSTM results were statistically superior to those from nontemporal machine learning algorithms and the actigraphy software. We show good generalization of the new algorithm by comparing participant-dependent and participant-independent models, and we show how to make the model nearly realtime with slightly reduced performance.

Original languageEnglish (US)
Article number8449917
Pages (from-to)1607-1617
Number of pages11
JournalIEEE Journal of Biomedical and Health Informatics
Issue number4
StatePublished - Jul 2019


  • long-short-term memory
  • LSTM
  • mobile health
  • mobile phone
  • recurrent neural networks
  • sleep detection
  • Sleep monitoring
  • smartphone
  • wearable sensor

ASJC Scopus subject areas

  • Biotechnology
  • Computer Science Applications
  • Electrical and Electronic Engineering
  • Health Information Management


Dive into the research topics of 'Multimodal Ambulatory Sleep Detection Using LSTM Recurrent Neural Networks'. Together they form a unique fingerprint.

Cite this