A probabilistic framework for deep learning

Ankit B. Patel, Tan Nguyen, Richard G. Baraniuk

Research output: Chapter in Book/Report/Conference proceedingConference contribution

22 Scopus citations

Abstract

We develop a probabilistic framework for deep learning based on the Deep Rendering Mixture Model (DRMM), a new generative probabilistic model that explicitly capture variations in data due to latent task nuisance variables. We demonstrate that max-sum inference in the DRMM yields an algorithm that exactly reproduces the operations in deep convolutional neural networks (DCNs), providing a first principles derivation. Our framework provides new insights into the successes and shortcomings of DCNs as well as a principled route to their improvement. DRMM training via the Expectation-Maximization (EM) algorithm is a powerful alternative to DCN back-propagation, and initial training results are promising. Classification based on the DRMM and other variants outperforms DCNs in supervised digit classification, training 2-3× faster while achieving similar accuracy. Moreover, the DRMM is applicable to semi-supervised and unsupervised learning tasks, achieving results that are state-of-the-art in several categories on the MNIST benchmark and comparable to state of the art on the CIFAR10 benchmark.

Original languageEnglish (US)
Title of host publicationAdvances in Neural Information Processing Systems
Pages2558-2566
Number of pages9
StatePublished - 2016
Event30th Annual Conference on Neural Information Processing Systems: NIPS 2016 - Barcelona; , Spain
Duration: Dec 5 2016Dec 10 2016

Conference

Conference30th Annual Conference on Neural Information Processing Systems
Abbreviated titleNIPS
CountrySpain
CityBarcelona;
Period12/5/1612/10/16

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Signal Processing

Fingerprint Dive into the research topics of 'A probabilistic framework for deep learning'. Together they form a unique fingerprint.

Cite this