Mad Max: Affine Spline Insights into Deep Learning

Randall Balestriero, Richard G. Baraniuk

Research output: Contribution to journalArticlepeer-review

28 Scopus citations

Abstract

We build a rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators. Our key result is that a large class of DNs can be written as a composition of max-affine spline operators (MASOs) that provide a powerful portal through which we view and analyze their inner workings. For instance, conditioned on the spline partition region containing the input signal, the output of an MASO DN can be written as a simple affine transformation of the input. This implies that a DN constructs a set of signal-dependent, class-specific templates against which the signal is compared via a simple inner product; we explore the links to the classical theory of optimal classification via matched filters and the effects of data memorization. Going further, we propose a simple penalty term that can be added to the cost function of any DN learning algorithm to force the templates to be orthogonal with each other; this leads to significantly improved classification performance and reduced overfitting with no change to the DN architecture. The spline partition of the input signal space that is implicitly induced by an MASO directly links DNs to the theory of vector quantization (VQ) and K-means clustering, which opens up new geometric avenues to study how DNs organize signals in a hierarchical fashion. To validate the utility of the VQ interpretation, we develop and validate a new distance metric for signals and images that quantify the difference between their VQ encodings.

Original languageEnglish (US)
Article number9296823
Pages (from-to)704-727
Number of pages24
JournalProceedings of the IEEE
Volume109
Issue number5
DOIs
StatePublished - May 2021

Keywords

  • Classification
  • continuous piecewise affine
  • deep neural networks
  • input space partition
  • max affine splines
  • template matching; Voronoi diagram

ASJC Scopus subject areas

  • General Computer Science
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'Mad Max: Affine Spline Insights into Deep Learning'. Together they form a unique fingerprint.

Cite this