THE RECURRENT NEURAL TANGENT KERNEL

Sina Alemohammad; Zichao Wang; Randall Balestriero; Richard G. Baraniuk

THE RECURRENT NEURAL TANGENT KERNEL

Sina Alemohammad, Zichao Wang, Randall Balestriero, Richard G. Baraniuk

Research output: Contribution to conference › Paper › peer-review

Abstract

The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key DNN architecture remains to be kernelized, namely, the recurrent neural network (RNN). In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs. A key property of the RNTK should greatly benefit practitioners is its ability to compare inputs of different length. To this end, we characterize how the RNTK weights different time steps to form its output under different initialization parameters and nonlinearity choices. A synthetic and 56 real-world data experiments demonstrate that the RNTK offers significant performance gains over other kernels, including standard NTKs, across a wide array of data sets.

Original language	English (US)
State	Published - 2021
Event	9th International Conference on Learning Representations, ICLR 2021 - Virtual, Online Duration: May 3 2021 → May 7 2021

Conference

Conference	9th International Conference on Learning Representations, ICLR 2021
City	Virtual, Online
Period	5/3/21 → 5/7/21

ASJC Scopus subject areas

Language and Linguistics
Computer Science Applications
Education
Linguistics and Language

Cite this

@conference{8960b74f752045c292e725a9d33b12b6,

title = "THE RECURRENT NEURAL TANGENT KERNEL",

abstract = "The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key DNN architecture remains to be kernelized, namely, the recurrent neural network (RNN). In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs. A key property of the RNTK should greatly benefit practitioners is its ability to compare inputs of different length. To this end, we characterize how the RNTK weights different time steps to form its output under different initialization parameters and nonlinearity choices. A synthetic and 56 real-world data experiments demonstrate that the RNTK offers significant performance gains over other kernels, including standard NTKs, across a wide array of data sets.",

author = "Sina Alemohammad and Zichao Wang and Randall Balestriero and Baraniuk, {Richard G.}",

note = "Funding Information: This work was supported by NSF grants CCF-1911094, IIS-1838177, and IIS-1730574; ONR grants N00014-18-12571, N00014-20-1-2787, and N00014-20-1-2534; AFOSR grant FA9550-18-1-0478; and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047. Publisher Copyright: {\textcopyright} 2021 ICLR 2021 - 9th International Conference on Learning Representations. All rights reserved.; 9th International Conference on Learning Representations, ICLR 2021 ; Conference date: 03-05-2021 Through 07-05-2021",

year = "2021",

language = "English (US)",

}

TY - CONF

T1 - THE RECURRENT NEURAL TANGENT KERNEL

AU - Alemohammad, Sina

AU - Wang, Zichao

AU - Balestriero, Randall

AU - Baraniuk, Richard G.

N1 - Funding Information: This work was supported by NSF grants CCF-1911094, IIS-1838177, and IIS-1730574; ONR grants N00014-18-12571, N00014-20-1-2787, and N00014-20-1-2534; AFOSR grant FA9550-18-1-0478; and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047. Publisher Copyright: © 2021 ICLR 2021 - 9th International Conference on Learning Representations. All rights reserved.

PY - 2021

Y1 - 2021

N2 - The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key DNN architecture remains to be kernelized, namely, the recurrent neural network (RNN). In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs. A key property of the RNTK should greatly benefit practitioners is its ability to compare inputs of different length. To this end, we characterize how the RNTK weights different time steps to form its output under different initialization parameters and nonlinearity choices. A synthetic and 56 real-world data experiments demonstrate that the RNTK offers significant performance gains over other kernels, including standard NTKs, across a wide array of data sets.

AB - The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key DNN architecture remains to be kernelized, namely, the recurrent neural network (RNN). In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs. A key property of the RNTK should greatly benefit practitioners is its ability to compare inputs of different length. To this end, we characterize how the RNTK weights different time steps to form its output under different initialization parameters and nonlinearity choices. A synthetic and 56 real-world data experiments demonstrate that the RNTK offers significant performance gains over other kernels, including standard NTKs, across a wide array of data sets.

UR - http://www.scopus.com/inward/record.url?scp=85150306992&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85150306992&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85150306992

T2 - 9th International Conference on Learning Representations, ICLR 2021

Y2 - 3 May 2021 through 7 May 2021

ER -

THE RECURRENT NEURAL TANGENT KERNEL

Abstract

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this