Minipatch learning as implicit ridge-like regularization

Tianyi Yao, Daniel Lejeune, Hamid Javadi, Richard G. Baraniuk, Genevera I. Allen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Ridge-like regularization often leads to improved generalization performance of machine learning models by mitigating overfitting. While ridge-regularized machine learning methods are widely used in many important applications, direct training via optimization could become challenging in huge data scenarios with millions of examples and features. We tackle such challenges by proposing a general approach that achieves ridge-like regularization through implicit techniques named Minipatch Ridge (MPRidge). Our approach is based on taking an ensemble of coefficients of unregularized learners trained on many tiny, random subsamples of both the examples and features of the training data, which we call minipatches. We empirically demonstrate that MPRidge induces an implicit ridge-like regularizing effect and performs nearly the same as explicit ridge regularization for a general class of predictors including logistic regression, SVM, and robust regression. Embarrassingly parallelizable, MPRidge provides a computationally appealing alternative to inducing ridge-like regularization for improving generalization performance in challenging big-data settings.

Original languageEnglish (US)
Title of host publicationProceedings - 2021 IEEE International Conference on Big Data and Smart Computing, BigComp 2021
EditorsHerwig Unger, Jinho Kim, U Kang, Chakchai So-In, Junping Du, Walid Saad, Young-guk Ha, Christian Wagner, Julien Bourgeois, Chanboon Sathitwiriyawong, Hyuk-Yoon Kwon, Carson Leung
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages65-68
Number of pages4
ISBN (Electronic)9781728189246
DOIs
StatePublished - Jan 2021
Event2021 IEEE International Conference on Big Data and Smart Computing, BigComp 2021 - Jeju Island, Korea, Republic of
Duration: Jan 17 2021Jan 20 2021

Publication series

NameProceedings - 2021 IEEE International Conference on Big Data and Smart Computing, BigComp 2021

Conference

Conference2021 IEEE International Conference on Big Data and Smart Computing, BigComp 2021
CountryKorea, Republic of
CityJeju Island
Period1/17/211/20/21

Keywords

  • Ensemble learning
  • Implicit regularization
  • Ridge-like regularization

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Information Systems
  • Signal Processing
  • Information Systems and Management

Fingerprint Dive into the research topics of 'Minipatch learning as implicit ridge-like regularization'. Together they form a unique fingerprint.

Cite this