CaltechAUTHORS
  A Caltech Library Service

Learning compositional functions via multiplicative weight updates

Bernstein, Jeremy and Zhao, Jiawei and Meister, Markus and Liu, Ming-Yu and Anandkumar, Anima and Yue, Yisong (2020) Learning compositional functions via multiplicative weight updates. In: Advances in Neural Information Processing Systems 33 pre-proceedings (NeurIPS 2020). Advances in Neural Information Processing Systems . https://resolver.caltech.edu/CaltechAUTHORS:20201106-120208748

[img] PDF - Published Version
See Usage Policy.

851Kb
[img] PDF - Supplemental Material
See Usage Policy.

336Kb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20201106-120208748

Abstract

Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam—a multiplicative version of the Adam optimiser—and show that it can train state of the art neural network architectures without learning rate tuning. We further show that Madam is easily adapted to train natively compressed neural networks by representing their weights in a logarithmic number system. We conclude by drawing connections between multiplicative weight updates and recent findings about synapses in biology.


Item Type:Book Section
Related URLs:
URLURL TypeDescription
https://proceedings.neurips.cc/paper/2020/hash/9a32ef65c42085537062753ec435750f-Abstract.htmlPublisherArticle
https://arxiv.org/abs/2006.14560arXivDiscussion Paper
ORCID:
AuthorORCID
Bernstein, Jeremy0000-0001-9110-7476
Meister, Markus0000-0003-2136-6506
Liu, Ming-Yu0000-0002-2951-2398
Yue, Yisong0000-0001-9127-1989
Additional Information:The authors would like to thank the anonymous reviewers for their helpful comments. JB was supported by an NVIDIA fellowship. The work was partly supported by funding from NASA.
Funders:
Funding AgencyGrant Number
NVIDIA CorporationUNSPECIFIED
NASAUNSPECIFIED
Record Number:CaltechAUTHORS:20201106-120208748
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20201106-120208748
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:106488
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:06 Nov 2020 22:26
Last Modified:06 Nov 2020 23:38

Repository Staff Only: item control page