A Caltech Library Service

Learning compositional functions via multiplicative weight updates

Bernstein, Jeremy and Zhao, Jiawei and Meister, Markus and Liu, Ming-Yu and Anandkumar, Anima and Yue, Yisong (2020) Learning compositional functions via multiplicative weight updates. In: Advances in Neural Information Processing Systems 33 pre-proceedings (NeurIPS 2020). Advances in Neural Information Processing Systems .

[img] PDF - Published Version
See Usage Policy.

[img] PDF - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


Compositionality is a basic structural feature of both biological and artificial neural networks. Learning compositional functions via gradient descent incurs well known problems like vanishing and exploding gradients, making careful learning rate tuning essential for real-world applications. This paper proves that multiplicative weight updates satisfy a descent lemma tailored to compositional functions. Based on this lemma, we derive Madam—a multiplicative version of the Adam optimiser—and show that it can train state of the art neural network architectures without learning rate tuning. We further show that Madam is easily adapted to train natively compressed neural networks by representing their weights in a logarithmic number system. We conclude by drawing connections between multiplicative weight updates and recent findings about synapses in biology.

Item Type:Book Section
Related URLs:
URLURL TypeDescription Paper
Bernstein, Jeremy0000-0001-9110-7476
Meister, Markus0000-0003-2136-6506
Liu, Ming-Yu0000-0002-2951-2398
Anandkumar, Anima0000-0002-6974-6797
Yue, Yisong0000-0001-9127-1989
Additional Information:The authors would like to thank the anonymous reviewers for their helpful comments. JB was supported by an NVIDIA fellowship. The work was partly supported by funding from NASA.
Funding AgencyGrant Number
Record Number:CaltechAUTHORS:20201106-120208748
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:106488
Deposited By: George Porter
Deposited On:06 Nov 2020 22:26
Last Modified:02 Jun 2023 01:08

Repository Staff Only: item control page