Raghavan, Guruprasad and Thomson, Matt (2022) Engineering flexible machine learning systems by traversing functionally invariant paths in weight space. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20220816-220025879
![]() |
PDF
- Submitted Version
Creative Commons Attribution Non-commercial No Derivatives. 3MB |
Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20220816-220025879
Abstract
Deep neural networks achieve human-like performance on a variety of perceptual and decision-making tasks. However, networks perform poorly when confronted with changing tasks or goals, and broadly fail to match the flexibility and robustness of human intelligence. Here, we develop a mathematical and algorithmic framework that enables flexible and continuous training of neural networks on a range of objectives by constructing path connected sets of networks that achieve equivalent functional performance on a given machine learning task. We view the weight space of a neural network as a curved Riemannian manifold and move a network along a functionally invariant path in weight space while searching for networks that satisfy secondary objectives. A path-sampling algorithm trains computer vision and natural language processing networks with millions of weight parameters to learn a series of classification tasks without performance loss while accommodating secondary objectives including network sparsification, incremental task learning, and increased adversarial robustness. Broadly, we conceptualize a neural network as a mathematical object that can be iteratively transformed into distinct configurations by the path-sampling algorithm to define a sub-manifold of networks that can be harnessed to achieve user goals.
Item Type: | Report or Paper (Discussion Paper) | ||||||
---|---|---|---|---|---|---|---|
Related URLs: |
| ||||||
ORCID: |
| ||||||
Additional Information: | Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0). | ||||||
Record Number: | CaltechAUTHORS:20220816-220025879 | ||||||
Persistent URL: | https://resolver.caltech.edu/CaltechAUTHORS:20220816-220025879 | ||||||
Usage Policy: | No commercial reproduction, distribution, display or performance rights in this work are provided. | ||||||
ID Code: | 116327 | ||||||
Collection: | CaltechAUTHORS | ||||||
Deposited By: | George Porter | ||||||
Deposited On: | 17 Aug 2022 14:02 | ||||||
Last Modified: | 17 Aug 2022 14:02 |
Repository Staff Only: item control page