A Caltech Library Service

Engineering flexible machine learning systems by traversing functionally invariant paths in weight space

Raghavan, Guruprasad and Thomson, Matt (2022) Engineering flexible machine learning systems by traversing functionally invariant paths in weight space. . (Unpublished)

[img] PDF - Submitted Version
Creative Commons Attribution Non-commercial No Derivatives.


Use this Persistent URL to link to this item:


Deep neural networks achieve human-like performance on a variety of perceptual and decision-making tasks. However, networks perform poorly when confronted with changing tasks or goals, and broadly fail to match the flexibility and robustness of human intelligence. Here, we develop a mathematical and algorithmic framework that enables flexible and continuous training of neural networks on a range of objectives by constructing path connected sets of networks that achieve equivalent functional performance on a given machine learning task. We view the weight space of a neural network as a curved Riemannian manifold and move a network along a functionally invariant path in weight space while searching for networks that satisfy secondary objectives. A path-sampling algorithm trains computer vision and natural language processing networks with millions of weight parameters to learn a series of classification tasks without performance loss while accommodating secondary objectives including network sparsification, incremental task learning, and increased adversarial robustness. Broadly, we conceptualize a neural network as a mathematical object that can be iteratively transformed into distinct configurations by the path-sampling algorithm to define a sub-manifold of networks that can be harnessed to achieve user goals.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Raghavan, Guruprasad0000-0002-1970-9963
Thomson, Matt0000-0003-1021-1234
Additional Information:Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0).
Record Number:CaltechAUTHORS:20220816-220025879
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:116327
Deposited By: George Porter
Deposited On:17 Aug 2022 14:02
Last Modified:17 Aug 2022 14:02

Repository Staff Only: item control page