CaltechAUTHORS
  A Caltech Library Service

Do ideas have shape? Plato's theory of forms as the continuous limit of artificial neural networks

Owhadi, Houman (2020) Do ideas have shape? Plato's theory of forms as the continuous limit of artificial neural networks. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20201109-155524397

[img] PDF - Submitted Version
See Usage Policy.

3472Kb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20201109-155524397

Abstract

We show that ResNets converge, in the infinite depth limit, to a generalization of image registration algorithms. In this generalization, images are replaced by abstractions (ideas) living in high dimensional RKHS spaces, and material points are replaced by data points. Whereas computational anatomy aligns images via deformations of the material space, this generalization aligns ideas by via transformations of their RKHS. This identification of ResNets as idea registration algorithms has several remarkable consequences. The search for good architectures can be reduced to that of good kernels, and we show that the composition of idea registration blocks with reduced equivariant multi-channel kernels (introduced here) recovers and generalizes CNNs to arbitrary spaces and groups of transformations. Minimizers of L2 regularized ResNets satisfy a discrete least action principle implying the near preservation of the norm of weights and biases across layers. The parameters of trained ResNets can be identified as solutions of an autonomous Hamiltonian system defined by the activation function and the architecture of the ANN. Momenta variables provide a sparse representation of the parameters of a ResNet. The registration regularization strategy provides a provably robust alternative to Dropout for ANNs. Pointwise RKHS error estimates lead to deterministic error estimates for ANNs.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/2008.03920arXivDiscussion Paper
ORCID:
AuthorORCID
Owhadi, Houman0000-0002-5677-1600
Additional Information:The author gratefully acknowledges support by the Air Force Office of Scientific Research under award number FA9550-18-1-0271 (Games for Computation and Learning). Thanks to Clint Scovel for a careful readthrough with detailed comments and feedback.
Funders:
Funding AgencyGrant Number
Air Force Office of Scientific Research (AFOSR)FA9550-18-1-0271
Record Number:CaltechAUTHORS:20201109-155524397
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20201109-155524397
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:106569
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:10 Nov 2020 15:08
Last Modified:10 Nov 2020 15:08

Repository Staff Only: item control page