Welcome to the new version of CaltechAUTHORS. Login is currently restricted to library staff. If you notice any issues, please email coda@library.caltech.edu
Published October 2022 | Published
Journal Article Open

Mean-field limits of trained weights in deep learning: A dynamical systems perspective


Training a residual neural network with L^2 regularization on weights and biases is equivalent to minimizing a discrete least action principle and to controlling a discrete Hamiltonian system representing the propagation of input data across layers. The kernel/feature map analysis of this Hamiltonian system suggests a mean-field limit for trained weights and biases as the number of data points goes to infinity. The purpose of this paper is to investigate this mean-field limit and illustrate its existence through numerical experiments and analysis (for simple kernels).

Additional Information

© 2022 The Author(s). Volume 15, Special Issue dedicated to Robert Schaback on the occasion of his 75th birthday (2022). Parts of this work were done when B. H. was a Marie Curie fellow at Imperial College London. B. H. thanks the European Commission for funding through the Marie Curie fellowship STALDYS-792919 (Statistical Learning for Dynamical Systems). H. O. gratefully acknowledges support by the Air Force Office of Scientific Research under award number FA9550-18-1-0271 (Games for Computation and Learning). Code. The Python codes of all numerical experiments in this paper are at https://github.com/alexsm98/Mean-field-limit-code-.git

Attached Files

Published - 12_smirnov.pdf


Files (1.7 MB)
Name Size Download all
1.7 MB Preview Download

Additional details

August 20, 2023
October 20, 2023