Farhang, Alexander R. and Bernstein, Jeremy D. and Tirumala, Kushal and Liu, Yang and Yue, Yisong (2022) Investigating Generalization by Controlling Normalized Margin. Proceedings of Machine Learning Research, 162 . pp. 6324-6336. ISSN 2640-3498. https://resolver.caltech.edu/CaltechAUTHORS:20220714-212426792
![]() |
PDF
- Published Version
See Usage Policy. 462kB |
![]() |
Archive (ZIP)
- Supplemental Material
See Usage Policy. 5MB |
![]() |
PDF (ArXiv discussion paper)
- Submitted Version
See Usage Policy. 570kB |
Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20220714-212426792
Abstract
Weight norm ‖w‖ and margin γ participate in learning theory via the normalized margin γ/‖w‖. Since standard neural net optimizers do not control normalized margin, it is hard to test whether this quantity causally relates to generalization. This paper designs a series of experimental studies that explicitly control normalized margin and thereby tackle two central questions. First: does normalized margin always have a causal effect on generalization? The paper finds that no -- networks can be produced where normalized margin has seemingly no relationship with generalization, counter to the theory of Bartlett et al. (2017). Second: does normalized margin ever have a causal effect on generalization? The paper finds that yes -- in a standard training setup, test performance closely tracks normalized margin. The paper suggests a Gaussian process model as a promising explanation for this behavior.
Item Type: | Article | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Related URLs: |
| |||||||||||||||
ORCID: |
| |||||||||||||||
Additional Information: | © 2022 by the authors. The authors are grateful to the anonymous reviewers for their helpful comments. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1745301. This material was also supported by the following grants: NSF #1918865; ONR N00014-21-1-2483. | |||||||||||||||
Funders: |
| |||||||||||||||
Record Number: | CaltechAUTHORS:20220714-212426792 | |||||||||||||||
Persistent URL: | https://resolver.caltech.edu/CaltechAUTHORS:20220714-212426792 | |||||||||||||||
Usage Policy: | No commercial reproduction, distribution, display or performance rights in this work are provided. | |||||||||||||||
ID Code: | 115571 | |||||||||||||||
Collection: | CaltechAUTHORS | |||||||||||||||
Deposited By: | George Porter | |||||||||||||||
Deposited On: | 15 Jul 2022 23:26 | |||||||||||||||
Last Modified: | 15 Jul 2022 23:26 |
Repository Staff Only: item control page