CaltechAUTHORS
  A Caltech Library Service

Investigating Generalization by Controlling Normalized Margin

Farhang, Alexander R. and Bernstein, Jeremy D. and Tirumala, Kushal and Liu, Yang and Yue, Yisong (2022) Investigating Generalization by Controlling Normalized Margin. Proceedings of Machine Learning Research, 162 . pp. 6324-6336. ISSN 2640-3498. https://resolver.caltech.edu/CaltechAUTHORS:20220714-212426792

[img] PDF - Published Version
See Usage Policy.

462kB
[img] Archive (ZIP) - Supplemental Material
See Usage Policy.

5MB
[img] PDF (ArXiv discussion paper) - Submitted Version
See Usage Policy.

570kB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20220714-212426792

Abstract

Weight norm ‖w‖ and margin γ participate in learning theory via the normalized margin γ/‖w‖. Since standard neural net optimizers do not control normalized margin, it is hard to test whether this quantity causally relates to generalization. This paper designs a series of experimental studies that explicitly control normalized margin and thereby tackle two central questions. First: does normalized margin always have a causal effect on generalization? The paper finds that no -- networks can be produced where normalized margin has seemingly no relationship with generalization, counter to the theory of Bartlett et al. (2017). Second: does normalized margin ever have a causal effect on generalization? The paper finds that yes -- in a standard training setup, test performance closely tracks normalized margin. The paper suggests a Gaussian process model as a promising explanation for this behavior.


Item Type:Article
Related URLs:
URLURL TypeDescription
https://proceedings.mlr.press/v162/farhang22a.htmlPublisherArticle
https://doi.org/10.48550/arXiv.2205.03940arXivDiscussion Paper
https://github.com/alexfarhang/marginRelated ItemCode
https://media.icml.cc/Conferences/ICML2022/supplementary/farhang22a-supp.zipPublisherSupporting Information
ORCID:
AuthorORCID
Bernstein, Jeremy D.0000-0001-9110-7476
Liu, Yang0000-0002-8155-9134
Yue, Yisong0000-0001-9127-1989
Additional Information:© 2022 by the authors. The authors are grateful to the anonymous reviewers for their helpful comments. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1745301. This material was also supported by the following grants: NSF #1918865; ONR N00014-21-1-2483.
Funders:
Funding AgencyGrant Number
NSF Graduate Research FellowshipDGE-1745301
NSFCCF-1918865
Office of Naval Research (ONR)N00014-21-1-2483
Record Number:CaltechAUTHORS:20220714-212426792
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20220714-212426792
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:115571
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:15 Jul 2022 23:26
Last Modified:15 Jul 2022 23:26

Repository Staff Only: item control page