A Caltech Library Service

Investigating Generalization by Controlling Normalized Margin

Farhang, Alexander R. and Bernstein, Jeremy D. and Tirumala, Kushal and Liu, Yang and Yue, Yisong (2022) Investigating Generalization by Controlling Normalized Margin. Proceedings of Machine Learning Research, 162 . pp. 6324-6336. ISSN 2640-3498.

[img] PDF - Published Version
See Usage Policy.

[img] Archive (ZIP) - Supplemental Material
See Usage Policy.

[img] PDF (ArXiv discussion paper) - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Weight norm ‖w‖ and margin γ participate in learning theory via the normalized margin γ/‖w‖. Since standard neural net optimizers do not control normalized margin, it is hard to test whether this quantity causally relates to generalization. This paper designs a series of experimental studies that explicitly control normalized margin and thereby tackle two central questions. First: does normalized margin always have a causal effect on generalization? The paper finds that no -- networks can be produced where normalized margin has seemingly no relationship with generalization, counter to the theory of Bartlett et al. (2017). Second: does normalized margin ever have a causal effect on generalization? The paper finds that yes -- in a standard training setup, test performance closely tracks normalized margin. The paper suggests a Gaussian process model as a promising explanation for this behavior.

Item Type:Article
Related URLs:
URLURL TypeDescription Paper ItemCode Information
Bernstein, Jeremy D.0000-0001-9110-7476
Liu, Yang0000-0002-8155-9134
Yue, Yisong0000-0001-9127-1989
Additional Information:© 2022 by the authors. The authors are grateful to the anonymous reviewers for their helpful comments. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1745301. This material was also supported by the following grants: NSF #1918865; ONR N00014-21-1-2483.
Funding AgencyGrant Number
NSF Graduate Research FellowshipDGE-1745301
Office of Naval Research (ONR)N00014-21-1-2483
Record Number:CaltechAUTHORS:20220714-212426792
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:115571
Deposited By: George Porter
Deposited On:15 Jul 2022 23:26
Last Modified:15 Jul 2022 23:26

Repository Staff Only: item control page