A Caltech Library Service

Angular Visual Hardness

Chen, Beidi and Liu, Weiyang and Yu, Zhiding and Kautz, Jan and Shrivastava, Anshumali and Garg, Animesh and Anandkumar, Anima (2020) Angular Visual Hardness. Proceedings of Machine Learning Research, 119 . pp. 1637-1648. ISSN 2640-3498.

[img] PDF - Published Version
See Usage Policy.

PDF (10 July 2020) - Submitted Version
See Usage Policy.

[img] PDF - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


Recent convolutional neural networks (CNNs) have led to impressive performance but often suffer from poor calibration. They tend to be overconfident, with the model confidence not always reflecting the underlying true ambiguity and hardness. In this paper, we propose angular visual hardness (AVH), a score given by the normalized angular distance between the sample feature embedding and the target classifier to measure sample hardness. We validate this score with an in-depth and extensive scientific study, and observe that CNN models with the highest accuracy also have the best AVH scores. This agrees with an earlier finding that state-of-art models improve on the classification of harder examples. We observe that the training dynamics of AVH is vastly different compared to the training loss. Specifically, AVH quickly reaches a plateau for all samples even though the training loss keeps improving. This suggests the need for designing better loss functions that can target harder examples more effectively. We also find that AVH has a statistically significant correlation with human visual hardness. Finally, we demonstrate the benefit of AVH to a variety of applications such as self-training for domain adaptation and domain generalization.

Item Type:Article
Related URLs:
URLURL TypeDescription Paper ItemSoftware
Garg, Animesh0000-0003-0482-4296
Additional Information:© 2020 by the author(s). Work done during internship at NVIDIA. We would like to thank Shiyu Liang, Yue Zhu and Yang Zou for the valuable discussions that enlighten our research. We are also grateful to the anonymous reviewers for their constructive comments that significantly helped to improve our paper. Weiyang Liu is partially supported by Baidu scholarship and NVIDIA GPU grant. This work was supported by NSF-1652131, NSF-BIGDATA 1838177, AFOSR-YIPFA9550-18-1-0152, Amazon Research Award, and ONR BRC grant for Randomized Numerical Linear Algebra.
Funding AgencyGrant Number
Baidu ScholarshipUNSPECIFIED
Air Force Office of Scientific Research (AFOSR)FA9550-18-1-0152
Amazon Research AwardUNSPECIFIED
Office of Naval Research (ONR)UNSPECIFIED
Record Number:CaltechAUTHORS:20200109-084932688
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:100577
Deposited By: Tony Diaz
Deposited On:09 Jan 2020 19:47
Last Modified:17 Sep 2021 19:12

Repository Staff Only: item control page