CaltechAUTHORS
  A Caltech Library Service

Angular Visual Hardness

Chen, Beidi and Liu, Weiyang and Yu, Zhiding and Kautz, Jan and Shrivastava, Anshumali and Garg, Animesh and Anandkumar, Anima (2019) Angular Visual Hardness. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20200109-084932688

[img]
Preview
PDF (10 July 2020) - Submitted Version
See Usage Policy.

6Mb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20200109-084932688

Abstract

Recent convolutional neural networks (CNNs) have led to impressive performance but often suffer from poor calibration. They tend to be overconfident, with the model confidence not always reflecting the underlying true ambiguity and hardness. In this paper, we propose angular visual hardness (AVH), a score given by the normalized angular distance between the sample feature embedding and the target classifier to measure sample hardness. We validate this score with an in-depth and extensive scientific study, and observe that CNN models with the highest accuracy also have the best AVH scores. This agrees with an earlier finding that state-of-art models improve on the classification of harder examples. We observe that the training dynamics of AVH is vastly different compared to the training loss. Specifically, AVH quickly reaches a plateau for all samples even though the training loss keeps improving. This suggests the need for designing better loss functions that can target harder examples more effectively. We also find that AVH has a statistically significant correlation with human visual hardness. Finally, we demonstrate the benefit of AVH to a variety of applications such as self-training for domain adaptation and domain generalization.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
https://arxiv.org/abs/1912.02279arXivDiscussion Paper
ORCID:
AuthorORCID
Garg, Animesh0000-0003-0482-4296
Additional Information:Work done during internship at NVIDIA. We would like to thank Shiyu Liang, Yue Zhu and Yang Zou for the valuable discussions that enlighten our research. We are also grateful to the anonymous reviewers for their constructive comments that significantly helped to improve our paper. Weiyang Liu is partially supported by Baidu scholarship and NVIDIA GPU grant. This work was supported by NSF-1652131, NSF-BIGDATA 1838177, AFOSR-YIPFA9550-18-1-0152, Amazon Research Award, and ONR BRC grant for Randomized Numerical Linear Algebra.
Funders:
Funding AgencyGrant Number
Baidu ScholarshipUNSPECIFIED
NVIDIA CorporationUNSPECIFIED
NSFIIS-1652131
NSFIIS-1838177
Air Force Office of Scientific Research (AFOSR)FA9550-18-1-0152
Amazon Research AwardUNSPECIFIED
Office of Naval Research (ONR)UNSPECIFIED
Record Number:CaltechAUTHORS:20200109-084932688
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20200109-084932688
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:100577
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:09 Jan 2020 19:47
Last Modified:25 Nov 2020 00:00

Repository Staff Only: item control page