CaltechAUTHORS
  A Caltech Library Service

Stochastically Rank-Regularized Tensor Regression Networks

Kolbeinsson, Arinbjörn and Kossaifi, Jean and Panagakis, Yannis and Anandkumar, Anima and Tzoulaki, Ioanna and Matthews, Paul (2019) Stochastically Rank-Regularized Tensor Regression Networks. . (Unpublished) http://resolver.caltech.edu/CaltechAUTHORS:20190327-162823824

[img] PDF - Submitted Version
See Usage Policy.

2364Kb

Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechAUTHORS:20190327-162823824

Abstract

Over-parametrization of deep neural networks has recently been shown to be key to their successful training. However, it also renders them prone to overfitting and makes them expensive to store and train. Tensor regression networks significantly reduce the number of effective parameters in deep neural networks while retaining accuracy and the ease of training. They replace the flattening and fully-connected layers with a tensor regression layer, where the regression weights are expressed through the factors of a low-rank tensor decomposition. In this paper, to further improve tensor regression networks, we propose a novel stochastic rank-regularization. It consists of a novel randomized tensor sketching method to approximate the weights of tensor regression layers. We theoretically and empirically establish the link between our proposed stochastic rank-regularization and the dropout on low-rank tensor regression. Extensive experimental results with both synthetic data and real world datasets (i.e., CIFAR-100 and the UK Biobank brain MRI dataset) support that the proposed approach i) improves performance in both classification and regression tasks, ii) decreases overfitting, iii) leads to more stable training and iv) improves robustness to adversarial attacks and random noise.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/1902.10758arXivDiscussion Paper
Additional Information:This research has been conducted using the UK Biobank Resource under Application Number 18545.
Record Number:CaltechAUTHORS:20190327-162823824
Persistent URL:http://resolver.caltech.edu/CaltechAUTHORS:20190327-162823824
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94225
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:28 Mar 2019 15:25
Last Modified:28 Mar 2019 15:32

Repository Staff Only: item control page