CaltechAUTHORS
  A Caltech Library Service

Training Input-Output Recurrent Neural Networks through Spectral Methods

Sedghi, Hanie and Anandkumar, Anima (2016) Training Input-Output Recurrent Neural Networks through Spectral Methods. . (Unpublished) http://resolver.caltech.edu/CaltechAUTHORS:20190401-123315920

[img] PDF - Submitted Version
See Usage Policy.

362Kb

Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechAUTHORS:20190401-123315920

Abstract

We consider the problem of training input-output recurrent neural networks (RNN) for sequence labeling tasks. We propose a novel spectral approach for learning the network parameters. It is based on decomposition of the cross-moment tensor between the output and a non-linear transformation of the input, based on score functions. We guarantee consistent learning with polynomial sample and computational complexity under transparent conditions such as non-degeneracy of model parameters, polynomial activations for the neurons, and a Markovian evolution of the input sequence. We also extend our results to Bidirectional RNN which uses both previous and future information to output the label at each time point, and is employed in many NLP tasks such as POS tagging.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/1603.00954arXivDiscussion Paper
Additional Information:The authors thank Majid Janzamin for discussions on sample complexity and constructive comments on the draft. We thank Ashish Sabharwal for editorial comments on the draft. This work was done during the time H. Sedghi was a visiting researcher at University of California, Irvine and was supported by NSF Career award FG15890. A. Anandkumar is supported in part by Microsoft Faculty Fellowship, NSF Career award CCF-1254106, ONR award N00014-14-1-0665, ARO YIP award W911NF-13-1-0084, and AFOSR YIP award FA9550-15-1-0221.
Funders:
Funding AgencyGrant Number
NSFFG15890
Microsoft Faculty FellowshipUNSPECIFIED
NSFCCF-1254106
Office of Naval Research (ONR)N00014-14-1-0665
Army Research Office (ARO)W911NF-13-1-0084
Air Force Office of Scientific Research (AFOSR)FA9550-15-1-0221
Subject Keywords:Recurrent neural networks, sequence labeling, spectral methods, score function
Record Number:CaltechAUTHORS:20190401-123315920
Persistent URL:http://resolver.caltech.edu/CaltechAUTHORS:20190401-123315920
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94325
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:01 Apr 2019 22:11
Last Modified:01 Apr 2019 22:11

Repository Staff Only: item control page