CaltechAUTHORS
  A Caltech Library Service

Deep vs. shallow networks: An approximation theory perspective

Mhaskar, Hrushikesh N. and Poggio, Tomaso (2016) Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications, 14 (6). pp. 829-848. ISSN 0219-5305. doi:10.1142/S0219530516400042. https://resolver.caltech.edu/CaltechAUTHORS:20161202-084438506

[img] PDF - Submitted Version
See Usage Policy.

983kB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20161202-084438506

Abstract

The paper briefly reviews several recent results on hierarchical architectures for learning from examples, that may formally explain the conditions under which Deep Convolutional Neural Networks perform much better in function approximation problems than shallow, one-hidden layer architectures. The paper announces new results for a non-smooth activation function — the ReLU function — used in present-day neural networks, as well as for the Gaussian networks. We propose a new definition of relative dimension to encapsulate different notions of sparsity of a function class that can possibly be exploited by deep networks but not by shallow ones to drastically reduce the complexity required for approximation and learning.


Item Type:Article
Related URLs:
URLURL TypeDescription
http://dx.doi.org/10.1142/S0219530516400042DOIArticle
http://www.worldscientific.com/doi/abs/10.1142/S0219530516400042PublisherArticle
https://arxiv.org/abs/1608.03287arXivDiscussion Paper
Additional Information:© 2016 World Scientific Publishing Co. Received: 7 July 2016; Accepted: 7 August 2016; Published: 14 October 2016. This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. H.M. is supported in part by ARO Grant W911NF-15-1-0385.
Funders:
Funding AgencyGrant Number
NSFCCF-1231216
Army Research Office (ARO)W911NF-15-1-0385
Subject Keywords:Deep and shallow networks; Gaussian networks; ReLU networks; blessed representation
Issue or Number:6
Classification Code:AMSC: 68Q32, 41A25, 41A46
DOI:10.1142/S0219530516400042
Record Number:CaltechAUTHORS:20161202-084438506
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20161202-084438506
Official Citation:Deep vs. shallow networks: An approximation theory perspective H. N. Mhaskar and T. Poggio Analysis and Applications 2016 14:06, 829-848
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:72517
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:02 Dec 2016 22:57
Last Modified:11 Nov 2021 05:02

Repository Staff Only: item control page