Mhaskar, Hrushikesh N. and Poggio, Tomaso (2016) Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications, 14 (6). pp. 829-848. ISSN 0219-5305. doi:10.1142/S0219530516400042. https://resolver.caltech.edu/CaltechAUTHORS:20161202-084438506
![]() |
PDF
- Submitted Version
See Usage Policy. 983kB |
Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20161202-084438506
Abstract
The paper briefly reviews several recent results on hierarchical architectures for learning from examples, that may formally explain the conditions under which Deep Convolutional Neural Networks perform much better in function approximation problems than shallow, one-hidden layer architectures. The paper announces new results for a non-smooth activation function — the ReLU function — used in present-day neural networks, as well as for the Gaussian networks. We propose a new definition of relative dimension to encapsulate different notions of sparsity of a function class that can possibly be exploited by deep networks but not by shallow ones to drastically reduce the complexity required for approximation and learning.
Item Type: | Article | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Related URLs: |
| ||||||||||||
Additional Information: | © 2016 World Scientific Publishing Co. Received: 7 July 2016; Accepted: 7 August 2016; Published: 14 October 2016. This work was supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216. H.M. is supported in part by ARO Grant W911NF-15-1-0385. | ||||||||||||
Funders: |
| ||||||||||||
Subject Keywords: | Deep and shallow networks; Gaussian networks; ReLU networks; blessed representation | ||||||||||||
Issue or Number: | 6 | ||||||||||||
Classification Code: | AMSC: 68Q32, 41A25, 41A46 | ||||||||||||
DOI: | 10.1142/S0219530516400042 | ||||||||||||
Record Number: | CaltechAUTHORS:20161202-084438506 | ||||||||||||
Persistent URL: | https://resolver.caltech.edu/CaltechAUTHORS:20161202-084438506 | ||||||||||||
Official Citation: | Deep vs. shallow networks: An approximation theory perspective H. N. Mhaskar and T. Poggio Analysis and Applications 2016 14:06, 829-848 | ||||||||||||
Usage Policy: | No commercial reproduction, distribution, display or performance rights in this work are provided. | ||||||||||||
ID Code: | 72517 | ||||||||||||
Collection: | CaltechAUTHORS | ||||||||||||
Deposited By: | Tony Diaz | ||||||||||||
Deposited On: | 02 Dec 2016 22:57 | ||||||||||||
Last Modified: | 11 Nov 2021 05:02 |
Repository Staff Only: item control page