A Caltech Library Service

Redundant Information Neural Estimation

Kleinman, Michael and Achille, Alessandro and Soatto, Stefano and Kao, Jonathan C. (2021) Redundant Information Neural Estimation. Entropy, 23 (7). Art. No. 922. ISSN 1099-4300. PMCID PMC8304362. doi:10.3390/e23070922.

[img] PDF - Published Version
Creative Commons Attribution.


Use this Persistent URL to link to this item:


We introduce the Redundant Information Neural Estimator (RINE), a method that allows efficient estimation for the component of information about a target variable that is common to a set of sources, known as the “redundant information”. We show that existing definitions of the redundant information can be recast in terms of an optimization over a family of functions. In contrast to previous information decompositions, which can only be evaluated for discrete variables over small alphabets, we show that optimizing over functions enables the approximation of the redundant information for high-dimensional and continuous predictors. We demonstrate this on high-dimensional image classification and motor-neuroscience tasks.

Item Type:Article
Related URLs:
URLURL TypeDescription CentralArticle ItemCode
Kao, Jonathan C.0000-0002-9298-0143
Additional Information:© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license ( Received: 11 May 2021 / Revised: 1 July 2021 / Accepted: 14 July 2021 / Published: 20 July 2021. We thank the reviewers for the helpful comments and suggestions. We thank Parthe Pandit and Hengjie Yang for helpful discussions. We thank Krishna Shenoy and Sergey Stavisky for permission to use the neural recording data sets from a delayed reach task. M.K. was supported by the National Sciences and Engineering Research Council (NSERC). J.C.K. was supported by an NSF CAREER Award (#1943467). This research was supported by a UCLA Computational Medicine Amazon Web Services Award. Author Contributions: Conceptualization, M.K.; methodology, M.K. and A.A.; software, M.K.; writing—original draft preparation, M.K.; writing—review and editing, M.K., A.A., S.S., J.C.K.; supervision, A.A., S.S., J.C.K. All authors have read and agreed to the published version of the manuscript. Data Availability Statement: Code is available at: The authors declare no conflict of interest.
Funding AgencyGrant Number
Natural Sciences and Engineering Research Council of Canada (NSERC)UNSPECIFIED
Amazon Web ServicesUNSPECIFIED
Issue or Number:7
PubMed Central ID:PMC8304362
Record Number:CaltechAUTHORS:20210803-170853656
Persistent URL:
Official Citation:Kleinman, M.; Achille, A.; Soatto, S.; Kao, J.C. Redundant Information Neural Estimation. Entropy 2021, 23, 922.
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:110134
Deposited By: Tony Diaz
Deposited On:04 Aug 2021 18:26
Last Modified:04 Aug 2021 18:26

Repository Staff Only: item control page