A Caltech Library Service

Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation

Chen, Yifang and Owhadi, Houman and Stuart, Andrew M. (2021) Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation. Mathematics of Computation, 90 . pp. 2527-2578. ISSN 0025-5718. doi:10.1090/mcom/3649.

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Gaussian process regression has proven very powerful in statistics, machine learning and inverse problems. A crucial aspect of the success of this methodology, in a wide range of applications to complex and real-world problems, is hierarchical modeling and learning of hyperparameters. The purpose of this paper is to study two paradigms of learning hierarchical parameters: one is from the probabilistic Bayesian perspective, in particular, the empirical Bayes approach that has been largely used in Bayesian statistics; the other is from the deterministic and approximation theoretic view, and in particular the kernel flow algorithm that was proposed recently in the machine learning literature. Analysis of their consistency in the large data limit, as well as explicit identification of their implicit bias in parameter learning, are established in this paper for a Matérn-like model on the torus. A particular technical challenge we overcome is the learning of the regularity parameter in the Matérn-like field, for which consistency results have been very scarce in the spatial statistics literature. Moreover, we conduct extensive numerical experiments beyond the Matérn-like model, comparing the two algorithms further. These experiments demonstrate learning of other hierarchical parameters, such as amplitude and lengthscale; they also illustrate the setting of model misspecification in which the kernel flow approach could show superior performance to the more traditional empirical Bayes approach.

Item Type:Article
Related URLs:
URLURL TypeDescription Paper
Owhadi, Houman0000-0002-5677-1600
Additional Information:© 2021 American Mathematical Society. Received by the editor May 22, 2020, and, in revised form, February 1, 2021. Published electronically: June 14, 2021. The first author gratefully acknowledged the support of the Caltech Kortchack Scholar Program. The second author gratefully acknowledged support from AFOSR (grant FA9550-18-1-0271) and ONR (grant N00014-18-1-2363). The third author was grateful to AFOSR (grant FA9550-17-1-0185) and NSF (grant DMS 18189770) for financial support. The first, second, and third authors gratefully acknowledged support from AFOSR MURI (FA9550-20-1-0358).
Funding AgencyGrant Number
Kortschak Scholars ProgramUNSPECIFIED
Air Force Office of Scientific Research (AFOSR)FA9550-18-1-0271
Office of Naval Research (ONR)N00014-18-1-2363
Air Force Office of Scientific Research (AFOSR)FA9550-17-1-0185
Air Force Office of Scientific Research (AFOSR)FA9550-20-1-0358
Classification Code:2020 Mathematics Subject Classification. Primary 62C10, 41A05, 35Q62
Record Number:CaltechAUTHORS:20201109-141002843
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:106560
Deposited By: George Porter
Deposited On:09 Nov 2020 22:45
Last Modified:14 Sep 2021 22:59

Repository Staff Only: item control page