Welcome to the new version of CaltechAUTHORS. Login is currently restricted to library staff. If you notice any issues, please email coda@library.caltech.edu
Published June 14, 2021 | Submitted
Journal Article Open

Consistency of empirical Bayes and kernel flow for hierarchical parameter estimation


Gaussian process regression has proven very powerful in statistics, machine learning and inverse problems. A crucial aspect of the success of this methodology, in a wide range of applications to complex and real-world problems, is hierarchical modeling and learning of hyperparameters. The purpose of this paper is to study two paradigms of learning hierarchical parameters: one is from the probabilistic Bayesian perspective, in particular, the empirical Bayes approach that has been largely used in Bayesian statistics; the other is from the deterministic and approximation theoretic view, and in particular the kernel flow algorithm that was proposed recently in the machine learning literature. Analysis of their consistency in the large data limit, as well as explicit identification of their implicit bias in parameter learning, are established in this paper for a Matérn-like model on the torus. A particular technical challenge we overcome is the learning of the regularity parameter in the Matérn-like field, for which consistency results have been very scarce in the spatial statistics literature. Moreover, we conduct extensive numerical experiments beyond the Matérn-like model, comparing the two algorithms further. These experiments demonstrate learning of other hierarchical parameters, such as amplitude and lengthscale; they also illustrate the setting of model misspecification in which the kernel flow approach could show superior performance to the more traditional empirical Bayes approach.

Additional Information

© 2021 American Mathematical Society. Received by the editor May 22, 2020, and, in revised form, February 1, 2021. Published electronically: June 14, 2021. The first author gratefully acknowledged the support of the Caltech Kortchack Scholar Program. The second author gratefully acknowledged support from AFOSR (grant FA9550-18-1-0271) and ONR (grant N00014-18-1-2363). The third author was grateful to AFOSR (grant FA9550-17-1-0185) and NSF (grant DMS 18189770) for financial support. The first, second, and third authors gratefully acknowledged support from AFOSR MURI (FA9550-20-1-0358).

Attached Files

Submitted - 2005.11375.pdf


Files (721.8 kB)
Name Size Download all
721.8 kB Preview Download

Additional details

August 20, 2023
March 5, 2024