Hyperparameter optimization for randomized algorithms: a case study on random features
-
1.
California Institute of Technology
-
2.
Princeton University
Abstract
Randomized algorithms exploit stochasticity to reduce computational complexity. One important example is random feature regression (RFR) that accelerates Gaussian process regression (GPR). RFR approximates an unknown function with a random neural network whose hidden weights and biases are sampled from a probability distribution. Only the final output layer is fit to data. In randomized algorithms like RFR, the hyperparameters that characterize the sampling distribution greatly impact performance, yet are not directly accessible from samples. This makes optimization of hyperparameters via standard (gradient-based) optimization tools inapplicable. Inspired by Bayesian ideas from GPR, this paper introduces a random objective function that is tailored for hyperparameter tuning of vector-valued random features. The objective is minimized with ensemble Kalman inversion (EKI). EKI is a gradient-free particle-based optimizer that is scalable to high-dimensions and robust to randomness in objective functions. A numerical study showcases the new black-box methodology to learn hyperparameter distributions in several problems that are sensitive to the hyperparameter selection: two global sensitivity analyses, integrating a chaotic dynamical system, and solving a Bayesian inverse problem from atmospheric dynamics. The success of the proposed EKI-based algorithm for RFR suggests its potential for automated optimization of hyperparameters arising in other randomized algorithms.
Copyright and License
© 2025, The Author(s), under exclusive licence to Springer Science Business Media, LLC, part of Springer Nature
Acknowledgement
ORAD is supported by Schmidt Sciences, LLC, the National Science Foundation (NSF) under award number AGS-1835860, and the Office of Naval Research (ONR) under award number N00014-23-1-2654. NHN acknowledges support from NSF award number DMS-2402036, the NSF Graduate Research Fellowship Program under award number DGE-1745301, the Amazon/Caltech AI4Science Fellowship, the Air Force Office of Scientific Research under MURI award number FA9550-20-1-0358 (Machine Learning and Physics-Based Modeling and Simulation), and the Department of Defense Vannevar Bush Faculty Fellowship held by Andrew M. Stuart under ONR award number N00014-22-1-2790. The computations presented in this paper were partially conducted on the Resnick High Performance Computing Center, a facility supported by the Resnick Sustainability Institute at the California Institute of Technology. The authors are grateful to Daniel J. Gauthier for helpful comments on a previous version of the paper and to the two anonymous referees for their valuable feedback.
Data Availability
All numerical experiments are reproducible at: https://github.com/CliMA/CalibrateEmulateSample.jl .
Code Availability
Users may interact with this code base as a playground to understand our EKI-based algorithm and how its various setup parameters influence performance in the context of emulator training. Documentation can be found here: :
https://clima.github.io/CalibrateEmulateSample.jl/dev/ .
Readers should navigate to Examples Emulator testing.
Additional details
Funding
- Schmidt Sciences, LLC
- National Science Foundation
- AGS-1835860
- Office of Naval Research
- N00014-23-1-2654
- National Science Foundation
- DMS-2402036
- National Science Foundation
- NSF Graduate Research Fellowship Program DGE-1745301
- California Institute of Technology
- Amazon/Caltech AI4Science Fellowship -
- United States Air Force Office of Scientific Research
- MURI (Machine Learning and Physics-Based Modeling and Simulation) FA9550-20-1-0358
- Office of Naval Research
- Department of Defense Vannevar Bush Faculty Fellowship N00014-22-1-2790
- Resnick Sustainability Institute