CaltechAUTHORS
Published February 25, 2025 | Published
Journal Article Metadata-only

Hyperparameter optimization for randomized algorithms: a case study on random features

Dunbar, Oliver R. A.1 ORCID icon
Nelsen, Nicholas H.1 ORCID icon
Mutic, Maya2
  • 1. ROR icon California Institute of Technology
  • 2. ROR icon Princeton University

Abstract

Randomized algorithms exploit stochasticity to reduce computational complexity. One important example is random feature regression (RFR) that accelerates Gaussian process regression (GPR). RFR approximates an unknown function with a random neural network whose hidden weights and biases are sampled from a probability distribution. Only the final output layer is fit to data. In randomized algorithms like RFR, the hyperparameters that characterize the sampling distribution greatly impact performance, yet are not directly accessible from samples. This makes optimization of hyperparameters via standard (gradient-based) optimization tools inapplicable. Inspired by Bayesian ideas from GPR, this paper introduces a random objective function that is tailored for hyperparameter tuning of vector-valued random features. The objective is minimized with ensemble Kalman inversion (EKI). EKI is a gradient-free particle-based optimizer that is scalable to high-dimensions and robust to randomness in objective functions. A numerical study showcases the new black-box methodology to learn hyperparameter distributions in several problems that are sensitive to the hyperparameter selection: two global sensitivity analyses, integrating a chaotic dynamical system, and solving a Bayesian inverse problem from atmospheric dynamics. The success of the proposed EKI-based algorithm for RFR suggests its potential for automated optimization of hyperparameters arising in other randomized algorithms.

Copyright and License

© 2025, The Author(s), under exclusive licence to Springer Science Business Media, LLC, part of Springer Nature

Acknowledgement

ORAD is supported by Schmidt Sciences, LLC, the National Science Foundation (NSF) under award number AGS-1835860, and the Office of Naval Research (ONR) under award number N00014-23-1-2654. NHN acknowledges support from NSF award number DMS-2402036, the NSF Graduate Research Fellowship Program under award number DGE-1745301, the Amazon/Caltech AI4Science Fellowship, the Air Force Office of Scientific Research under MURI award number FA9550-20-1-0358 (Machine Learning and Physics-Based Modeling and Simulation), and the Department of Defense Vannevar Bush Faculty Fellowship held by Andrew M. Stuart under ONR award number N00014-22-1-2790. The computations presented in this paper were partially conducted on the Resnick High Performance Computing Center, a facility supported by the Resnick Sustainability Institute at the California Institute of Technology. The authors are grateful to Daniel J. Gauthier for helpful comments on a previous version of the paper and to the two anonymous referees for their valuable feedback.

Data Availability

All numerical experiments are reproducible at: https://github.com/CliMA/CalibrateEmulateSample.jl .

Code Availability

Users may interact with this code base as a playground to understand our EKI-based algorithm and how its various setup parameters influence performance in the context of emulator training. Documentation can be found here: :

https://clima.github.io/CalibrateEmulateSample.jl/dev/ .

Readers should navigate to Examples  Emulator testing.

Additional details

Funding Caltech Custom Metadata
Created:
July 16, 2025
Modified:
July 16, 2025