CaltechAUTHORS
  A Caltech Library Service

Decision Theoretic Bootstrapping

Tavallali, Peyman and Bajgiran, Hamed Hamze and Esaid, Danial J. and Owhadi, Houman (2021) Decision Theoretic Bootstrapping. . (Submitted) https://resolver.caltech.edu/CaltechAUTHORS:20210323-130821498

[img] PDF - Submitted Version
Creative Commons Attribution.

759kB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20210323-130821498

Abstract

The design and testing of supervised machine learning models combine two fundamental distributions: (1) the training data distribution (2) the testing data distribution. Although these two distributions are identical and identifiable when the data set is infinite; they are imperfectly known (and possibly distinct) when the data is finite (and possibly corrupted) and this uncertainty must be taken into account for robust Uncertainty Quantification (UQ). We present a general decision-theoretic bootstrapping solution to this problem: (1) partition the available data into a training subset and a UQ subset (2) take m subsampled subsets of the training set and train m models (3) partition the UQ set into n sorted subsets and take a random fraction of them to define n corresponding empirical distributions μ_j (4) consider the adversarial game where Player I selects a model i∈{1,…,m}, Player II selects the UQ distribution μ_j and Player I receives a loss defined by evaluating the model i against data points sampled from μ_j (5) identify optimal mixed strategies (probability distributions over models and UQ distributions) for both players. These randomized optimal mixed strategies provide optimal model mixtures and UQ estimates given the adversarial uncertainty of the training and testing distributions represented by the game. The proposed approach provides (1) some degree of robustness to distributional shift in both the distribution of training data and that of the testing data (2) conditional probability distributions on the output space forming aleatory representations of the uncertainty on the output as a function of the input variable.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/2103.09982arXivDiscussion Paper
ORCID:
AuthorORCID
Tavallali, Peyman0000-0001-7166-5489
Owhadi, Houman0000-0002-5677-1600
Additional Information:Attribution 4.0 International (CC BY 4.0). © 2021. California Institute of Technology. Government sponsorship acknowledged. This research was carried out at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration and support from Beyond Limits (Learning Optimal Models) and AFOSR (Grant number FA9550-18-1-0271, Games for Computation and Learning). The authors are thankful to Amy Braverman, Lukas Mandrake and Kiri Wagstaff, for their insights.
Funders:
Funding AgencyGrant Number
NASA/JPL/CaltechUNSPECIFIED
Air Force Office of Scientific Research (AFOSR)FA9550-18-1-0271
Record Number:CaltechAUTHORS:20210323-130821498
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20210323-130821498
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:108529
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:23 Mar 2021 23:28
Last Modified:23 Mar 2021 23:28

Repository Staff Only: item control page