CaltechAUTHORS
  A Caltech Library Service

PerAda: Parameter-Efficient and Generalizable Federated Learning Personalization with Guarantees

Xie, Chulin and Huang, De-An and Chu, Wenda and Xu, Daguang and Xiao, Chaowei and Li, Bo and Anandkumar, Anima (2023) PerAda: Parameter-Efficient and Generalizable Federated Learning Personalization with Guarantees. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20230316-153727855

[img] PDF - Submitted Version
See Usage Policy.

986kB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20230316-153727855

Abstract

Personalized Federated Learning (pFL) has emerged as a promising solution to tackle data heterogeneity across clients in FL. However, existing pFL methods either (1) introduce high communication and computation costs or (2) overfit to local data, which can be limited in scope, and are vulnerable to evolved test samples with natural shifts. In this paper, we propose PerAda, a parameter-efficient pFL framework that reduces communication and computational costs and exhibits superior generalization performance, especially under test-time distribution shifts. PerAda reduces the costs by leveraging the power of pretrained models and only updates and communicates a small number of additional parameters from adapters. PerAda has good generalization since it regularizes each client's personalized adapter with a global adapter, while the global adapter uses knowledge distillation to aggregate generalized information from all clients. Theoretically, we provide generalization bounds to explain why PerAda improves generalization, and we prove its convergence to stationary points under non-convex settings. Empirically, PerAda demonstrates competitive personalized performance (+4.85% on CheXpert) and enables better out-of-distribution generalization (+5.23% on CIFAR-10-C) on different datasets across natural and medical domains compared with baselines, while only updating 12.6% of parameters per model based on the adapter.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/2302.06637arXivDiscussion Paper
ORCID:
AuthorORCID
Xie, Chulin0000-0002-5460-3785
Huang, De-An0000-0002-6945-7768
Xu, Daguang0000-0002-4621-881X
Xiao, Chaowei0000-0002-7043-4926
Anandkumar, Anima0000-0002-6974-6797
Record Number:CaltechAUTHORS:20230316-153727855
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20230316-153727855
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:120084
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:16 Mar 2023 22:46
Last Modified:16 Mar 2023 22:46

Repository Staff Only: item control page