A Caltech Library Service

Automated Synthetic-to-Real Generalization

Chen, Wuyang and Yu, Zhiding and Wang, Zhangyang and Anandkumar, Anima (2020) Automated Synthetic-to-Real Generalization. Proceedings of Machine Learning Research, 119 . pp. 1746-1756. ISSN 2640-3498. doi:10.48550/arXiv.2007.06965.

[img] PDF - Published Version
See Usage Policy.

[img] PDF - Accepted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Models trained on synthetic images often face degraded generalization to real data. As a convention, these models are often initialized with ImageNet pretrained representation. Yet the role of ImageNet knowledge is seldom discussed despite common practices that leverage this knowledge to maintain the generalization ability. An example is the careful hand-tuning of early stopping and layer-wise learning rates, which is shown to improve synthetic-to-real generalization but is also laborious and heuristic. In this work, we explicitly encourage the synthetically trained model to maintain similar representations with the ImageNet pretrained model, and propose a learning-to-optimize (L2O) strategy to automate the selection of layer-wise learning rates. We demonstrate that the proposed framework can significantly improve the synthetic-to-real generalization performance without seeing and training on real data, while also benefiting downstream tasks such as domain adaptation. Code is available at:

Item Type:Article
Related URLs:
URLURL TypeDescription Paper ItemCode
Wang, Zhangyang0000-0002-2050-5693
Additional Information:© 2020 by the author(s). Work done during internship at NVIDIA. We appreciate the computing power supported by NVIDIA GPU infrastructure. We also thank for the discussion and suggestions from four anonymous reviewers and the help from Yang Zou for the domain adaptation experiments. The research of Z. Wang was partially supported by NSF Award RI-1755701.
Funding AgencyGrant Number
Record Number:CaltechAUTHORS:20201106-120205331
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:106487
Deposited By: George Porter
Deposited On:06 Nov 2020 22:24
Last Modified:02 Jun 2023 01:08

Repository Staff Only: item control page