CaltechAUTHORS
  A Caltech Library Service

Controllable and Compositional Generation with Latent-Space Energy-Based Models

Nie, Weili and Vahdat, Arash and Anandkumar, Anima (2021) Controllable and Compositional Generation with Latent-Space Energy-Based Models. In: Advances in Neural Information Processing Systems 34 (NeurIPS 2021). Advances in Neural Information Processing Systems , pp. 15498-15512. ISBN 9781713845393. https://resolver.caltech.edu/CaltechAUTHORS:20220714-224708055

[img] PDF (Paper) - Published Version
See Usage Policy.

11MB
[img] PDF - Supplemental Material
See Usage Policy.

69MB
[img] PDF (ArXiv discussion paper) - Submitted Version
See Usage Policy.

32MB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20220714-224708055

Abstract

Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications, but it still remains as a great challenge. In particular, the compositional ability to generate novel concept combinations is out of reach for most current models. In this work, we use energy-based models (EBMs) to handle compositional generation over a set of attributes. To make them scalable to high-resolution image generation, we introduce an EBM in the latent space of a pre-trained generative model such as StyleGAN. We propose a novel EBM formulation representing the joint distribution of data and attributes together, and we show how sampling from it is formulated as solving an ordinary differential equation (ODE). Given a pre-trained generator, all we need for controllable generation is to train an attribute classifier. Sampling with ODEs is done efficiently in the latent space and is robust to hyperparameters. Thus, our method is simple, fast to train, and efficient to sample. Experimental results show that our method outperforms the state-of-the-art in both conditional sampling and sequential editing. In compositional generation, our method excels at zero-shot generation of unseen attribute combinations. Also, by composing energy functions with logical operators, this work is the first to achieve such compositionality in generating photo-realistic images of resolution 1024 x 1024. Code is available at this https URL https://github.com/NVlabs/LACE.


Item Type:Book Section
Related URLs:
URLURL TypeDescription
https://proceedings.neurips.cc/paper/2021/hash/701d804549a4a23d3cae801dac6c2c75-Abstract.htmlPublisherAbstract
https://proceedings.neurips.cc/paper/2021/file/701d804549a4a23d3cae801dac6c2c75-Paper.pdfPublisherPaper
https://proceedings.neurips.cc/paper/2021/file/82cadb0649a3af4968404c9f6031b233-Supplemental.pdfPublisherSupplementary Materials
https://doi.org/10.48550/arXiv.2110.10873arXivDiscussion Paper
https://github.com/NVlabs/LACERelated ItemCode
ORCID:
AuthorORCID
Anandkumar, Anima0000-0002-6974-6797
Record Number:CaltechAUTHORS:20220714-224708055
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20220714-224708055
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:115610
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:15 Jul 2022 23:04
Last Modified:15 Jul 2022 23:04

Repository Staff Only: item control page