A Caltech Library Service

Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Xue, Tianfan and Wu, Jiajun and Bouman, Katherine L. and Freeman, William T. (2016) Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks. In: Advances in Neural Information Processing Systems (NIPS 2016). Neural Information Processing Systems Foundation , La Jolla, CA. ISBN 9781510838819.

[img] PDF - Published Version
See Usage Policy.

[img] Archive (ZIP) - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods, which have tackled this problem in a deterministic or non-parametric way, we propose a novel approach which models future frames in a probabilistic manner. Our proposed method is therefore able to synthesize multiple possible next frames using the same model. Solving this challenging problem involves low- and high-level image and motion understanding for successful image synthesis. Here, we propose a novel network structure, namely a Cross Convolutional Network, that encodes images as feature maps and motion information as convolutional kernels to aid in synthesizing future frames. In experiments, our model performs well on both synthetic data, such as 2D shapes and animated game sprites, as well as on real-wold video data. We show that our model can also be applied to tasks such as visual analogy-making, and present analysis of the learned network representations.

Item Type:Book Section
Related URLs:
URLURL TypeDescription
Wu, Jiajun0000-0002-4176-343X
Bouman, Katherine L.0000-0003-0077-4367
Freeman, William T.0000-0002-2231-7995
Additional Information:© 2016 Neural Information Processing Systems Foundation. The authors thank Yining Wang for helpful discussions. This work is supported by NSF Robust Intelligence 1212849, NSF Big Data 1447476, ONR MURI 6923196, Adobe, and Shell Research. The authors would also like to thank Nvidia for GPU donation. The first two authors contributed equally to this work.
Funding AgencyGrant Number
Office of Naval Research6923196
Record Number:CaltechAUTHORS:20190405-161634029
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94522
Deposited By: George Porter
Deposited On:09 Apr 2019 15:25
Last Modified:09 Mar 2020 13:19

Repository Staff Only: item control page