CaltechAUTHORS
  A Caltech Library Service

Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks

Xue, Tianfan and Wu, Jiajun and Bouman, Katherine L. and Freeman, William T. (2019) Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 (9). pp. 2236-2250. ISSN 0162-8828. https://resolver.caltech.edu/CaltechAUTHORS:20190405-140148834

Full text is not posted in this repository. Consult Related URLs below.

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20190405-140148834

Abstract

We study the problem of synthesizing a number of likely future frames from a single input image. In contrast to traditional methods that have tackled this problem in a deterministic or non-parametric way, we propose to model future frames in a probabilistic manner. Our probabilistic model makes it possible for us to sample and synthesize many possible future frames from a single input image. To synthesize realistic movement of objects, we propose a novel network structure, namely a Cross Convolutional Network; this network encodes image and motion information as feature maps and convolutional kernels, respectively. In experiments, our model performs well on synthetic data, such as 2D shapes and animated game sprites, and on real-world video frames. We present analyses of the learned network representations, showing it is implicitly learning a compact encoding of object appearance and motion. We also demonstrate a few of its applications, including visual analogy-making and video extrapolation.


Item Type:Article
Related URLs:
URLURL TypeDescription
https://doi.org/10.1109/tpami.2018.2854726DOIArticle
ORCID:
AuthorORCID
Wu, Jiajun0000-0002-4176-343X
Bouman, Katherine L.0000-0003-0077-4367
Freeman, William T.0000-0002-2231-7995
Additional Information:© 2018 IEEE. Manuscript received 7 Sept. 2017; revised 16 June 2018; accepted 26 June 2018. Date of publication 9 July 2018; date of current version 13 Aug. 2019. We thank Zhijian Liu and Yining Wang for helpful discussions and anonymous reviewers for constructive comments. This work was supported by NSF Robust Intelligence 1212849, NSF Big Data 1447476, ONR MURI 6923196, Adobe, Shell Research, and a hardware donation from Nvidia. T. Xue and J. Wu contributed equally to this work.
Funders:
Funding AgencyGrant Number
NSFIIS-1212849
NSFIIS-1447476
Office of Naval Research (ONR)6923196
AdobeUNSPECIFIED
Shell ResearchUNSPECIFIED
nVidiaUNSPECIFIED
Subject Keywords:future prediction, frame synthesis, probabilistic modeling, convolutional networks, cross convolution
Issue or Number:9
Record Number:CaltechAUTHORS:20190405-140148834
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20190405-140148834
Official Citation:T. Xue, J. Wu, K. L. Bouman and W. T. Freeman, "Visual Dynamics: Stochastic Future Generation via Layered Cross Convolutional Networks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 9, pp. 2236-2250, 1 Sept. 2019. doi: 10.1109/TPAMI.2018.2854726
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94505
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:05 Apr 2019 21:33
Last Modified:03 Oct 2019 21:04

Repository Staff Only: item control page