A Caltech Library Service

Generating Multi-Agent Trajectories using Programmatic Weak Supervision

Zhan, Eric and Zheng, Stephan and Yue, Yisong and Sha, Long and Lucey, Patrick (2018) Generating Multi-Agent Trajectories using Programmatic Weak Supervision. In: Seventh International Conference on Learning Representations (ICLR 2019), 6-9 May 2019, New Orleans, LA.

[img] PDF - Published Version
See Usage Policy.


Use this Persistent URL to link to this item:


We study the problem of training sequential generative models for capturing coordinated multi-agent trajectory behavior, such as offensive basketball gameplay. When modeling such settings, it is often beneficial to design hierarchical models that can capture long-term coordination using intermediate variables. Furthermore, these intermediate variables should capture interesting high-level behavioral semantics in an interpretable and manipulable way. We present a hierarchical framework that can effectively learn such sequential generative models. Our approach is inspired by recent work on leveraging programmatically produced weak labels, which we extend to the spatiotemporal regime. In addition to synthetic settings, we show how to instantiate our framework to effectively model complex interactions between basketball players and generate realistic multi-agent trajectories of basketball gameplay over long time periods. We validate our approach using both quantitative and qualitative evaluations, including a user study comparison conducted with professional sports analysts.

Item Type:Conference or Workshop Item (Poster)
Related URLs:
URLURL TypeDescription ItemCode
Yue, Yisong0000-0001-9127-1989
Additional Information:This research is supported in part by NSF #1564330, NSF #1637598, and gifts from Bloomberg, Activision/Blizzard and Northrop Grumman. Dataset was provided by STATS: Code is available at
Funding AgencyGrant Number
Bloomberg Data ScienceUNSPECIFIED
Northrop Grumman CorporationUNSPECIFIED
Subject Keywords:deep learning, generative models, imitation learning, hierarchical methods, data programming, weak supervision, spatiotemporal
Record Number:CaltechAUTHORS:20190410-120555166
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94622
Deposited By: George Porter
Deposited On:11 Apr 2019 18:29
Last Modified:03 Oct 2019 21:05

Repository Staff Only: item control page