Learning-Online-Smooth-Predictors-for-Realtime-Camera-Planning-using-Recurrent-Decision-Trees-Paper2.pdf

Learning Online Smooth Predictors for Realtime Camera Planning

using Recurrent Decision Trees

Jianhui Chen

Hoang M. Le

Peter Carr

;

Yisong Yue

James J. Little

University of British Columbia

California Institute of Technology

;

Disney Research

{

jhchen14, little

}

@cs.ubc.ca

{

hmle, yyue

}

@caltech.edu

carr@disneyresearch.com

Abstract

We study the problem of online prediction for realtime

camera planning, where the goal is to predict smooth trajec-

tories that correctly track and frame objects of interest (e.g.,

players in a basketball game). The conventional approach

for training predictors does not directly consider temporal

consistency, and often produces undesirable jitter. Although

post-hoc smoothing (e.g., via a Kalman filter) can mitigate

this issue to some degree, it is not ideal due to overly strin-

gent modeling assumptions (e.g., Gaussian noise). We pro-

pose a recurrent decision tree framework that can directly

incorporate temporal consistency into a data-driven pre-

dictor, as well as a learning algorithm that can efficiently

learn such temporally smooth models. Our approach does

not require any post-processing, making online smooth pre-

dictions much easier to generate when the noise model is

unknown. We apply our approach to sports broadcasting:

given noisy player detections, we learn where the camera

should look based on human demonstrations. Our exper-

iments exhibit significant improvements over conventional

baselines and showcase the practicality of our approach.

1. Introduction

We investigate the problem of determining where a cam-

era should look when broadcasting a team sporting event,

such as basketball or soccer (see Fig.

1). Realtime camera

planning shares many similarities with online object track-

ing: in both cases, the algorithm must constantly revise an

estimated target position as new evidence is acquired. Noise

and other ambiguities can cause non-ideal jittery trajecto-

ries, which in camera planning lead to unaesthetic results

(see Fig.

2). In contrast to object tracking, smoothness is of

paramount importance in camera control: fluid movements

that maintain adequate framing are preferable to erratic mo-

tions that constantly pursue perfect composition.

Non-parametric or model-free estimation methods, such

as random forests [10], are very popular because they can

learn (almost) arbitrary predictors directly from training

Figure 1.

Camera Planning

The goal is to predict the pan angle

for a broadcast camera based on noisy player detections. Con-

sider two planning algorithms (the blue and red curves) which

both make the same mistake at time

but recover to a good fram-

ing by

(the ideal camera trajectory is shown in black). The blue

solution quickly corrects by time

using a jerky motion, whereas

the red curve conducts a gradual correction. Although the red

curve has a larger discrepancy with the ideal motion curve, its

velocity characteristics are most similar to the ideal motion path.

data. When applied to smooth trajectory prediction, the esti-

mator is often learned within a time-independent paradigm,

with temporal regularization integrated afterwards as a post-

processing stage (e.g., via a Kalman filter) [28,

30]. One

major limitation of this two-stage approach for camera plan-

ning is that the smoothing is done in a context-independent

way, which can lead to uninformed tradeoffs between accu-

racy and smoothness (see Fig.

1).

In this paper, we propose a recurrent decision tree frame-

work that can make predictions

conditioned on its own pre-

vious predictions

, which allows it to learn temporal patterns

within the data (in addition to any direct feature-based re-

lationships). However, this recursive formulation (similar

to reinforcement learning [31]) makes the learning problem

much more challenging compared to the time-independent

approach. We develop a learning algorithm based on the

“search and learn” (SEARN) approach [12] to efficiently

converge to a stable recurrent model.

We applied our approach to autonomous camera control

in sports, where the goal is to generate smooth camera mo-

tion that imitates a human expert. We provide both quan-

titative and qualitative evidence showing our approach sig-

nificantly outperforms several strong baselines.

2. Related work

Sequential Supervised Learning

Sequential supervised

learning is broadly applied in many domains, including nat-

ural language processing tasks such as part-of-speech tag-

ging [9] and computational biology tasks such as protein

alignment [32]. Unlike standard supervised learning, se-

quential supervised learning operates on sequences: each

training example

x

y

is a sequence of features

x

“

〈

,...,x

i,T

〉

and a corresponding sequence of la-

bels

y

“

〈

,...,y

i,T

〉

. The learned predictor out-

puts a sequence of labels for an input sequence of features.

Our setting is distinguished from conventional sequential

learning because we must learn an online predictor. In other

words, conventional sequential learning typically assumes

access to all of

before predicting

[1, 9, 13, 23, 24, 32].

Our setting is further distinguished from most previous se-

quential prediction work by aiming to learn model-free or

non-parametric predictors. For instance, the bulk of previ-

ous work utilize linear predictors and thus require a well-

specified feature representation [1, 9, 24, 32].

One approach for utilizing arbitrary predictors is via a

sliding window [13, 14, 23], in which any supervised learn-

ing method can be used to learn the sliding window predic-

tor. However, if the predictor is defined recurrently (i.e., it

depends on its previous predictions), then it is not obvious

what values to use for previous predictions when generat-

ing supervised training examples. One way to address this

issue is via “stacked” sequential learning [8, 28, 30], which

is essentially the two-stage approach of first training a non-

recurrent predictor, and then employing recurrent smooth-

ing (e.g., a hidden Markov model or Kalman filter).

Our approach instead directly trains a predictor to make

good predictions given previous predictions, which leads

to a “chicken and egg” problem of how to define a train-

ing set that depends on the predictions of the model to be

trained. We employ a learning reduction approach, based

on SEARN [12], that iteratively constructs a sequence of

supervised training sets such that the resulting sequence of

predictors efficiently converges to a stable recurrent pre-

dictor. Other learning reduction approaches for sequential

learning include DAgger [29], which can be more efficient

in the number of iterations needed to converge, but with

each iteration being more computationally expensive.

Camera Planning

Algorithms for determining where a

camera should look have been investigated for a variety

of scenarios from scripted cooking shows and college lec-

tures to team sports [5, 22]. It is widely accepted that a

smoothly moving camera is critical for generating aesthetic

video [16]. One approach is post-processing to regulate the

sequential predictions, which leads to a trade-off between

smoothness and responsiveness [6]. Alternatively, offline

batch processes [19] can be used if there is not an online re-

Figure 2.

Jitter and Overshooting Artifact

(a) A plan generated

by [6] includes a sudden direction change (green dashed box) and

jitter. (b) Post-processing independent predictions with a Kalman

filter reduces jitter but causes over shooting (black dashed box).

quirement. Significant filtering constrains the motion of the

camera, making it unable to track fast moving objects [26].

Other approaches can offer more refined control but first re-

quire users to select the main features of interest [17].

Camera Motion Models

Camera motion is well-studied

in the context of stabilization. Liu

et al

. [26] used a low-pass

filter to create constant angular velocity motions for video

stabilization. The rotation component was filtered by lin-

earizing the rotation space using derivatives with respect to

time to obtain angular velocities [25]. The method was ex-

tended to polynomial eigen-trajectories for subspace video

stabilization [27]. In contrast, we are interested in the prob-

lem of camera planning. In other words, rather than trying

to reconstruct a stable camera trajectory from existing tra-

jectories, our goal is to plan a brand new trajectory.

3. Problem Setup

Let

denote a distribution of input sequences:

“

,...,x

y „

. In camera planning,

can be a se-

quence of (noisy) detected player locations from stationary

cameras. For clarity, we assume each sequence has the same

length

, but in general

can vary and/or be unknown.

Let

denote a class of policies that our learner is con-

sidering. Given a stream of inputs

“ x

,...,x

, each

generates a stream of outputs

“ x

,...,y

. In

camera planning,

can be a sequence of pan angles of the

broadcast camera. Section 4 describes our recurrent deci-

sion tree framework that instantiates

in our experiments.

Operationally, each

is a streaming predictor that takes

state

“ t

,...x

,...,y

composed of the

recent inputs and predictions, and generates a new predic-

tion

“

. Note that the next state

depends only

on the new input

, the current state

, and the current

prediction

. Hence, for any input sequence

equiva-

lently generates a state sequence

“ x

,...,s

. We also

abuse notation to say that

“

denotes the sequence

of predictions

generated by streaming

into

(with the

construction of the state sequence

being implicit).

For any policy

, let

denote the distribution

of states at time

is executed for the first

time

steps (

is defined exactly by

and

). Furthermore, let

“

be the average distribution of states if we

for all

steps. The goal of the learner is to find

a policy

that minimizes the imitation loss under its

own induced distribution of states:

“

argmin

E

„

s,π,π

(1)

Since our goal is to learn a policy

that both imitates the

(human) expert

and is also smooth, we decompose our

loss function into precision and smoothness components:

s,π,π

q “

s,π,π

ω`

s,π

(2)

where

measures how well

agrees with

, and

measures how smooth the prediction of

is relative

to the current state

, with

controlling the trade-off

between the two. The precision error

is typically the

squared deviation:

s,π,π

q “ }

q ́

A standard way to instantiate the smoothness error

is via the squared deviation of the velocity:

s,π

q “

}

q ́

qq}

, where

denotes the velocity in state

, which can be computed from the information encoded in

. One can thus interpret the smoothness error as encourag-

ing the curvature of the predicted trajectories to be low.

We assume the agnostic setting, where the minimizer

of (1) does not necessarily achieve 0 loss (i.e., we cannot

perfectly imitate the human expert). In practice, we ap-

proximate the expectation in (1) with a finite sample (e.g.,

a training set of basketball gameplay sequences), and also

solve (1) via alternating minimization: collect training data

according to current

, and then train a new

using that

training data (see Section 5 for more details).

Discussion and Interpretation.

By utilizing a recurrent

policy class

, such as our recurrent decision tree frame-

work (see Section 4), the learner is able to reason about

inherent temporal relationships. For instance, suppose the

learner incorrectly predicted a previous value (see Fig. 1).

Then the learner could make a drastic correction (blue line)

to minimize the discrepancy as quickly as possible, which

might result in unaesthetic high-frequency camera motion.

Instead, a more gradual correction (red line) may better

trade off between instantaneous error and smooth motion.

Estimating the best

is challenging due to its depen-

dence on its own previous predictions. Most notably, this

recurrent relationship makes it nontrivial to extract a set of

independent training examples to use with conventional su-

pervised learning. This issue is formally highlighted in the

learning objective (1), where the distribution that the loss is

evaluated over is induced by the policy under consideration.

In other words, the distribution

that the loss is evaluated

on in (1) depends on the

being considered, which leads

to complicated learning problem since conventional super-

vised learning assumes that distribution is fixed.

One straightforward approach is to approximate

in (1)

using the distribution of states,

, that the human expert

induces on

, and then select the

to minimize:

“

argmin

E

„

s,π,π

(3)

Note that (3) is easy to evaluate since the expectation is over

and does not depend on the policy

under considera-

tion.

However, as soon as the behavior of

deviates from

, then the distribution of states experienced by

will dif-

fer from

, and thus optimizing (3) will not be aligned with

optimizing (1).

We show in Section 5 how to address this

distribution mismatch problem via an alternating minimiza-

tion approach that efficiently converges to a stable solution.

4. Recurrent Decision Tree Framework

Decision trees are amongst the best performing learn-

ing approaches, and are popular whenever a non-parametric

predictor is desirable [10, 14, 23]. We propose a recurrent

extension, where the prediction at the leaf node is not nec-

essarily constant, but rather is a (smooth) function of both

static leaf node prediction and previous predictions from the

tree. For simplicity, we present our framework using a sin-

gle decision tree, although our approach extends trivially to

ensemble methods such as random forests [3, 10].

A decision tree specifies a partitioning of the input

space (i.e, the space of all possible states

). Let

“

denote a training set of state/target pairs.

Conventional regression tree learning aims to learn a parti-

tioning such that each leaf node, denoted by

node

, makes

a constant prediction to minimize the squared loss:

node

“

argmin

s,y

node

(4)

where

node

denotes the training data from

that has par-

titioned into the leaf

node

. The solution to (4) is:

node

“

size

node

s,y

node

(5)

One typically trains via greedy top-down induction [10],

which is an iterative process that repeatly chooses a leaf

node to be split based on a binary query of the input state

The above formulation (4) can be viewed as the simplest

version of our recurrent decision tree framework. In partic-

ular, the decision tree is allowed to branch on the input state

In practice, optimizing (3) reduces to a standard supervised learning

scenario. One simply collects the decisions made by the human expert

to use as a static training set, and then choose the

that agrees most

with

on the states induced by

One simplified interpretation is that training on (3) does not teach the

predictor how to recover from its own mistakes.

, which, as discussed in Section 3, includes the previous

predictions

,...,y

. Thus, decision trees that mini-

mize (4) form a valid recurrent policy class, and can be used

to instantiate

. In the following, we will describe a more

general class of recurrent decision trees that enforce more

explicit smoothness requirements.

Note that recurrent neural networks (RNNs) [2] impose

a slightly different notion of “recurrent” than our approach.

Whereas our approach is only recurrently defined with re-

spect to the previous predictions of our approach, RNNs are

recurrently defined with respect to the previous hidden unit

activations and (or) previous predictions.

4.1. Jointly Capturing Dynamics and Control

Let

,...,y

denote an autoregressor of the

temporal dynamics of

over the distribution of input se-

quences

, while

ignoring

the exogenous inputs

. In other

words, at time step

predicts the behavior

”

given only

,...,y

. Typically,

is selected from a

class of autoregressors

(e.g., smooth autoregressors). For

our experiments, we use regularized linear autoregressors

, although one could also use more complex functions

(e.g., based on Kalman filters). See the supplemental mate-

rial for more details on linear autoregressors.

We now present a policy class

of recurrent decision

trees

that make smoothed predictions by regularizing to

be close to its autoregressor

. For any tree/policy

, each

leaf node is associated with a “control” or “target” signal

node

such that the prediction

given input state

is:

”

q ”

argmin

node

(6)

where

node

denotes the leaf node of the decision tree

that

branches to, and

trades off between

matching

the target signal

node

versus the smooth autoregressor

. The closed-form solution to (6) is:

q “

node

λf

(7)

Compared to just predicting

”

node

, the prediction in

(6) varies much more smoothly with

, since

is now an

interpolation between the target signal

node

and smooth

extrapolation

from previous predictions.

Training requires estimating both the autoregressor

and the decision tree of target signals

node

. In practice,

given training data

, we perform greedy training by first

estimating

(which ignores the exogenous inputs

and then estimating the decision tree of target signals

to “course correct”

. Given a fixed

and decision tree

Intuitively, if we train

, then the state distribution of

should

be similar to the state distribution that will be induced by the resulting

trained

. We address this point further in Section 5.

structure, one can set

node

as:

node

“

argmin

s,y

node

(8)

for

defined as in (7) with

”

node

. Similar to

(5), we can write the closed-form solution of (8) as:

node

“

size

node

s,y

node

λf

(9)

Note that when

“

, then the autoregressor has no influ-

ence, and (9) reduces to (5). Note also that (8) is a simplified

setting that only looks at imitation loss

, but not smooth-

ness loss

. We refer to the supplemental material for more

details regarding our full training procedure.

One can interpret our recurrent decision tree framework

as holistically integrating model-based temporal dynamics

with model-free control

node

. The target signal

node

can thus focus on generating “course corrections” of the

smooth temporal dynamics imposed by

. In practice, the

target signal

node

can also be set using an ensemble such

as random forests [3, 10, 18], or regression trees that pre-

dict piecewise-linear variables in the leaf nodes [15]. In this

way, our framework is completely complementary with pre-

vious work on learning smoother regression tree predictors.

Extensions.

One could also define the predictions

as ve-

locity rather than absolute coordinates, which coupled with

the current state

, will encode the absolute coordinates.

Such an approach would be desirable if one wants to do per-

form autoregressor regularization in the velocity domain.

Another extension is to replace the L2 autoregression

penalty in (8) with an L1 penalty. With an L1 penalty, small

target signals would not deviate the prediction from the au-

toregressor, thus making the temporal curvature potentially

piecewise linear (which may be desirable). However, the

non-smooth nature of L1 regularization would make the

prediction more sensitive to the choice of

5. Iterative Training Procedure

In general, it can be difficult to exactly solve (1) due to

the circular dependency between the distribution of states

and the policy under consideration. One meta-learning ap-

proach is to alternate between estimating

over a fixed

and optimizing

over a fixed

. At a high level, this can

be described as:

1. Start with some initial policy

2. At the

-th iteration, use the previously estimated poli-

cies

,...,

to construct a new distribution

3. Estimate the best policy

via:

“

argmin

E

„

s,π,π

(10)

Note that the loss function

need not be the original

loss function, but simply needs to converge to it.

4. Repeat from Step 2 with

until some termi-

nation condition is met.

One typically initializes using the human expert

“

, which we have demonstrations for in the training set

(i.e.,

is memorized on the training set). The estimation

problem in (10) can be solved via standard supervised learn-

ing (see the supplemental material for how we solve (10) for

our recurrent decision tree framework). It remains to decide

how to construct a stable and converging sequence of distri-

butions

. For instance, it is known that simply choosing

“

1

does not guarantee stable behavior [12, 29].

We build upon the SEARN approach [12] to iteratively

construct a stable sequence of distributions

for training

each

. Algorithm 1 describes our approach, which is a

meta-learning approach that can be applied to other policy

classes beyond our recurrent decision tree framework (i.e.,

by using a different implementation of

Learn

). Note also

that we approximate each state distribution

using a finite

empirical sample

— i.e.,

is constructed from the given

input sequences

and the predictions iteration

Algorithm 1 is an instance of the alternating procedure

described above. Given a state distribution

, the training

problem is a straightforward supervised learning problem

(Line 7). Note that the iteration-specific loss

is simply

the original loss

using a (potentially) modified imitation

target

that converges to the original

(Line 6).

The key innovation of SEARN [12] is that the new dis-

tribution

should be induced by an exponentially decay-

ing interpolation between every

,...,

. In practice,

we found it more convenient to interpolate the trajectories

,...,

(Line 10), which leads to a set of trajectories

that can be combined with the input

to form an empir-

ical sample of states

to approximate

(Line 5). Because

of this interpolation, the sequence of distributions

will

converge in a stable way (see [12] for a rigorous analysis

of the convergence behavior). This stable behavior is espe-

cially important considering our greedy training procedure

for our recurrent decision tree framework: if the resulting

has a very different behavior from the training distribution

, then

will not be a good autoregressor of

5.1. Implementation Details

Choosing

A straightforward choice of interpolation

parameter is to set

to a small constant (close to 0) to

ensure that the new distribution

stays close to the pre-

vious distribution

, which is the original approach of

SEARN [12]. One drawback of this conservative approach

is slower convergence rate, especially when the learner

needs to quickly move away from a bad policy

We can also adaptively select

based on relative empir-

ical loss of learned policies (Line 9 of Algorithm 1). Let

Algorithm 1

Iterative Training Procedure

Input:

input sequences

Input:

expert demonstrations

for

Input:

autoregression function class

Input:

history time horizon

for generating states

//see Section 3

Input:

...

//distribution drift step size

Initialize

0

state sequences defined by

and

0

Learn

0

Initialize

1

Ð t

0

//initialize exploration

for

“

,...,N

state sequences defined by

and

expert feedback on each

//see Section 5.1

Learn

//minimizing

(10)

Ð t

//roll-out

error

error

//see Section 5.1

10:

1

//distribution interpolation

11:

end for

12:

return

P t

1

,...,

with best validation performance

error

and

error

denote the loss (2) of rolled-

out trajectories

, respectively, with respect to ground

truth

. We can then set

as:

“

error

error

error

(11)

This adaptive selection of

encourages the learner to dis-

regard bad policies when interpolating, thus allowing fast

convergence to a good policy.

Choosing Y

Intuitively, whenever the current policy

makes a mistake, the expert feedback

should recommend

smooth corrections (e.g., follow the red line in Figure 1). In

many cases, it suffices to simply use the original expert tra-

jectories

. However, when

differs substantially

from

, especially during early iterations, it can be benefi-

cial to use “smoothed” expert feedback

in place of

Specifically, each step along

is computed to be a

gradual 1-step correction of

towards

, which will grad-

ually converge to

in later iterations. It is not strictly nec-

essary, as our recurrent decision tree framework can offset

non-smooth corrections by enforcing a larger regularization

parameter

(trading higher smoothness for lower imita-

tion accuracy, cf. 8). Generally however, given the same

smoothness weight

, smoothed expert feedback leads to

more stable learning. One simple way to provide smoothed

is to reduce the distance ratio

q{p

by a

small decaying rate:

“

6. Experiments

We evaluate our approach automated broadcasting for

basketball and soccer (see Fig. 3). A high-school basketball

match was recorded using two cameras: one near the ceiling

for

player

detection,

and

one

ground

level

for

broadcast-

ing

(operated

human

expert).

The

videos

were

syn-

chronized

fps.

‘Timeouts’

were

manually

removed,

re-

sulting

minutes

‘in-play’

data

divided

into

roughly

segments

(each

about

seconds

long),

with

two

held

out

for

validation

and

testing.

semi-professional

soccer

match

was

recorded

using

three

cameras:

two

near

the

flood

l ights

f or

p layer

detec-

tion,

and

robotic

PTZ

located

mid-field

remotely

oper-

ated

human

expert.

The

videos

were

synchronized

fps.

About

minutes

was

used

for

training,

and

two

minute

sequences

were

held

out

for

validation

and

testing.

Features

The

ground

locations

players

were

deter-

mined

from

geometric

primitives

which

best

justified

the

background

subtraction

results

[4].

Each

ground

po-

sition

was

projected

spherical

coordinate

system

cen-

tered

and

aligned

with

the

broadcast

camera.

Because

the

number

detections

varies

due

clutter

and

occlusions,

fixed

l ength

f eature

v ector

w as

c onstructed

u sing

spatial

frequency

counts.

The

surface

the

sphere

was

quantized

three

resolutions

(

and

)

resulting

dimensional

feature

vector

x

[6].

Labels

Pan/tilt/zoom

parameters

are

estimated

for

each

frame

the

broadcast

video

matching

detected

SIFT

key

points

collection

manually

calibrated

reference

frames

similar

fashion

[20].

The

homography

be-

tween

the

current

frame

and

the

best

match

the

database

reference

images

estimated,

from

which

the

camera’s

pan-tilt-zoom

settings

are

extracted.

Because

the

tilt

and

zoom

the

broadcast

camera

not

vary

significantly

over

the

dataset,

our

experiments

only

focus

building

esti-

mator

for

online

prediction

pan

angles.

6.1. Baselines

Savitzky-Golay.

[6]

learns

predictor

using

random

for-

est

trained

using

only

current

player

locations.

Savitzky-

Golay

(SG)

filter

smooths

the

predictions,

but

induces

de-

lay. Our implementation of this method augments the cur-

rent player locations with previous player locations. This

modification makes the instantaneous predictions more re-

liable, as the predictor has more temporal information.

Kalman Filter.

We replace the Savitzky-Golay filter with

a Kalman filter employing a constant velocity process

model. Parameters were determined through validation (see

supplemental material).

Dual Kalman Filter.

A dual Kalman filter [21] simulta-

neously estimates the unknown state of the system, as well

as its process matrix. Similar to our formulation, we as-

sume the system adheres to an autoregressive model. This

method then applies two Kalman filters in parallel: one to

estimate the coefficients of the autoregressive function, and

a second to estimate the trajectory of the camera, based on

the current estimate of the autoregressive model. Again, pa-

rameters were tuned through validation.

Conditional Regression Forests.

Conditional regression

forests (CRFs) [11] split the training data into multiple sub-

sets. We tested various splitting methods based on camera

position and velocity, such as dividing the data into

and

subsets of pan angle. We also tried both disjoint sets and

joint sets with different overlap ratios. We report the best

result from

subsets with 50% overlap. The output of the

CRF is further smoothed by a SG filter.

Filter Forests.

Filter forests (FF) [

15] is an efficient dis-

criminative approach for predicting continuous variables

given a signal. FF can learn the optimal filtering kernels

to smooth temporal signals. Our implementation includes

some adaptations, such as limited candidate window sizes,

to improve the performance on our datasets.

6.2. Benchmark Experiments

Fig.

4 shows the benchmark performance evaluated for

both basketball and soccer. We evaluate using joint loss (2)

with

“

500

. The precision and smoothness losses are

plotted separately to illustrate their relative contributions to

the joint loss. For both settings, we see that our approach

achieves the best performance, with the performance gap

being especially pronounced in basketball.

We note that the soccer setting is significantly more chal-

lenging than basketball, and no method performs particu-

larly well for soccer. One possible explanation is that soccer

camera planning using only player detections is unreliable

due to players not following the ball (unlike in basketball).

A visual inspection of the generated videos also suggests

that lack of ball tracking in the input signal

is a significant

limitation in the soccer setting.

We also observe that our approach achieves very low

smoothness loss, despite not utilizing a post-processing

smoothing step (see Table

1 for a summary description of

(a)

KF Dual KF CRF

Ours

loss

precision

smoothness

(b)

KF Dual KF CRF

Ours

100

150

200

250

precision

smoothness

Figure 4.

Prediction loss

. (a) Basketball; (b) Soccer. Loss mea-

sured by (2). Our method achieves the lowest overall loss.

Method

Noise Model

Post-Smooth

Delay

SG & CRF

high-freq

required

Yes

KF & Dual KF

Gaussian

required

self-learned

not-required

Ours

self-learned

not-required

Table 1.

Qualitative comparison

. Both our method and FF [15]

learn noise model from the data, do not require post-processing

and achieve real-time response. However, our model requires less

data and is efficient for both training and testing.

the different approaches). For instance, in the basketball

setting, our approach actually achieves the smoothest per-

formance. In fact, for the basketball setting, our approach

dominates all baselines in both imitation loss and smooth-

ness loss. The KF and FF baselines tend to achieve compet-

itive smoothness loss, but can suffer substantial imitation

loss. For soccer, the SG and CRF baselines achieve lower

imitation loss, suggesting that they might be better able to

track the action. However, they suffer from substantial jitter.

6.3. Visual Inspection

Figs. 5 and 6 show a visualization of the predictions.

From this visual inspection, we can verify qualitatively

that our method achieves the best balance of precision and

smoothness. Our predicted trajectory remains close to the

human operator’s trajectory and has less jitter than the other

methods. Even with post-smoothing, SG and CRF exhibit

significant jitter. KF struggles between jitter and over shoot-

ing when the noise is not Gaussian. Surprisingly, dual KF

performance is worse than KF, which is again presumably

because the noise is not Gaussian, and errors in the pro-

cess estimation propagate to the state estimation (see sup-

plemental material). FF is very competitive in the basket-

ball dataset, but its performance suffers from large jitter in

the more challenging soccer dataset.

Table 1 summarizes qualitative properties of all the

methods. SG and CRF assume the noise only has high-

frequency components. As a result, they struggle with low-

frequency noise. KF and dual KF rely on a Gaussian noise

assumption, which is not reasonable on our datasets.

Both our method and FF [15] learn control and dynamics

(a) SG

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

-30

-20

-10

(b) KF

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

-30

-20

-10

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

-30

-20

-10

(d) CRF

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

-30

-20

-10

(e) FF

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

-30

-20

-10

(f) Our method

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

-30

-20

-10

Figure 5.

Comparison on basketball data.

(a) Method of [6], (b)

Kalman filter, (c) Dual Kalman filter , (d) Conditional regression

forests [11], (e) Filter forests [15], (f) Our method. The blue line

is from human operators and the red line is from predictions. Note

our method does not need any post-processing.

(a) SG

1000

2000

3000

4000

5000

6000

7000

-40

-20

(b) KF

1000

2000

3000

4000

5000

6000

7000

-40

-20

1000

2000

3000

4000

5000

6000

7000

-40

-20

(d) CRF

1000

2000

3000

4000

5000

6000

7000

-40

-20

(e) FF

1000

2000

3000

4000

5000

6000

7000

-40

-20

(f) Our method

1000

2000

3000

4000

5000

6000

7000

-40

-20

Figure 6.

Comparison on soccer data.

. (a) Method of [6], (b)

Kalman filter, (c) Dual Kalman filter , (d) Conditional regression

forests [11], (e) Filter forests [15], (f) Our method. The blue line

is from human operators and the red line is from predictions. Note

our method does not need any post-processing.

aspects from the data. Neither requires post-processing and

both are able to generate a zero delay response. In camera

planning, our method has three advantages over FF. First,

our method has fewer parameters so that it requires less data

to train. Second, our model can be trained much faster than

FF because FF has to solve a large linear equation in each

split. Third, the experimental results show that our method

is more stable in both datasets.

Fig. 7 provides a visual comparison of our framework

using a random forest of 100 trees and varying influence of

the autoregressor,

. When

“

, the autoregressor has no

influence, and so the smoothness of the prediction depends

solely on the smoothness of decision tree ensemble and the

size of the history window,

. Since decision trees are non-

Figure 7.

Comparing varying

for basketball data.

“

300

is a good trade-off between smoothness and accuracy. Very small

allows more accurate but noisier predictions, and very large

leads to smoother but less accurate predictions.

parametric, one could in principle learn a smooth predic-

tor given sufficient training examples and enough trees, but

the data and computational costs would be immense. As

increases, the autoregressor causes the predictions to be

increasingly smooth. Recall that the learner (Algorithm

always seeks to find the predictor within the policy class

with the lowest loss, and so it can adaptively trade off be-

tween smoothness and accuracy depending on the input sig-

nal

(rather than rely on post-processing). As

becomes

very large, the policy class becomes restricted to extremely

smooth predictors. In practice,

can be set via a user pref-

erence study or validation performance.

6.4. User Preference Study

We also conducted a user preference study to comple-

ment our benchmark experiments. Fig.

8 shows our user

study interface. Videos were generated by warping the

video captured by the human operator. We evaluated our

approach against the five baseline algorithms for both bas-

ketball and soccer. In each trial, participants were instructed

to choose the video that was more appealing (our method

was randomly assigned the left or right view).

Table

2 shows the results. For basketball, our method

is significantly preferred over the other methods based on a

two-tailed binomial test (

001

). For soccer, none of

the methods performed particularly well (see Section

6.2),

making it challenging for users to judge which method gen-

erated better videos. Note that there is still a sizable prefer-

ence gap compared to the human expert.

7. Discussion

Although our approach achieved good results for basket-

ball, the results for soccer were much poorer. It is likely that

we require more expressive inputs in order to learn good

policy, such as tracking the ball in addition to the players.

Basketball

Soccer

Comparison

win / loss

vs SG

22 / 3

14 / 11

vs KF

23 / 2

12 / 13

vs dual KF

25 / 0

24 / 1

vs CRF

24 / 1

12 / 13

vs FF

23 / 2

14 / 11

vs human

1 / 24

4 / 21

Table 2.

User study results

. For basketball, our method is signif-

icantly preferred over all baselines. For soccer, all methods per-

formed poorly, and users did not have a strong preference. There

is still a sizable preference gap compared to expert human.

The human demonstrations in the soccer dataset were al-

most entirely piece-wise linear. In other words, the human

expert is almost always directing the camera in a straight

line with very sharp course corrections. As such, it may be

that we require L1 regularization to an autoregressor that

prefers zero acceleration in order to better capture the tem-

poral dynamics of camera planning in soccer.

We chose decision trees due to their non-parametric or

model-free properties as well as ease of training. Using

other complex predictors, such as deep neural nets or Gaus-

sian processes, could potentially also work well.

Finally, our approach only addresses the planning prob-

lem, and cannot be directly applied to physical camera con-

trol without a control model. It would be interesting to

jointly learn to both planning and physical control [7].

8. Summary

We have introduced the problem of smooth online im-

itation learning, where the goal is to learn a predictor to

smoothly imitate a human expert given a stream of in-

put signals.

We have also presented a recurrent non-

parametric model class for generating smooth predictions,

which jointly integrates a model-free control signal with a

model-based autoregressor. In order to guarantee stability

of training, we extended previous work on iterative learning

of dynamical models to our setting. We applied our ap-

proach to camera control in sports, where we demonstrated

significant improvements over several strong baselines.