Brattoli, Biagio and Tighe, Joseph and Zhdanov, Fedor and Perona, Pietro and Chalupka, Krzysztof (2020) Rethinking Zero-shot Video Classification: End-to-end Training for Realistic Applications. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20200526-133440422
![]() |
PDF
- Submitted Version
See Usage Policy. 2MB |
Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20200526-133440422
Abstract
Trained on large datasets, deep learning (DL) can accurately classify videos into hundreds of diverse classes. However, video data is expensive to annotate. Zero-shot learning (ZSL) proposes one solution to this problem. ZSL trains a model once, and generalizes to new tasks whose classes are not present in the training dataset. We propose the first end-to-end algorithm for ZSL in video classification. Our training procedure builds on insights from recent video classification literature and uses a trainable 3D CNN to learn the visual features. This is in contrast to previous video ZSL methods, which use pretrained feature extractors. We also extend the current benchmarking paradigm: Previous techniques aim to make the test task unknown at training time but fall short of this goal. We encourage domain shift across training and test data and disallow tailoring a ZSL model to a specific test dataset. We outperform the state-of-the-art by a wide margin. Our code, evaluation procedure and model weights are available at this http URL.
Item Type: | Report or Paper (Discussion Paper) | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Related URLs: |
| |||||||||
ORCID: |
| |||||||||
Additional Information: | We thank Amazon for generously supporting the project, and Alina Roitberg for a productive discussion on the evaluation protocol. | |||||||||
Funders: |
| |||||||||
Record Number: | CaltechAUTHORS:20200526-133440422 | |||||||||
Persistent URL: | https://resolver.caltech.edu/CaltechAUTHORS:20200526-133440422 | |||||||||
Usage Policy: | No commercial reproduction, distribution, display or performance rights in this work are provided. | |||||||||
ID Code: | 103459 | |||||||||
Collection: | CaltechAUTHORS | |||||||||
Deposited By: | Tony Diaz | |||||||||
Deposited On: | 26 May 2020 20:39 | |||||||||
Last Modified: | 26 May 2020 20:39 |
Repository Staff Only: item control page