CaltechAUTHORS
  A Caltech Library Service

Compact Tensor Pooling for Visual Question Answering

Shi, Yang and Furlanello, Tommaso and Anandkumar, Animashree (2017) Compact Tensor Pooling for Visual Question Answering. . (Unpublished) http://resolver.caltech.edu/CaltechAUTHORS:20190402-104101850

[img] PDF - Submitted Version
See Usage Policy.

758Kb

Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechAUTHORS:20190402-104101850

Abstract

Performing high level cognitive tasks requires the integration of feature maps with drastically different structure. In Visual Question Answering (VQA) image descriptors have spatial structures, while lexical inputs inherently follow a temporal sequence. The recently proposed Multimodal Compact Bilinear pooling (MCB) forms the outer products, via count-sketch approximation, of the visual and textual representation at each spatial location. While this procedure preserves spatial information locally, outer-products are taken independently for each fiber of the activation tensor, and therefore do not include spatial context. In this work, we introduce multi-dimensional sketch ({MD-sketch}), a novel extension of count-sketch to tensors. Using this new formulation, we propose Multimodal Compact Tensor Pooling (MCT) to fully exploit the global spatial context during bilinear pooling operations. Contrarily to MCB, our approach preserves spatial context by directly convolving the MD-sketch from the visual tensor features with the text vector feature using higher order FFT. Furthermore we apply MCT incrementally at each step of the question embedding and accumulate the multi-modal vectors with a second LSTM layer before the final answer is chosen.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/1706.06706arXivDiscussion Paper
Record Number:CaltechAUTHORS:20190402-104101850
Persistent URL:http://resolver.caltech.edu/CaltechAUTHORS:20190402-104101850
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94362
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:02 Apr 2019 18:20
Last Modified:02 Apr 2019 18:20

Repository Staff Only: item control page