A Caltech Library Service

Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions

Jiang, Huaizu and Ma, Xiaojian and Nie, Weili and Yu, Zhiding and Zhu, Yuke and Anandkumar, Anima (2022) Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE , Piscataway, NJ, pp. 19034-19043.

Full text is not posted in this repository. Consult Related URLs below.

Use this Persistent URL to link to this item:


A significant gap remains between today's visual pattern recognition models and humanlevel visual cognition especially when it comes to fewshot learning and compositional reasoning of novel concepts. We introduce Bongard-HOI, a new visual reasoning benchmark that focuses on compositional learning of humanobject interactions (HOIs) from natural images. It is inspired by two desirable characteristics from the classical Bongard problems (BPs): 1) fewshot concept learning, and 2) contextdependent reasoning. We carefully curate the fewshot instances with hard negatives, where positive and negative images only disagree on action labels, making mere recognition of object categories insufficient to complete our benchmarks. We also design multiple test sets to systematically study the generalization of visual learning models, where we vary the overlap of the HOI concepts between the training and test sets of fewshot instances, from partial to no overlaps. Bongard-HOI presents a substantial challenge to today's visual recognition models. The state-of-the-art HOI detection model achieves only 62% accuracy on fewshot binary prediction while even amateur human testers on MTurk have 91% accuracy. With the Bongard-HOI benchmark, we hope to further advance research efforts in visual reasoning, especially in holistic perception-reasoning systems and better representation learning.

Item Type:Book Section
Related URLs:
URLURL TypeDescription ItemDiscussion Paper
Jiang, Huaizu0000-0002-2300-4237
Yu, Zhiding0000-0003-1776-996X
Zhu, Yuke0000-0002-9198-2227
Anandkumar, Anima0000-0002-6974-6797
Record Number:CaltechAUTHORS:20230315-336421000.7
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:120065
Deposited By: George Porter
Deposited On:16 Mar 2023 19:10
Last Modified:16 Mar 2023 19:10

Repository Staff Only: item control page