A Caltech Library Service

Visual Recognition with Humans in the Loop

Branson, Steve and Wah, Catherine and Schroff, Florian and Babenko, Boris and Welinder, Peter and Perona, Pietro and Belongie, Serge (2010) Visual Recognition with Humans in the Loop. In: Computer Vision – ECCV 2010. Lecture Notes in Computer Science. No.6314. Springer , Berlin, pp. 438-451. ISBN 9783642155604.

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


We present an interactive, hybrid human-computer method for object classification. The method applies to classes of objects that are recognizable by people with appropriate expertise (e.g., animal species or airplane model), but not (in general) by people without such expertise. It can be seen as a visual version of the 20 questions game, where questions based on simple visual attributes are posed interactively. The goal is to identify the true class while minimizing the number of questions asked, using the visual content of the image. We introduce a general framework for incorporating almost any off-the-shelf multi-class object recognition algorithm into the visual 20 questions game, and provide methodologies to account for imperfect user responses and unreliable computer vision algorithms. We evaluate our methods on Birds-200, a difficult dataset of 200 tightly-related bird species, and on the Animals With Attributes dataset. Our results demonstrate that incorporating user input drives up recognition accuracy to levels that are good enough for practical applications, while at the same time, computer vision reduces the amount of human interaction required.

Item Type:Book Section
Related URLs:
URLURL TypeDescription
Perona, Pietro0000-0002-7583-5809
Belongie, Serge0000-0002-0388-5217
Additional Information:© 2010 Springer-Verlag Berlin Heidelberg. Funding for this work was provided by NSF CAREER Grant #0448615, NSF Grant AGS-0941760, ONR MURI Grant N00014-06-1-0734, ONR MURI Grant #N00014-08-1-0638, Google Research Award. The authors would like to give special thanks to Takeshi Mita for his efforts in constructing the birds dataset.
Funding AgencyGrant Number
Office of Naval Research (ONR)N00014-06-1-0734
Office of Naval Research (ONR)N00014-08-1-0638
Subject Keywords:Computer Vision; Bird Species; Object Recognition; Visual Recognition; User Response
Series Name:Lecture Notes in Computer Science
Issue or Number:6314
Record Number:CaltechAUTHORS:20190327-132640686
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94220
Deposited By: Tony Diaz
Deposited On:27 Mar 2019 20:35
Last Modified:03 Oct 2019 21:02

Repository Staff Only: item control page