A Caltech Library Service

The Ignorant Led by the Blind: A Hybrid Human-Machine Vision System for Fine-Grained Categorization

Branson, Steve and Van Horn, Grant and Wah, Catherine and Perona, Pietro and Belongie, Serge (2014) The Ignorant Led by the Blind: A Hybrid Human-Machine Vision System for Fine-Grained Categorization. International Journal of Computer Vision, 108 (1-2). pp. 3-29. ISSN 0920-5691. doi:10.1007/s11263-014-0698-4.

Full text is not posted in this repository. Consult Related URLs below.

Use this Persistent URL to link to this item:


We present a visual recognition system for fine-grained visual categorization. The system is composed of a human and a machine working together and combines the complementary strengths of computer vision algorithms and (non-expert) human users. The human users provide two heterogeneous forms of information object part clicks and answers to multiple choice questions. The machine intelligently selects the most informative question to pose to the user in order to identify the object class as quickly as possible. By leveraging computer vision and analyzing the user responses, the overall amount of human effort required, measured in seconds, is minimized. Our formalism shows how to incorporate many different types of computer vision algorithms into a human-in-the-loop framework, including standard multiclass methods, part-based methods, and localized multiclass and attribute methods. We explore our ideas by building a field guide for bird identification. The experimental results demonstrate the strength of combining ignorant humans with poor-sighted machines the hybrid system achieves quick and accurate bird identification on a dataset containing 200 bird species.

Item Type:Article
Related URLs:
URLURL TypeDescription DOIArticle
Perona, Pietro0000-0002-7583-5809
Belongie, Serge0000-0002-0388-5217
Additional Information:© 2014 Springer Science+Business Media New York. Received: 7 March 2013; Accepted: 8 January 2014; Published online: 20 February 2014.
Issue or Number:1-2
Record Number:CaltechAUTHORS:20140606-130400010
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:46129
Deposited By: Tony Diaz
Deposited On:06 Jun 2014 20:47
Last Modified:10 Nov 2021 17:21

Repository Staff Only: item control page