A Caltech Library Service

Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection

Van Horn, Grant and Branson, Steve and Farrell, Ryan and Haber, Scott and Barry, Jessie and Ipeirotis, Panos and Perona, Pietro and Belongie, Serge (2015) Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE , Piscataway, NJ, pp. 595-604. ISBN 978-1-4673-6964-0.

Full text is not posted in this repository. Consult Related URLs below.

Use this Persistent URL to link to this item:


We introduce tools and methodologies to collect high quality, large scale fine-grained computer vision datasets using citizen scientists - crowd annotators who are passionate and knowledgeable about specific domains such as birds or airplanes. We worked with citizen scientists and domain experts to collect NABirds, a new high quality dataset containing 48,562 images of North American birds with 555 categories, part annotations and bounding boxes. We find that citizen scientists are significantly more accurate than Mechanical Turkers at zero cost. We worked with bird experts to measure the quality of popular datasets like CUB-200-2011 and ImageNet and found class label error rates of at least 4%. Nevertheless, we found that learning algorithms are surprisingly robust to annotation errors and this level of training data corruption can lead to an acceptably small increase in test error if the training set has sufficient size. At the same time, we found that an expert-curated high quality test set like NABirds is necessary to accurately measure the performance of fine-grained computer vision systems. We used NABirds to train a publicly available bird recognition service deployed on the web site of the Cornell Lab of Ornithology.

Item Type:Book Section
Related URLs:
URLURL TypeDescription DOIArticle
Perona, Pietro0000-0002-7583-5809
Belongie, Serge0000-0002-0388-5217
Additional Information:© 2015 IEEE. We would like to thank Nathan Goldberg, Ben Barkley, Brendan Fogarty, Graham Montgomery, and Nathaniel Hernandez for assisting with the user experiments. We appreciate the feedback and general guidance from Miyoko Chu, Steve Kelling, Chris Wood and Alex Chang. This work was supported in part by a Google Focused Research Award, the Jacobs Technion-Cornell Joint Research Fund, and Office of Naval Research MURI N000141010933.
Funding AgencyGrant Number
Google Focused Research AwardUNSPECIFIED
Jacobs Technion-Cornell Joint Research FundUNSPECIFIED
Office of Naval Research (ONR)N000141010933
Record Number:CaltechAUTHORS:20151021-090104437
Persistent URL:
Official Citation:Van Horn, Grant; Branson, Steve; Farrell, Ryan; Haber, Scott; Barry, Jessie; Ipeirotis, Panos; Perona, Pietro; Belongie, Serge, "Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection," in Computer Vision and Pattern Recognition (CVPR), 2015 IEEE Conference on , vol., no., pp.595-604, 7-12 June 2015 doi: 10.1109/CVPR.2015.7298658
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:61363
Deposited By: Tony Diaz
Deposited On:21 Oct 2015 19:26
Last Modified:09 Mar 2020 13:18

Repository Staff Only: item control page