A Caltech Library Service

Seeing into Darkness: Scotopic Visual Recognition

Chen, Bo and Perona, Pietro (2017) Seeing into Darkness: Scotopic Visual Recognition. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Conference on Computer Vision and Pattern Recognition . IEEE , New York, NY, pp. 7292-7301. ISBN 978-1-5386-0457-1.

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Images are formed by counting how many photons traveling from a given set of directions hit an image sensor during a given time interval. When photons are few and far in between, the concept of image breaks down and it is best to consider directly the flow of photons. Computer vision in this regime, which we call scotopic, is radically different from the classical image-based paradigm in that visual computations (classification, control, search) have to take place while the stream of photons is captured and decisions may be taken as soon as enough information is available. The scotopic regime is important for biomedical imaging, security, astronomy and many other fields. Here we develop a framework that allows a machine to classify objects with as few photons as possible, while maintaining the error rate below an acceptable threshold. A dynamic and asymptotically optimal speed-accuracy tradeoff is a key feature of this framework. We propose and study an algorithm to optimize the tradeoff of a convolutional network directly from lowlight images and evaluate on simulated images from standard datasets. Surprisingly, scotopic systems can achieve comparable classification performance as traditional vision systems while using less than 0.1% of the photons in a conventional image. In addition, we demonstrate that our algorithms work even when the illuminance of the environment is unknown and varying. Last, we outline a spiking neural network coupled with photon-counting sensors as a power-efficient hardware realization of scotopic algorithms.

Item Type:Book Section
Related URLs:
URLURL TypeDescription Paper
Chen, Bo0000-0001-5566-7361
Perona, Pietro0000-0002-7583-5809
Additional Information:© 2017 IEEE. Date Added to IEEE Xplore: 09 November 2017.
Series Name:IEEE Conference on Computer Vision and Pattern Recognition
Record Number:CaltechAUTHORS:20180111-151458519
Persistent URL:
Official Citation:B. Chen and P. Perona, "Seeing into Darkness: Scotopic Visual Recognition," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 7292-7301. doi: 10.1109/CVPR.2017.771
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:84276
Deposited By: Ruth Sustaita
Deposited On:11 Jan 2018 23:25
Last Modified:03 Oct 2019 19:16

Repository Staff Only: item control page