A Caltech Library Service

Augmented Reality Powers a Cognitive Assistant for the Blind

Liu, Yang and Stiles, Noelle R. B. and Meister, Markus (2018) Augmented Reality Powers a Cognitive Assistant for the Blind. eLife, 7 . Art. No. e37841. ISSN 2050-084X. PMCID PMC6257813. doi:10.7554/eLife.37841.001.

[img] PDF - Published Version
Creative Commons Attribution.

[img] PDF - Submitted Version
See Usage Policy.

[img] PDF (Transparent reporting form) - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


To restore vision for the blind, several prosthetic approaches have been explored that convey raw images to the brain. So far, these schemes all suffer from a lack of bandwidth. An alternate approach would restore vision at the cognitive level, bypassing the need to convey sensory data. A wearable computer captures video and other data, extracts important scene knowledge, and conveys that to the user in compact form. Here, we implement an intuitive user interface for such a device using augmented reality: each object in the environment has a voice and communicates with the user on command. With minimal training, this system supports many aspects of visual cognition: obstacle avoidance, scene understanding, formation and recall of spatial memories, navigation. Blind subjects can traverse an unfamiliar multi-story building on their first attempt. To spur further development in this domain, we developed an open-source environment for standardized benchmarking of visual assistive devices.

Item Type:Article
Related URLs:
URLURL TypeDescription Paper CentralArticle
Liu, Yang0000-0002-8155-9134
Stiles, Noelle R. B.0000-0002-7352-5815
Meister, Markus0000-0003-2136-6506
Alternate Title:Augmented Reality Powers a Cognitive Prosthesis for the Blind
Additional Information:© 2017 Liu et al. This article is distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use and redistribution provided that the original author and source are credited. Received: 24 April 2018; Accepted: 27 October 2018; Published: 27 November 2018. Data and materials availability: Data and code that produced the figures are available on the Dryad Digital Repository. Data availability: All data generated or analysed during this study are included in the manuscript and supporting files. Source data files for all figures have been deposited in Dryad. Additional code is published on a Github repository (; copy archived at Supported by grant 103212 from the Shurl and Kay Curci Foundation. We thank Ralph Adolphs, David Anderson, and Shin Shimojo for comments on the manuscript, and Kristina Dylla for posing for Figure 1A and field testing all of the tasks. Author contributions: Yang Liu, Conceptualization, Data curation, Software, Formal analysis, Validation, Investigation, Visualization, Methodology, Writing—review and editing; Noelle RB Stiles, Resources, Validation, Investigation, Methodology; Markus Meister, Conceptualization, Supervision, Funding acquisition, Writing—original draft, Project administration, Writing—review and editing. Ethics: Human subjects: All procedures involving human subjects were reviewed and approved by the Institutional Review Board at Caltech, Human Subjects Protocol 16-0663. All subjects gave their informed consent to the experiments, and where applicable to publication of videos that accompany this article.
Funding AgencyGrant Number
Shurl and Kay Curci Foundation103212
PubMed Central ID:PMC6257813
Record Number:CaltechAUTHORS:20181030-140944700
Persistent URL:
Official Citation:Augmented reality powers a cognitive assistant for the blind. Yang Liu, Noelle RB Stiles, Markus Meister eLife 2018;7:e37841. DOI: 10.7554/eLife.37841
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:90520
Deposited By: Tony Diaz
Deposited On:30 Oct 2018 22:38
Last Modified:02 Jun 2023 01:38

Repository Staff Only: item control page