A Caltech Library Service

Mechanical Search on Shelves using Lateral Access X-RAY

Huang, Huang and Dominguez-Kuhne, Marcus and Satish, Vishal and Danielczuk, Michael and Sanders, Kate and Ichnowski, Jeffrey and Lee, Andrew and Angelova, Anelia and Vanhoucke, Vincent and Goldberg, Ken (2021) Mechanical Search on Shelves using Lateral Access X-RAY. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE , Piscataway, NJ, pp. 2045-2052. ISBN 978-1-6654-1714-3.

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Finding an occluded object in a lateral access environment such as a shelf or cabinet is a problem that arises in many contexts such as warehouses, retail, healthcare, shipping, and homes. While this problem, known as mechanical search, is well-studied in overhead access environments, lateral access environments introduce constraints on the poses of objects and on available grasp actions, and pushing actions are preferred to preserve the environment structure. We propose LAX-RAY (Lateral Access maXimal Reduction in support Area of occupancY distribution): a system that combines target object occupancy distribution prediction with a mechanical search policy that sequentially pushes occluding objects to reveal a given target object. For scenarios with extruded polygonal objects, we introduce two lateral-access search policies that encode a history of predicted target distributions and can plan up to three actions into the future. We introduce a First-Order Shelf Simulator (FOSS) and use it to evaluate these policies in 800 simulated random shelf environments per policy. We also evaluate in 5 physical shelf environments using a Fetch robot with an embedded PrimeSense RGBD Camera and an attached pushing blade. The policies outperform baselines by up to 25% in simulation and up to 60% in physical experiments. Additionally, the two-step prediction policy is the highest performing in simulation for 8 objects with a 69% success rate, suggesting a tradeoff between future information and prediction errors. Code, videos, and supplementary material can be found at

Item Type:Book Section
Related URLs:
URLURL TypeDescription Paper ItemCode, videos, and supplementary material
Goldberg, Ken0000-0001-6747-9499
Additional Information:© 2021 IEEE. This research was performed at the AUTOLAB at UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab. This research was supported in part by: NSF National Robotics Initiative Award 1734633 and by a Focused Research Award from Google Cloud. The authors were supported in part by donations from Google and Toyota Research Institute, the National Science Foundation Graduate Research Fellowship Program under Grant No. 1752814, and by equipment grants from PhotoNeo and NVidia. We thank our colleague Daniel Seita who provided helpful feedback and suggestions.
Funding AgencyGrant Number
Toyota Research InstituteUNSPECIFIED
NSF Graduate Research FellowshipDGE-1752814
Record Number:CaltechAUTHORS:20220322-742283000
Persistent URL:
Official Citation:H. Huang et al., "Mechanical Search on Shelves using Lateral Access X-RAY," 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021, pp. 2045-2052, doi: 10.1109/IROS51168.2021.9636629
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:114002
Deposited By: George Porter
Deposited On:23 Mar 2022 16:27
Last Modified:23 Mar 2022 16:27

Repository Staff Only: item control page