A Caltech Library Service

3D Shape Reconstruction from Vision and Touch

Smith, Edward J. and Calandra, Roberto and Romero, Adriana and Gkioxari, Georgia and Meger, David and Malik, Jitendra and Drozdzal, Michal (2020) 3D Shape Reconstruction from Vision and Touch. . (Unpublished)

[img] PDF - Accepted Version
See Usage Policy.


Use this Persistent URL to link to this item:


When a toddler is presented a new toy, their instinctual behaviour is to pick it upand inspect it with their hand and eyes in tandem, clearly searching over its surface to properly understand what they are playing with. At any instance here, touch provides high fidelity localized information while vision provides complementary global context. However, in 3D shape reconstruction, the complementary fusion of visual and haptic modalities remains largely unexplored. In this paper, we study this problem and present an effective chart-based approach to multi-modal shape understanding which encourages a similar fusion vision and touch do so, we introduce a dataset of simulated touch and vision signals from the interaction between a robotic hand and a large array of 3D objects. Our results show that (1) leveraging both vision and touch signals consistently improves single-modality baselines; (2) our approach outperforms alternative modality fusion methods and strongly benefits from the proposed chart-based structure; (3) there construction quality increases with the number of grasps provided; and (4) the touch information not only enhances the reconstruction at the touch site but also extrapolates to its local neighborhood.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper ItemConference Paper
Calandra, Roberto0000-0001-9430-8433
Malik, Jitendra0000-0003-3695-1580
Additional Information:We would like to acknowledge the NSERC Canadian Robotics Network, the Natural Sciences and Engineering Research Council, and the Fonds de recherche du Québec – Nature et Technologies for their funding support, as granted to the McGill University authors. We would also like to thank Scott Fujimoto and Shaoxiong Wang for their helpful feedback.
Funding AgencyGrant Number
Natural Sciences and Engineering Research Council of Canada (NSERC)UNSPECIFIED
Fonds de recherche du Québec - Nature et technologies (FRQNT)UNSPECIFIED
Record Number:CaltechAUTHORS:20221219-204806086
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:118420
Deposited By: George Porter
Deposited On:20 Dec 2022 15:14
Last Modified:02 Jun 2023 01:08

Repository Staff Only: item control page