A Caltech Library Service

Building Generalizable Agents with a Realistic and Rich 3D Environment

Wu, Yi and Wu, Yuxin and Gkioxari, Georgia and Tian, Yuandong (2018) Building Generalizable Agents with a Realistic and Rich 3D Environment. In: Sixth International Conference on Learning Representations, 30 April-3 May 2018, Vancouver, BC.

[img] PDF - Accepted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Teaching an agent to navigate in an unseen 3D environment is a challenging task, even in the event of simulated environments. To generalize to unseen environments, an agent needs to be robust to low-level variations (e.g. color, texture, object changes), and also high-level variations (e.g. layout changes of the environment). To improve overall generalization, all types of variations in the environment have to be taken under consideration via different level of data augmentation steps. To this end, we propose House3D, a rich, extensible and efficient environment that contains 45,622 human-designed 3D scenes of visually realistic houses, ranging from single-room studios to multi-storied houses, equipped with a diverse set of fully labeled 3D objects, textures and scene layouts, based on the SUNCG dataset (Song et al., 2017). The diversity in House3D opens the door towards scene-level augmentation, while the label-rich nature of House3D enables us to inject pixel- & task-level augmentations such as domain randomization (Tobin et al., 2017) and multi-task training. Using a subset of houses in House3D, we show that reinforcement learning agents trained with an enhancement of different levels of augmentations perform much better in unseen environments than our baselines with raw RGB input by over 8% in terms of navigation success rate. House3D is publicly available at

Item Type:Conference or Workshop Item (Paper)
Related URLs:
URLURL TypeDescription Paper ItemDiscussion Paper
Record Number:CaltechAUTHORS:20221219-215117759
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:118451
Deposited By: George Porter
Deposited On:19 Dec 2022 21:57
Last Modified:20 Dec 2022 04:07

Repository Staff Only: item control page