A Caltech Library Service

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

Fan, Linxi and Wang, Guanzhi and Jiang, Yunfan and Mandlekar, Ajay and Yang, Yuncong and Zhu, Haoyi and Tang, Andrew and Huang, De-An and Zhu, Yuke and Anandkumar, Anima (2022) MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge. . (Unpublished)

[img] PDF - Submitted Version
Creative Commons Attribution.


Use this Persistent URL to link to this item:


Autonomous agents have made great strides in specialist domains like Atari games and Go. However, they typically learn tabula rasa in isolated environments with limited and manually conceived objectives, thus failing to generalize across a wide spectrum of tasks and capabilities. Inspired by how humans continually learn and adapt in the open world, we advocate a trinity of ingredients for building generalist agents: 1) an environment that supports a multitude of tasks and goals, 2) a large-scale database of multimodal knowledge, and 3) a flexible and scalable agent architecture. We introduce MineDojo, a new framework built on the popular Minecraft game that features a simulation suite with thousands of diverse open-ended tasks and an internet-scale knowledge base with Minecraft videos, tutorials, wiki pages, and forum discussions. Using MineDojo's data, we propose a novel agent learning algorithm that leverages large pre-trained video-language models as a learned reward function. Our agent is able to solve a variety of open-ended tasks specified in free-form language without any manually designed dense shaping reward. We open-source the simulation suite and knowledge bases (this https URL: to promote research towards the goal of generally capable embodied agents.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper Itemopen-source simulation suite and knowledge bases
Zhu, Yuke0000-0002-9198-2227
Anandkumar, Anima0000-0002-6974-6797
Additional Information:Attribution 4.0 International (CC BY 4.0) We are extremely grateful to Dieter Fox, Bryan Catanzaro, Shikun Liu, Zhiding Yu, Chaowei Xiao, Weili Nie, Jean Kossaifi, Jonathan Raiman, Jaakko Haapasalo, John Spitzer, Zhiyuan “Jerry” Lin, Yingqi Zheng, and many other colleagues and friends for their helpful feedback and insightful discussions. NVIDIA provides the necessary computing resource and infrastructure for this project. Guanzhi Wang is supported by the Kortschak fellowship in Computing and Mathematical Sciences at Caltech.
Funding AgencyGrant Number
Kortschak Scholars ProgramUNSPECIFIED
Record Number:CaltechAUTHORS:20220714-212441682
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:115575
Deposited By: George Porter
Deposited On:15 Jul 2022 22:45
Last Modified:15 Jul 2022 22:45

Repository Staff Only: item control page