Skip to content

Latest commit

 

History

History
174 lines (140 loc) · 4.82 KB

README.md

File metadata and controls

174 lines (140 loc) · 4.82 KB
Raw

ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings

This is a PyTorch implementation of the NeurIPS-22 paper: https://arxiv.org/abs/2206.12403

Arjun Majumdar*, Gunjan Aggarwal*, Bhavika Devnani, Judy Hoffman and Dhruv Batra

Georgia Institute of Technology, Meta AI

Details

We present a scalable approach for learning open-world object-goal navigation (ObjectNav) – the task of asking a virtual robot (agent) to find any instance of an object in an unexplored environment (e.g., “find a sink”). Our approach is entirely zero-shot – i.e., it does not require ObjectNav rewards or demonstrations of any kind.

Model Architecture for ZSON.

Installation

All the required data can be downloaded from here.

  1. Create a conda environment:

    conda create -n zson python=3.7 cmake=3.14.0

    conda activate zson

  2. Install pytorch version 1.10.2:

    conda install pytorch==1.10.2 torchvision==0.11.3 cudatoolkit=11.3 -c pytorch -c conda-forge

  3. Install habitat-sim:

    conda install habitat-sim-challenge-2022 headless -c conda-forge -c aihabitat

  4. Install habitat-lab:

    git clone --branch challenge-2022 https://github.com/facebookresearch/habitat-lab.git habitat-lab-challenge-2022

    cd habitat-lab-challenge-2022

    pip install -r requirements.txt

    python setup.py develop --all # install habitat and habitat_baselines

    cd ..

Download and Install zson:

  1. Setup steps

    git clone [email protected]:gunagg/zson.git

    cd zson

    pip install -r requirements.txt

    python setup.py develop

  2. Follow the instructions here to set up the data/scene_datasets/ directory. gibson scenes can be found here.

  3. Download the HM3D ImageNav training dataset:

    wget https://huggingface.co/gunjan050/ZSON/resolve/main/imagenav_hm3d.zip

    unzip imagenav_hm3d.zip

    rm imagenav_hm3d.zip  # clean-up

  4. Download the MP3D objectnav dataset.

    wget https://dl.fbaipublicfiles.com/habitat/data/datasets/objectnav/m3d/v1/objectnav_mp3d_v1.zip

    mkdir -p data/datasets/objectnav/mp3d/v1

    unzip objectnav_mp3d_v1.zip -d data/datasets/objectnav/mp3d/v1

    rm objectnav_mp3d_v1.zip  # clean-up

  5. Download the HM3D objectnav dataset.

    wget https://dl.fbaipublicfiles.com/habitat/data/datasets/objectnav/hm3d/v1/objectnav_hm3d_v1.zip

    unzip objectnav_hm3d_v1.zip -d data/datasets/objectnav/

    rm objectnav_hm3d_v1.zip  # clean-up

  6. Download the trained checkpoints zson_conf_A.pth and zson_conf_B.pth, and move to data/checkpoints.

  7. To train policies using OVRL pretrained RGB encoder, download the model weights from here and move to data/models/. More details on the encoder can be found here.

  8. Setup data/goal_datasets using the script tools/extract-goal-features.py. This caches CLIP goal embeddings for faster training.

    Your directory structure should now look like this:

    .
+-- habitat-lab-v0.2.1/
|   ...
+-- zson/
|   +-- data/
|   |   +-- datasets/
|   |   |   +-- objectnav/
|   |   |   +-- imagenav/
|   |   +-- scene_datasets/
|   |   |   +-- hm3d/
|   |   |   +-- mp3d/
|   |   +-- goal_datasets/
|   |   |   +-- imagenav/
|   |   |   |   +-- hm3d/
|   |   +-- models/
|   |   +-- checkpoints/
|   +-- zson/
|   ...

Usage

ZSON configuration A ImageNav Training

sbatch scripts/imagenav-v1-hm3d-ovrl-rn50.sh

ZSON configuration B ImageNav Training

sbatch scripts/imagenav-v2-hm3d-ovrl-rn50.sh

ObjectNav Evaluation

To evaluate a checkpoint trained using ZSON checkpoint use the following command:

sbatch scripts/objnav-eval-$DESIRED-CONFIGURATION$-$DATASET$.sh

Citation

If you use this code in your research, please consider citing:

@inproceedings{majumdar2022zson,
  title={ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings},
  author={Majumdar, Arjun and Aggarwal, Gunjan and Devnani, Bhavika and Hoffman, Judy and Batra, Dhruv},
  booktitle={Neural Information Processing Systems (NeurIPS)},
  year={2022}
}