To begin, clone this repository locally
git clone --recurse-submodules https://github.com/Hongkuan-Zhou/spil
export ROOT=$(pwd)/spil
Install requirements:
cd $ROOT
conda create -n spil_venv python=3.8 # or use virtualenv
conda activate spil_venv
sh install.shIf you encounter problems installing pyhash, you might have to downgrade setuptools to a version below 58.
If you want to train on the CALVIN dataset, choose a split with:
cd $ROOT/dataset
sh download_data.sh D | ABC | ABCD | debugIf you want to get started without downloading the whole dataset, use the argument debug to download a small debug dataset (1.3 GB).
We provide the precomputed embeddings of the different Language Models we evaluate in the paper. The script assumes the corresponding split has been already downloaded.
cd $ROOT/dataset
sh download_lang_embeddings.sh D | ABC | ABCD- Pre-trained SKILL Generator can be downloaded here (epoch-98)
- Pre-trained Model can be downloaded here
- Pre-trained SKILL Generator can be downloaded here (epoch-86)
- Pre-trained Model can be downloaded here
Trained with:
- GPU - 1x NVIDIA Tesla V100 16GB
- RAM - 256GB
- OS - Ubuntu 20.04
Before you start your training, please remember to update the wandb account at
- conf/logger/wandb.yaml
- skill_generator/conf_sg/logger/wandb.yaml
To login your wandb account, first run:
wandb login --relogin
To train the spil model with the maximum amount of available GPUS, run:
python spil/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset model.action_decoder.sg_chk_path=path/to/skill_generator datamodule/datasets=vision_lang
To accelerate training process, the dataset can be first loaded into shared memory. (Note this way requires more RAM, please make sure your server has enough RAM)
python spil/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset model.action_decoder.sg_chk_path=path/to/skill_generator datamodule/datasets=vision_lang_shm
- The
vision_lang_shmoption loads the CALVIN dataset into shared memory at the beginning of the training, speeding up the data loading during training. The preparation of the shared memory cache will take some time (approx. 20 min at our SLURM cluster). - You can either use the following command to train the
skill-generatoror use a pre-trained one.
To train the skill generator, run:
python skill_generator/skill_generator/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset
Note that you should first train the skill-generator if you did not download the pre-trained skill generator.
Hierarchical Universal Language Conditioned Policies (HULC), (Oier et al. 2022)
python spil/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset datamodule/datasets=vision_lang_shm model=mcil
datamodule=hulc loss=hulc
Multi-context imitation learning (MCIL), (Lynch et al., 2019):
python spil/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset datamodule/datasets=vision_lang_shm model=mcil
datamodule=mcil
Goal-conditioned behavior cloning (GCBC), (Lynch et al., 2019):
python spil/training.py trainer.gpus=-1 datamodule.root_data_dir=path/to/dataset datamodule/datasets=vision_lang_shm model=gcbc
See detailed inference instructions on the CALVIN repo.
python spil/evaluation/evaluate_policy.py --dataset_path <PATH/TO/DATASET> --train_folder <PATH/TO/TRAINING/FOLDER>
Optional arguments:
--checkpoint <PATH/TO/CHECKPOINT>: by default, the evaluation loads the last checkpoint in the training log directory. You can instead specify the path to another checkpoint by adding this to the evaluation command.--debug: print debug information and visualize environment.
Single.Task.Completion.4.mp4
This work uses code from the following open-source projects and datasets:
Original: https://github.com/lukashermann/hulc License: MIT
Original: https://github.com/mees/calvin License: MIT
Original: https://github.com/UKPLab/sentence-transformers License: Apache 2.0
Original: https://github.com/openai/CLIP License: MIT
MIT License