Hierarchical Video Prediction

PyTorch implementation of our paper, Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction, which will be presented in ICLR 2021. Please check out more qualitative results in our project page.

Architecture

How to use

Step 1: Clone and setup the repo.

REPO_DIR='/path/to/clone/this/repo'
git clone https://www.github.com/1Konny/hierarchicalvideoprediction $REPO_DIR

cd $REPO_DIR
bash scripts/dependency.sh

cd $REPO_DIR/image_generator
python scripts/download_models_flownet2.py
python scripts/download_flownet2.py

cd $REPO_DIR

Step 2: Prepare datasets.

Check it out in this link

Step 3: Train structure generator.

CUDA_VISIBLE_DEVICES='0,1,2,3' bash scripts/train_structure_generator.sh $DATASET

, where DATASET can be one of KITTI or Cityscapes.

Step 4: Extract semantic-level predictions using the trained structure generator.

CUDA_VISIBLE_DEVICES='0' bash scripts/test_structure_generator.sh $DATASET

Step 5: Train image generator.

CUDA_VISIBLE_DEVICES='0,1,2,3' bash scripts/train_image_generator.sh $DATASET

Step 6: Extract RGB-level predictions using the trained image generator and predictions from the structure generator.

CUDA_VISIBLE_DEVICES='0' bash scripts/test_image_generator.sh $DATASET

Acknowledgement

• This repo is largely borrowed and modfied from SVG and Vid2Vid.

Citation

@inproceedings{
lee2021revisiting,
title={Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction},
author={Wonkwang Lee and Whie Jung and Han Zhang and Ting Chen and Jing Yu Koh and Thomas Huang and Hyungsuk Yoon and Honglak Lee and Seunghoon Hong},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=3RLN4EPMdYd}
}