TokenCut Video (extension of TokenCut CVPR2022)

Pytorch implementation of Tokencut video section:

TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut

Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, , James L. Crowley, Dominique Vaufreydaz

[ Project page ] [ Github (Image Salient Object Detection) ] [ Paper ]

The project is an extension work to our CVPR work Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut .

If our project is helpful for your research, please consider citing :

   @unpublished{wang2022tokencut2, 
	  title = {{TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer 
		    and Normalized Cut}}, 
	  author = {Wang, Yangtao and Shen, Xi and Yuan, Yuan and Du, Yuming and Li, Maomao and 
		    Hu, Shell Xu and Crowley, James L and Vaufreydaz, Dominique}, 
	  url = {https://hal.archives-ouvertes.fr/hal-03765422}, 
	  note = {working paper or preprint}, 
	  year = {2022}, 
	  hal_id = {hal-03765422}, 
	  hal_version = {v1}
	}

Table of Content

• 1. Updates
• 2. Installation
  • 2.1 Dependencies
  • 2.2 Data
• 3. Quick Start
• 4. Unsupervised video segmentation
  • 4.1 Generate Segmentation
  • 4.2 Evaluation
• 5. Acknowledgement

1. Updates

09/06/2022 Initial commit: Code of TokenCut unsupervised video segmentation section is released.

2. Installation

2.1 Dependencies

This code was implemented with Python 3.8, PyTorch 1.9.0 and CUDA 11.2. Please refer to the official installation. If CUDA 11.2 has been properly installed :

pip install torch==1.9.0 torchvision==0.10.0

In order to install the additionnal dependencies, please launch the following command:

pip install -r requirements.txt
conda install -c conda-forge pydensecrf

2.2 Data

We provide quick download commands in PREPARE_DATA.md for DAVIS, FBMS, SegTrackv2 as well as flow RGB representation generation.

3. Quick Start

We provide TokenCut visualization for single video, if the segmentation result is not very accurate, please chekck whether the optical flow results is clear enough, using gap > 1 to compensate small motions.

python quickstart.py --video-path ./figures/horsejump-high --gap 1

To build the graph on single frame, which is faster but less accurate:

python quickstart.py --video-path ./figures/horsejump-high --gap 1 --single-frame

4. Unsupervised video segmentation

4.1 Generate Segmentation

DAVIS

python main.py --out-dir /output/path --dataset DAVIS

FBMS

python main.py --out-dir /output/path --dataset FBMS --gap 3

SegTrackv2

python main.py --out-dir /output/path --dataset SegTrackv2

4.2 Evaluation

Method	Flow	Dataset
		DAVIS	FBMS	SegTV2
TokenCut	RAFT	64.3	60.2	59.6
TokenCut+BS	RAFT	75.1	61.2	56.4
TokenCut+CRF	RAFT	76.7	66.6	61.6
TokenCut	ARFlow	62.0	61.0	58.9
TokenCut+BS	ARFlow	73.1	64.7	54.6
TokenCut+CRF	ARFlow	74.4	69.0	60.8

Davis Evaluation API

Davis evaluation API is in data/davis, please find the instruction in data/davis/README.md

FBMS

python eval.py --dataset FBMS --mask_dir /path/to/mask_data --gt_dir /path/to/FBMS/Annotations

SegTrackv2

python eval.py --dataset SegTrackv2 --mask_dir /path/to/mask_data --gt_dir /path/to/SegTrackv2/GroundTruth

5. Acknowledgement

TokenCut video code is built on top of LOST, DINO, Segswap, Bilateral_Sovlver, MotionGrouping, ARFlow. We would like to sincerely thanks those authors for their great works.