Skip to content

Latest commit

 

History

History
 
 

XCiT

README.md

XCiT: Cross-Covariance Image Transformer, arxiv

PaddlePaddle training/validation code and pretrained models for XCiT.

The official pytorch implementation is here.

This implementation is developed by PaddleViT.

drawing

XCiT Model Overview

Update

  • Update (2021-12-8): Code is updated and ported weights are uploaded.
  • Update (2021-11-7): Code is released

Models Zoo

Model Acc@1 Acc@5 #Params FLOPs Image Size Crop_pct Interpolation Link
xcit_nano_12_p16_224_dist 72.32 90.86 0.6G 3.1M 224 1.0 bicubic google/baidu(7qvz)
xcit_nano_12_p16_384_dist 75.46 92.70 1.6G 3.1M 384 1.0 bicubic google/baidu(1y2j)
xcit_large_24_p16_224_dist 84.92 97.13 35.9G 189.1M 224 1.0 bicubic google/baidu(kfv8)
xcit_large_24_p16_384_dist 85.76 97.54 105.5G 189.1M 384 1.0 bicubic google/baidu(ffq3)
xcit_nano_12_p8_224_dist 76.33 93.10 2.2G 3.0M 224 1.0 bicubic google/baidu(jjs7)
xcit_nano_12_p8_384_dist 77.82 94.04 6.3G 3.0M 384 1.0 bicubic google/baidu(dmc1)
xcit_large_24_p8_224_dist 85.40 97.40 141.4G 188.9M 224 1.0 bicubic google/baidu(y7gw)
xcit_large_24_p8_384_dist 85.99 97.69 415.5G 188.9M 384 1.0 bicubic google/baidu(9xww)

*The results are evaluated on ImageNet2012 validation set.

Notebooks

We provide a few notebooks in aistudio to help you get started:

*(coming soon)*

Requirements

Data

ImageNet2012 dataset is used in the following folder structure:

│imagenet/
├──train/
│  ├── n01440764
│  │   ├── n01440764_10026.JPEG
│  │   ├── n01440764_10027.JPEG
│  │   ├── ......
│  ├── ......
├──val/
│  ├── n01440764
│  │   ├── ILSVRC2012_val_00000293.JPEG
│  │   ├── ILSVRC2012_val_00002138.JPEG
│  │   ├── ......
│  ├── ......

Usage

To use the model with pretrained weights, download the .pdparam weight file and change related file paths in the following python scripts. The model config files are located in ./configs/.

For example, assume the downloaded weight file is stored in ./swin_base_patch4_window7_224.pdparams, to use the swin_base_patch4_window7_224 model in python:

from config import get_config
from xcit import build_xcit as build_model
# config files in ./configs/
config = get_config('./configs/xcit_nano_12_p16_224.yaml')
# build model
model = build_model(config)
# load pretrained weights, .pdparams is NOT needed
model_state_dict = paddle.load('./xcit_nano_12_p16_224_dist')
model.set_dict(model_state_dict)

Evaluation

To evaluate XCiT model performance on ImageNet2012 with a single GPU, run the following script using command line:

sh run_eval.sh

or

CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
    -cfg='./configs/xcit_nano_12_p16_224.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \
    -eval \
    -pretrained='./xcit_nano_12_p16_224_dist'
Run evaluation using multi-GPUs: 
sh run_eval_multi.sh

or

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python main_multi_gpu.py \
    -cfg='./configs/xcit_nano_12_p16_224.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \
    -eval \
    -pretrained='./xcit_nano_12_p16_224_dist'

Training

To train the XCiT model on ImageNet2012 with single GPU, run the following script using command line:

sh run_train.sh

or

CUDA_VISIBLE_DEVICES=0 \
python main_singel_gpu.py \
  -cfg='./configs/xcit_nano_12_p16_224.yaml' \
  -dataset='imagenet2012' \
  -batch_size=32 \
  -data_path='/dataset/imagenet' \
Run training using multi-GPUs: 
sh run_train_multi.sh

or

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python main_multi_gpu.py \
    -cfg='./configs/xcit_nano_12_p16_224.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \

Visualization Attention Map

(coming soon)

Reference

@article{el2021xcit,
  title={XCiT: Cross-Covariance Image Transformers},
  author={El-Nouby, Alaaeldin and Touvron, Hugo and Caron, Mathilde and Bojanowski, Piotr and Douze, Matthijs and Joulin, Armand and Laptev, Ivan and Neverova, Natalia and Synnaeve, Gabriel and Verbeek, Jakob and others},
  journal={arXiv preprint arXiv:2106.09681},
  year={2021}
}