Skip to content
This repository has been archived by the owner on Oct 31, 2023. It is now read-only.
/ Ego-Exo Public archive

Code accompanying Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (CVPR 2021)

License

Notifications You must be signed in to change notification settings

facebookresearch/Ego-Exo

Repository files navigation

Ego-Exo

This repository is an official pytorch implementation of Ego-Exo work:

Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos, CVPR, 2021

Yanghao Li, Tushar Nagarajan, Bo Xiong and Kristen Grauman.

Requirements

Installation

Please follow the instructions in INSTALL.md.

Datasets

Please follow the descriptions in DATASET.md for dataset preparation.

Pre-computed Ego-Exo scores for auxiliary tasks

We provided the pre-computed Ego-Score, Object-Score and Interaction-Map for auxiliary tasks. Please check the paper for the details about the Ego-Exo scores on auxiliary tasks,

  1. [optional] Kinetics. We already provide the pre-trained Kinetics models, so you don't need the Ego-Exo scores of auxiliary tasks on Kinetics unless you want to train Ego-Exo from scratch on Kinetics. Please download the Ego-Score, Object-Score and Interaction-Map on Kinetics, respectively.

  2. [optional] Ego datasets. For Charades-Ego, Epic-55 and Epic-100 datasets, we need Interaction-Map for Ego-Exo* models. Please find the pre-computed Interaction-Map for Charades-Ego, Epic-55 and Epic-100, respectively.

After downloading the Ego-Exo score files, we can put them under data/aux_predictions/.

Pre-trained Kinetics models

We provide pre-trained models on Kinetics for both baseline and Ego-Exo.

Train/evaluate on Egocentric datasets

Training Ego-Exo and Ego-Exo* models on the downstream egocentric datasets can be done using the following command:

python tools/run_net.py \
    --cfg configs/ego-exo/charades-ego/Ego_Exo_SLOWFAST_8x8_R50.yaml \
    TRAIN.CHECKPOINT_FILE_PATH  /path/to/kinetics/models/Ego_Exo_SLOWFAST_8x8_R50.pyth \

We use 8 GPUs to train Charades-Ego and Epic-Kitchen-55, and use 8x2 GPUs for Epic-Kitchen-100. You can specify different pre-trained weights and Ego-Exo and Ego-Exo* configs (under configs/ego-exo/) on different datasets.

Results on Egocentric datasets

The results of different methods on the validation set are shown as bellow:

Charades-Ego

Method mAP
Slow_8x8_R50 24.73
Ego_Exo, Slow_8x8_R50 26.64
SlowFast_8x8_R50 26.27
Ego_Exo, SlowFast_8x8_R50 28.03
Ego_Exo*, SlowFast_8x8_R50 29.11
SlowFast_8x8_R101 27.47
Ego_Exo, SlowFast_8x8_R101 28.69
Ego_Exo*, SlowFast_8x8_R101 30.13

Validation set on Epic-Kitchen-55

Method verb-top1 verb-top5 noun-top1 noun-top5
SlowFast_8x8_R50 64.05 88.49 48.58 70.06
Ego_Exo, SlowFast_8x8_R50 65.97 88.91 49.42 72.35
Ego_Exo*, SlowFast_8x8_R50 66.43 89.16 49.79 71.6

Validation set on Epic-Kitchen-100

Method verb-top1 verb-top5 noun-top1 noun-top5
SlowFast_8x8_R50 66.02 90.13 50.51 75.02
Ego_Exo, SlowFast_8x8_R50 65.87 90.09 51.81 75.49
Ego_Exo*, SlowFast_8x8_R50 66.77 90.3 52.03 76.25
SlowFast_8x8_R101 65.41 90.25 50.97 75.43
Ego_Exo, SlowFast_8x8_R101 67.01 90.62 52.87 76.53
Ego_Exo*, SlowFast_8x8_R101 67.48 90.66 52.93 76.83

For comparisons on the test set of Epic-Kitchen datasets, please check the results on the paper or submit results to the test server of EPIC-KITCHENS.

License

The majority of this work is licensed under CC-NC 4.0 International license. However, portions of the project are available under separate license terms: PySlowFast is licensed under the Apache 2.0 license.

Cite

Ego-Exo is built on top of PySlowFast. If you find this repository useful in your own research, please consider citing:

@inproceedings{ego-exo,
  title={Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos},
  author={Li, Yanghao and Nagarajan, Tushar and Xiong, Bo and Grauman, Kristen},
  booktitle = {CVPR},
  year={2021}
}
@misc{fan2020pyslowfast,
  author =       {Haoqi Fan and Yanghao Li and Bo Xiong and Wan-Yen Lo and
                  Christoph Feichtenhofer},
  title =        {PySlowFast},
  howpublished = {\url{https://github.com/facebookresearch/slowfast}},
  year =         {2020}
}

About

Code accompanying Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (CVPR 2021)

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages