TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio-Motion Embedding and Diffusion Interpolation
){kind=icon}
Welcome contributors! Feel free to submit the pull requests!
- [2024/10] Welcome to try our TANGO on Hugging face space !
- [2024/10] Code for create gesture graph is available.
- Training codes for AuMoClip and ACInterp
- Inference codes for ACInterp
- Processed Youtube Buiness Video data (very small, around 15 mins)
- Scripts for creating gesture graph
- Inference codes with AuMoClip and pretrained weights
git clone https://github.com/CyberAgentAILab/TANGO.git
cd TANGO
git clone https://github.com/justinjohn0306/Wav2Lip.git
git clone https://github.com/dajes/frame-interpolation-pytorch.gitWe Recommend a python version ==3.9.20 and cuda version ==11.8. Then build environment as follows:
# [Optional] Create a virtual env
conda create -n tango python==3.9.20
conda activate tango
# Install with pip:
pip install -r ./pre-requirements.txt
pip install -r ./requirements.txtHere is the command for running inference scripts under the path <your root>/TANGO/, it will take around 3 min to generate two 8s vidoes. You can visualize by directly check the video or check the result .npz files via blender using our blender addon in EMAGE.
Necessary checkpoints and pre-computed graphs will be automatically downloaded during the first run. Please ensure that at least 35GB of disk space is available.
python app.pypython create_graph.pyWe thanks the open-source project Wav2Lip, FiLM, SMPLerX.
Check out our previous works for Co-Speech 3D motion Generation DisCo, BEAT, EMAGE.
This project is only for research or education purposes, and not freely available for commercial use or redistribution. The srcipt is available only under the terms of the Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.