SDGTALK: STRUCTURED FACIAL PRIORS AND DUAL-BRANCH MOTION FIELDS FOR GENERALIZABLE GAUSSIAN TALKING HEAD SYNTHESIS

Official repository for paper SDGTALK: STRUCTURED FACIAL PRIORS AND DUAL-BRANCH MOTION FIELDS FOR GENERALIZABLE GAUSSIAN TALKING HEAD SYNTHESIS.

Installation

Tested on Ubuntu 20.04, CUDA 12.1, PyTorch 2.4.1

git clone [email protected]:MSA-LMC/SDGTalk.git
cd SDGTalk

Environment

conda env create --file environment.yml
conda activate sdgtalk

Install the 3DGS renderer

git clone --recurse-submodules [email protected]:xg-chu/diff-gaussian-rasterization.git
pip install ./diff-gaussian-rasterization
rm -rf ./diff-gaussian-rasterization

Preparation

bash prepare.sh

Inference

python inference.py --image_dir demo/raw/video/source_imgs --pose_dir demo/track_res --audio_dir demo/raw/audio/raw_audio --resume_path assets/SDGTalk.pt

The result is in the render_results.

Acknowledgement

Partial codes are from GAGAvatar, TalkingGaussian. Face Parsing is from face-parsing. Thanks for these great projects!