GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians

project / arxiv / video / face tracker / bibtex

Licenses

This work is made available under CC-BY-NC-SA-4.0 and is subject to the following statement:

Toyota Motor Europe NV/SA and its affiliated companies retain all intellectual property and proprietary rights in and to this software and related documentation. Any commercial use, reproduction, disclosure or distribution of this software and related documentation without an express license agreement from Toyota Motor Europe NV/SA is strictly prohibited.

This project uses Gaussian Splatting, which carries its original license. The GUI is inspired by INSTA. The mesh rendering operations are adapted from NVDiffRec and NVDiffRast.

Setup

1. Installation

2. Download

Usage

0. Demo

You can play with a trained GaussianAvatar without downloading the dataset:

python local_viewer.py --point_path media/306/point_cloud.ply

1. Training

SUBJECT=306

python train.py \
-s data/UNION10_${SUBJECT}_EMO1234EXP234589_v16_DS2-0.5x_lmkSTAR_teethV3_SMOOTH_offsetS_whiteBg_maskBelowLine \
-m output/UNION10EMOEXP_${SUBJECT}_eval_600k \
--eval --bind_to_mesh --white_background --port 60000

Command Line Arguments

• --source_path / -s

  Path to the source directory containing a COLMAP or Synthetic NeRF data set.

• --model_path / -m

  Path where the trained model should be stored (output/<random> by default).

• --eval

  Add this flag to use a training/val/test split for evaluation. Otherwise, all images are used for training.

• --bind_to_mesh

  Add this flag to bind 3D Gaussians to a driving mesh, e.g., FLAME.

• --resolution / -r

  Specifies resolution of the loaded images before training. If provided 1, 2, 4 or 8, uses original, 1/2, 1/4 or 1/8 resolution, respectively. For all other values, rescales the width to the given number while maintaining image aspect. If not set and input image width exceeds 1.6K pixels, inputs are automatically rescaled to this target.

• --white_background / -w

  Add this flag to use white background instead of black (default), e.g., for evaluation of NeRF Synthetic dataset.

• --sh_degree

  Order of spherical harmonics to be used (no larger than 3). 3 by default.

• --iterations

  Number of total iterations to train for, 30_000 by default.

• --port

  Port to use for GUI server, 60000 by default.

NOTE: During training, a complete evaluation are conducted on both the validation set (novel-view synthesis) and test set (self-reenactment) every --interval iterations. You can check the metrics in the commandline or Tensorboard. The metrics are computed on all images, although we only save partial images in Tensorboard.

2. Interactive Viewers

Remote Viewer

During training, one can monitor the training progress with the remote viewer

python remote_viewer.py --port 60000

NOTE:

• The remote viewer can slow down training a lot. You may want to close it or check "pause rendering" when not viewing.

• The viewer could get frozen and disconnected the first time you enable "show mesh". You can try switching it on and off or simply wait for a few seconds.

Local Viewer

After training, one can load and render the optimized 3D Gaussians with the local viewer

SUBJECT=306
ITER=300000

python local_viewer.py \
--point_path output/UNION10EMOEXP_${SUBJECT}_eval_600k/point_cloud/iteration_${ITER}/point_cloud.ply

Command Line Arguments

• --point_path

  Path to the gaussian splatting file (ply)

• --motion_path

  Path to the motion file (npz). You only need this if you want to load a different motion sequence than the original one for training.

3. Offline Rendering

Cite

If you find our paper or code useful in your research, please cite with the following BibTeX entry:

@article{qian2023gaussianavatars,
  title={GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians},
  author={Qian, Shenhan and Kirschstein, Tobias and Schoneveld, Liam and Davoli, Davide and Giebenhain, Simon and Nie{\ss}ner, Matthias},
  journal={arXiv preprint arXiv:2312.02069},
  year={2023}
}