Despite its name, its not a waveform based VAE, but a melspec one with a melGAN decoder. There is also an adversarial regularization on the latent space in order to extract the loudness from it. You can even use it on your dataset, and train the whole thing in maybe 2-4 days on a single GPU.
This model has realtime generation and a highly-compressed and expressive latent representation.
Train the spectral model
python train.py -c vanilla --wav-loc YOUR_DATA_FOLDER --name ENTER_A_COOL_NAMERemember to delete the preprocessed folder between each training, as the models don't use the same preprocessing pipeline. (You can also use the --lmdb-loc flag with a different path for each model)
Train the waveform model
python train.py -c melgan --wav-loc YOUR_DATA_FOLDER --name ENTER_THE_SAME_COOL_NAMEThe training scripts logs into the runs folder, you can visualize it using tensorboard.
Onced both models are trained, trace them using
python make_wrapper.py --name AGAIN_THE_SAME_COOL_NAMEIt will produce a traced script in runs/COOL_NAME/COOLNAME_LOTSOFWEIRDNUMBERS.ts. It can be deployed, used in a libtorch C++ environement, without having to use the source code. AND if you want to use the realtime abilities of this model, just pass the --use-cached-padding true --buffer-size 2048.
To compile the pd externals, you can use CMAKE
cmake -DCMAKE_PREFIX_PATH=/path.to.libtorch -DCMAKE_BUILD_TYPE=[Release / Debug] -DCUDNN_LIBRARY_PATH=path.to.libcudnn.so -DCUDNN_INCLUDE_PATH=path.to.cudnn.include -G [Ninja / Xcode / Makefile] ../Or even better, use the precompiled binaries available in the Release section of this project.
Just remember to download the CUDA 10.1 cxx11 ABI version of libtorch and unzip it in /usr/lib/
(only tested on ubuntu 18.04 - 19.10 - 20.04)