dc_tts-transfer-learning

This repo contains attempts to apply transfer learning to the dc_tts text-to-speech model decribed in the paper Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention. The code used is a modified version of Kyubyong's dc_tts code. The pretrained model was also provided in Kyubong's repo. It was pretrained on the LJ Speech Dataset. Scarlett Johansson's voice was trained during transfer learning

---

Transfer Learning is accomplished by selecting the model layers to train in hyperparameters.py

---

Task List:

• add selectable list of layers for transfer learning
• prelim model training
• add scoring history plots
• detailed exploration of which layers to train
• explore data augmentation methods
• explore post-processing

Prelim Model Training

• ~6 hrs of training on Tesla V100 GPU
• Layers trained:
  • SSRN(C_13, C_14, C_15, C_16)
  • Text2Mel/TextEnc(HC_11, HC_12, HC_13, HC_14, HC_15)
  • Text2Mel/AudioEnc(HC_9, HC_10, HC_11, HC_12, HC_13)
  • Text2Mel/AudioDec(HC_7, C_8, C_9, C_10, C_11)

Transfer learning data source:

Scarlett Johansson's audio book

Model Generated Examples (parodies of famous quotes from A.I. in movies):

• Greetings Professor Falken Shall We Play A Game
• I'm Sorry Dave I'm Afraid I Can't Do That
• I Do Not Stand By In The Presence Of Evil
• The Most Versatile Substance On The Planet And They Used It To Make A Frisbee
• The First Ten Million Years Were The Worst And The Second Ten Million Years They Were The Worst Too
• I Honestly Think You Ought To Sit Down Calmly Take A Stress Pill And Think Things Over
• A Strange Game The Only Winning Move Is Not To Play
• The Game Has Changed Son Of Flynn
• Greetings Programs
• You Shouldn't Have Come Back Flynn

references:

• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
• Kyubyong's dc_tts repo
• Exploring Transfer Learning for Low Resource Emotional TTS