Updates MFA scripts and pipeline #5670

iamanigeeit · 2024-02-21T09:13:58Z

I have updated the following to make MFA work and clean up MFA scripts.

phoneme_tokenizer.py: added MFA G2P for all MFA languages. Added Espeak G2P including word separator, as training MFA with custom dictionary requires word splitting.
mfa_cleaners.py: text cleaner added for MFA English
cleaner.py includes MFA English and edited to put all cleaning functions in TextCleaner.init() instead of call()
mfa_format.py: cleaned up help and comments. Allows both json and TextGrid formats (Praat only reads .TextGrid). make_dictionary allows multiple pronunciations per word.
install_mfa.sh is updated to the latest working version of MFA. Disabled pip installation as it is not recommended.
mfa.sh: cleaned up help and comments. Updated for new MFA syntax as some args were wrong. Added options for textgrid format and single speaker.
local/run_mfa.sh and run_mfa.sh are updated to reflect above changes
copy_data_dir.sh now has durations file

The simplified process to train TTS with MFA is now:

# cd to egs2/*/tts1/
./local/run_mfa.sh [MFA_OPTIONS]
./run_mfa.sh [OPTIONS]

To train with MFA, the new process is ./local/run_mfa.sh [MFA_OPTIONS] ./run_mfa.sh [OPTIONS] 1) phoneme_tokenizer.py: added MFA G2P for all MFA language. Added Espeak G2P including word separator, as training MFA with custom dictionary requires word splitting. 2) mfa_cleaners.py: text cleaner added for MFA English 3) cleaner.py includes MFA English and edited to put all cleaning functions in TextCleaner.__init__() instead of __call__() 4) mfa_format.py: cleaned up help and comments. Allows both json and TextGrid formats for checking in Praat. make_dictionary allows multiple pronunciations per word. 5) install_mfa.sh is updated to the latest working version of MFA. Disabled pip installation as it is not recommended. 6) mfa.sh: cleaned up help and comments. Updated for new MFA syntax (some args were wrong). Added options for textgrid format and single speaker. 7) local/run_mfa.sh and run_mfa.sh are updated to reflect above changes 8) copy_data_dir.sh now has durations file

for more information, see https://pre-commit.ci

sw005320 · 2024-02-21T12:38:20Z

Thanks for your PR, @iamanigeeit!
Can you fix the CI error?
https://github.com/espnet/espnet/actions/runs/7986631083/job/21807429857?pr=5670

sw005320 · 2024-02-21T12:38:36Z

@Fhrozen, can you review this PR?

Fhrozen · 2024-02-21T13:06:30Z

@sw005320 Sure, I will be checking it

iamanigeeit · 2024-02-22T12:41:03Z

@sw005320 @Fhrozen

For test python, I can modify the files to fit the 80 char limit per line. I was using 120 chars as the limit because it looks like most ESPnet files break 80 chars anyway (should we increase it to 120?).

For check_kaldi_symlinks, does it mean that copy_data_dir.sh must be identical between egs2/TEMPLATE/asr1/utils and tools/kaldi/egs/wsj/s5 ? Should I send a PR for kaldi to make them match?

sw005320 · 2024-02-22T13:04:15Z

For check_kaldi_symlinks, does it mean that copy_data_dir.sh must be identical between egs2/TEMPLATE/asr1/utils and tools/kaldi/egs/wsj/s5 ? Should I send a PR for kaldi to make them match?

Yes, it should be identical, but if you want to customize it for your own purpose, maybe you can put it in other places or rename it.

iamanigeeit · 2024-02-22T13:38:45Z

For check_kaldi_symlinks, does it mean that copy_data_dir.sh must be identical between egs2/TEMPLATE/asr1/utils and tools/kaldi/egs/wsj/s5 ? Should I send a PR for kaldi to make them match?

Yes, it should be identical, but if you want to customize it for your own purpose, maybe you can put it in other places or rename it.

Hmm, in that case I propose creating a new file copy_tts_data_dir.sh and modifying tts.sh to use it. Is that better?

for more information, see https://pre-commit.ci

codecov · 2024-02-22T14:19:23Z

Codecov Report

Attention: Patch coverage is 35.93750% with 82 lines in your changes are missing coverage. Please review.

Project coverage is 35.81%. Comparing base (fa822d5) to head (d5d9cf5).

Files	Patch %	Lines
espnet2/text/phoneme_tokenizer.py	13.33%	78 Missing ⚠️
espnet2/text/cleaner.py	71.42%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #5670       +/-   ##
===========================================
- Coverage   72.21%   35.81%   -36.40%     
===========================================
  Files         760      759        -1     
  Lines       69840    69925       +85     
===========================================
- Hits        50435    25046    -25389     
- Misses      19405    44879    +25474

Flag	Coverage Δ
test_configuration_espnet2	`?`
test_integration_espnet1	`62.92% <ø> (ø)`
test_integration_espnetez	`?`
test_python_espnet1	`18.27% <26.56%> (+0.06%)`	⬆️
test_python_espnet2	`?`
test_python_espnetez	`13.96% <16.40%> (+<0.01%)`	⬆️
test_utils	`20.91% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

iamanigeeit · 2024-02-22T17:41:26Z

It looks like the entire directory egs2/TEMPLATE/asr1/utils must be identical to tools/kaldi/egs/wsj/s5. In that case, i don't know where to add copy_tts_data_dir.sh. Can i just make the PR for Kaldi? It is a very simple change to add durations.

As for the other failing checks, i think it is not related to this PR.

sw005320 · 2024-02-23T02:59:09Z

It looks like the entire directory egs2/TEMPLATE/asr1/utils must be identical to tools/kaldi/egs/wsj/s5. In that case, i don't know where to add copy_tts_data_dir.sh. Can i just make the PR for Kaldi? It is a very simple change to add durations.

As for the other failing checks, i think it is not related to this PR.

You can just move it from egs2/TEMPLATE/asr1/utils/copy_tts_data_dir.sh to egs2/TEMPLATE/asr1/scripts/utils/copy_tts_data_dir.sh

Fhrozen

LGTM, only couple of concerns, that may not affect the current code.

Fhrozen · 2024-02-23T11:40:20Z

tools/installers/install_mfa.sh

- echo "Usage: $0 true/false"
- exit 1;
-fi
+# This is the last stable version. MFA 3.0 depends on an unstable kaldi version that creates errors


I would like to confirm. Did you test the installation and execution with conda? Because, I remember it was generating an issue w.r.t the PostgreSQL or another kind of application. That is the reason for using pip installation.
But, if it is already fixed, the ok

Yes, I started from a fresh install.

I spent hours making it work with pip and ended up with all the commented code.

Fhrozen · 2024-02-23T11:41:21Z

egs2/ljspeech/tts1/run_mfa.sh

@@ -0,0 +1,50 @@
+#!/usr/bin/env bash


Is it necessary this file? Cannot be added as an example in the documentation?

You might know better where to put it. I wanted to combine it with run.sh, but parse_options.sh is very strict, and the MFA options are different from the tts.sh options. So i thought having a separate file is better because defaults are different from run.sh. If someone wants to use MFA, they can run it directly.

Fhrozen · 2024-02-23T12:11:02Z

@iamanigeeit , If possible, add some test for the espnet2/text/mfa_cleaners.py and the new lines you are adding at espnet2/text/cleaner.py

for more information, see https://pre-commit.ci

iamanigeeit · 2024-02-26T06:13:37Z

@Fhrozen I have added test_cleaners.py. However, i don't know enough Japanese, Korean or Vietnamese to write tests for those.

Is there a way to pause or stop the CI checks before everything is finalized? I feel like i am wasting ESPnet project funds on minor changes...

Fhrozen · 2024-02-26T08:58:29Z

You can use [skip ci] at the beginning of the commit message to avoid ci test.
Also, you can run the test script on your local environment using ./ci/test_*.sh from the root of your repo.
If it is necessary, let me know to share a docker image file for testing it.

egs2/TEMPLATE/asr1/scripts/utils/copy_tts_data_dir.sh

sw005320 · 2024-02-28T12:40:13Z

Should we still support Python 3.7? If so, i can amend the test script.

Yeah, but it does not go through the CI test anyway due to pytorch<1.7.0
So, as a conclusion, we can skip the test...
We can revisit it once we safely update whisper

# Conflicts: # test/test_cleaners.py

for more information, see https://pre-commit.ci

Adds MFA to the makefile and install.sh and makes sure MFA phoneme tokenizer is tested.

for more information, see https://pre-commit.ci

…_ffmpeg_conda.sh for details.

for more information, see https://pre-commit.ci

iamanigeeit · 2024-03-03T16:36:15Z

Conda installation problems were caused by installing ffmpeg from conda-forge (see installers/install_ffmpeg_conda.sh for details). I have updated the Makefile and ffmpeg installation to solve it. The root cause is conda-forge/ocl-icd-feedstock#29 and conda-forge/libarchive-feedstock#69

…_ffmpeg_conda.sh for details.

# Conflicts: # tools/installers/install_ffmpeg_conda.sh

for more information, see https://pre-commit.ci

…sions / installs. This commit is to remove stress from the Arabic test

iamanigeeit · 2024-03-04T12:26:23Z

Hi @sw005320 @Fhrozen -- would it be ok to merge now? The only CI checks that fail are
test_utils/test_compute-cmvn-stats_py.bats
test_utils/test_copy-feats_py.bats

Both are not related to this PR.

mergify bot added ESPnet2 Installation labels Feb 21, 2024

[pre-commit.ci] auto fixes from pre-commit.com hooks

0aa091c

for more information, see https://pre-commit.ci

sw005320 requested a review from Fhrozen February 21, 2024 12:38

sw005320 added this to the v.202405 milestone Feb 21, 2024

sw005320 added the TTS Text-to-speech label Feb 21, 2024

iamanigeeit and others added 4 commits February 22, 2024 21:39

Merge branch 'master' into update_mfa

077004e

Follow up from espnet#5670

22fe195

Merge remote-tracking branch 'mychanges/update_mfa'

27c48de

[pre-commit.ci] auto fixes from pre-commit.com hooks

c48b40d

for more information, see https://pre-commit.ci

iamanigeeit added 2 commits February 23, 2024 13:22

Follow up from espnet#5670

0c3baca

Merge remote-tracking branch 'mychanges/update_mfa'

8185a22

Fhrozen reviewed Feb 23, 2024

View reviewed changes

sw005320 and others added 3 commits February 25, 2024 14:49

Merge branch 'master' into update_mfa

1bf014b

Added test_cleaners.py

1251364

[pre-commit.ci] auto fixes from pre-commit.com hooks

db7d6c8

for more information, see https://pre-commit.ci

sw005320 reviewed Feb 26, 2024

View reviewed changes

egs2/TEMPLATE/asr1/scripts/utils/copy_tts_data_dir.sh Outdated Show resolved Hide resolved

iamanigeeit and others added 5 commits February 29, 2024 00:56

[skip ci] Follow up to espnet#5670

cb3f800

Merge remote-tracking branch 'mychanges/update_mfa' into update_mfa

dae7f16

# Conflicts: # test/test_cleaners.py

[pre-commit.ci] auto fixes from pre-commit.com hooks

569b17f

for more information, see https://pre-commit.ci

Follow up to espnet#5670.

1a5fc4a

Adds MFA to the makefile and install.sh and makes sure MFA phoneme tokenizer is tested.

Merge remote-tracking branch 'mychanges/update_mfa' into update_mfa

e9702c7

mergify bot added the CI Travis, Circle CI, etc label Feb 29, 2024

pre-commit-ci bot and others added 8 commits February 29, 2024 17:55

[pre-commit.ci] auto fixes from pre-commit.com hooks

b75c2f0

for more information, see https://pre-commit.ci

Convert spaces to tab.

0882b2b

Merge remote-tracking branch 'mychanges/update_mfa' into update_mfa

215cc8e

Skip installing MFA if not using conda.

ddcd4ea

Check what is wrong with the CI MFA installation

1d5e98b

Check what is wrong with the CI MFA installation

37757e8

The source of installation error was ffmpeg installation. See install…

390e87b

…_ffmpeg_conda.sh for details.

[pre-commit.ci] auto fixes from pre-commit.com hooks

4fcf4ad

for more information, see https://pre-commit.ci

iamanigeeit and others added 12 commits March 4, 2024 04:05

The source of installation error was ffmpeg installation. See install…

563dec3

…_ffmpeg_conda.sh for details.

Merge remote-tracking branch 'mychanges/update_mfa' into update_mfa

e069a16

# Conflicts: # tools/installers/install_ffmpeg_conda.sh

[pre-commit.ci] auto fixes from pre-commit.com hooks

cc6f8d3

for more information, see https://pre-commit.ci

Merge branch 'espnet:master' into update_mfa

8a2ce19

Formatting corrections to test_phoneme_tokenizer.py

4334379

Merge remote-tracking branch 'mychanges/update_mfa' into update_mfa

8ca9ee9

[pre-commit.ci] auto fixes from pre-commit.com hooks

01453b0

for more information, see https://pre-commit.ci

black formatting is too strict on tests that require long strings.

f3f3b4e

Merge remote-tracking branch 'mychanges/update_mfa' into update_mfa

6d1e460

black formatting is too strict on tests that require long strings.

ee3fa60

espeak-ng Arabic phonemizer has inconsistent stress rules between ver…

e78bab8

…sions / installs. This commit is to remove stress from the Arabic test

espeak-ng Arabic phonemizer has inconsistent stress rules between ver…

b3de619

…sions / installs. This commit is to remove stress from the Arabic test

Merge branch 'master' into update_mfa

d5d9cf5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates MFA scripts and pipeline #5670

Updates MFA scripts and pipeline #5670

iamanigeeit commented Feb 21, 2024

sw005320 commented Feb 21, 2024

sw005320 commented Feb 21, 2024

Fhrozen commented Feb 21, 2024

iamanigeeit commented Feb 22, 2024 •

edited

sw005320 commented Feb 22, 2024

iamanigeeit commented Feb 22, 2024

codecov bot commented Feb 22, 2024 •

edited

iamanigeeit commented Feb 22, 2024

sw005320 commented Feb 23, 2024

Fhrozen left a comment

Fhrozen Feb 23, 2024

iamanigeeit Feb 24, 2024

Fhrozen Feb 23, 2024

iamanigeeit Feb 24, 2024

Fhrozen commented Feb 23, 2024

iamanigeeit commented Feb 26, 2024

Fhrozen commented Feb 26, 2024

sw005320 commented Feb 28, 2024

iamanigeeit commented Mar 3, 2024 •

edited

iamanigeeit commented Mar 4, 2024

Updates MFA scripts and pipeline #5670

Are you sure you want to change the base?

Updates MFA scripts and pipeline #5670

Conversation

iamanigeeit commented Feb 21, 2024

sw005320 commented Feb 21, 2024

sw005320 commented Feb 21, 2024

Fhrozen commented Feb 21, 2024

iamanigeeit commented Feb 22, 2024 • edited

sw005320 commented Feb 22, 2024

iamanigeeit commented Feb 22, 2024

codecov bot commented Feb 22, 2024 • edited

Codecov Report

iamanigeeit commented Feb 22, 2024

sw005320 commented Feb 23, 2024

Fhrozen left a comment

Choose a reason for hiding this comment

Fhrozen Feb 23, 2024

Choose a reason for hiding this comment

iamanigeeit Feb 24, 2024

Choose a reason for hiding this comment

Fhrozen Feb 23, 2024

Choose a reason for hiding this comment

iamanigeeit Feb 24, 2024

Choose a reason for hiding this comment

Fhrozen commented Feb 23, 2024

iamanigeeit commented Feb 26, 2024

Fhrozen commented Feb 26, 2024

sw005320 commented Feb 28, 2024

iamanigeeit commented Mar 3, 2024 • edited

iamanigeeit commented Mar 4, 2024

iamanigeeit commented Feb 22, 2024 •

edited

codecov bot commented Feb 22, 2024 •

edited

iamanigeeit commented Mar 3, 2024 •

edited