Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training run errors #3

Open
planetrocke opened this issue Jun 5, 2022 · 6 comments
Open

Training run errors #3

planetrocke opened this issue Jun 5, 2022 · 6 comments

Comments

@planetrocke
Copy link

planetrocke commented Jun 5, 2022

Ubuntu 20.04, nvidia 470 running a 3090.

I get the following errors on training run, halting things at 0.00%

error: the following arguments are required: --user_data_path

AND

ERROR conda.cli.main_run:execute(49): .... etc..... (See above for error)

I am not sure if the logs are in the .db file or somewhere else, but any input on this would be helpful.

Thanks.

@dunky11
Copy link
Owner

dunky11 commented Jun 6, 2022

The .db file is a sqlite database file.

The project is still pretty much a work in progress, it's good that you use ubuntu because I currently test on ubuntu only. I know where the bug originates from, its simply that a command line argument is not passed to the training script. Logs are usually written in /home/username/.config/voice-smith/data/training_runs/training_run_id/logs but if the script doesn't even start nothing is written there.

I will test everything on Ubuntu before the next release and come back to you when it's out, shouldn't take longer than 2-3 days.

@dunky11
Copy link
Owner

dunky11 commented Jun 6, 2022

v0.1.4 works for me. You may need to start the app using the command line by typing "voice-smith". Otherwise it may currently not detect your conda installation.

@planetrocke
Copy link
Author

planetrocke commented Jun 8, 2022

I'm getting past some of this, now getting

Oops, an error occured, check logs for more info ...
ERROR conda.cli.main_run:execute(49): conda run python training_run.py --run_id 2 --training_runs_path /home/vector/.config/voice-smith/data/training_runs --assets_path /usr/lib/voice-smith/resources/assets --db_path /home/vector/.config/voice-smith/db/voice_smith.db --models_path /home/vector/.config/voice-smith/data/models --datasets_path /home/vector/.config/voice-smith/data/datasets --user_data_path /home/vector/.config/voice-smith/data --environment_name __voice_smith__ failed. (See above for error)

When running this from CLI, I get:

Traceback (most recent call last):
File "/usr/lib/voice-smith/resources/backend/voice_smith/training_run.py", line 12, in
from voice_smith.acoustic_training import train_acoustic
ModuleNotFoundError: No module named 'voice_smith'

ERROR conda.cli.main_run:execute(49): conda run python /usr/lib/voice-smith/resources/backend/voice_smith/training_run.py --run_id 2 --training_runs_path /home/vector/.config/voice-smith/data/training_runs --assets_path /usr/lib/voice-smith/resources/assets --db_path /home/vector/.config/voice-smith/db/voice_smith.db --models_path /home/vector/.config/voice-smith/data/models --datasets_path /home/vector/.config/voice-smith/data/datasets --user_data_path /home/vector/.config/voice-smith/data --environment_name __voice_smith__ failed. (See above for error)

@planetrocke
Copy link
Author

I ran setup.py on the backend, and was able to get past the previous error, but now it says it can't find voice_smith.config.

@planetrocke
Copy link
Author

The actual error in logs is:

Traceback (most recent call last):
File "training_run.py", line 873, in
continue_training_run(
File "training_run.py", line 847, in continue_training_run
runner.run(
File "/usr/lib/voice-smith/resources/backend/voice_smith/utils/runs.py", line 44, in run
finished = stage(**kwargs)
File "training_run.py", line 327, in preprocessing_stage
extract_data(
File "/usr/lib/voice-smith/resources/backend/voice_smith/preprocessing/extract_data.py", line 319, in extract_data
pitch_mean, pitch_std = calculate_pitch_stats(
File "/usr/lib/voice-smith/resources/backend/voice_smith/preprocessing/extract_data.py", line 438, in calculate_pitch_stats
pitch_mean, pitch_std = scaler.get_mean_std()
File "/usr/lib/voice-smith/resources/backend/voice_smith/utils/tools.py", line 328, in get_mean_std
return self.t_1_m / self.m, torch.sqrt(self.s_1_m / self.m)
TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'

@dunky11
Copy link
Owner

dunky11 commented Jun 11, 2022

You should try 0.2.0, everything python runs in Docker now, which should fix most issues, hope this one works for you ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants