Error during Training Models process

[Benson-Hsueh]

Dear AxonDeepSeg developers,

I'm trying the IVADOMED method of Training Models procedure you provided.
https://ivadomed.org/tutorials/two_class_microscopy_seg_2d_unet.html#download-dataset

I have downloaded the dataset and tried to train the model.
There's a Value Error description at the last line of the process, mentioning config_bids.json is not a valid path.
I pasted the entire process from my terminal window.
It seems like it could not find the correct path of that dataset.
After I typed in "ivadomed_download_data -d data_axondeepseg_sem", it downloaded the dataset to the folder in the following path. /Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/ivadomed

Do you have any suggestions for this situation?
Thank you so much!

All the best,
Benson

--
(ads_venv) Bensons-MBP:~ BensonHsueh$ ivadomed -c config_microscopy.json
2024-01-26 16:42:12.630 | INFO | ivadomed.utils:init_ivadomed:454 -
ivadomed (2.9.8)

2024-01-26 16:42:12.634 | INFO | ivadomed.utils:get_command:387 - No CLI argument given for command: ( --train | --test | --segment ). Will check config file for command...
2024-01-26 16:42:12.634 | INFO | ivadomed.utils:get_path_output:404 - CLI flag --path-output not used to specify output directory. Will check config file for directory...
2024-01-26 16:42:12.634 | INFO | ivadomed.utils:get_path_data:417 - CLI flag --path-data not used to specify BIDS data directory. Will check config file for directory...
2024-01-26 16:42:12.635 | INFO | ivadomed.main:set_output_path:215 - Output path already exists: log_microscopy_sem
2024-01-26 16:42:12.774 | INFO | ivadomed.utils:define_device:160 - Cuda is not available.
2024-01-26 16:42:12.777 | INFO | ivadomed.utils:define_device:161 - Working on cpu.
2024-01-26 16:42:12.781 | INFO | ivadomed.utils:display_selected_model_spec:176 - Selected architecture: Unet, with the following parameters:
2024-01-26 16:42:12.783 | INFO | ivadomed.utils:display_selected_model_spec:179 - dropout_rate: 0.2
2024-01-26 16:42:12.783 | INFO | ivadomed.utils:display_selected_model_spec:179 - bn_momentum: 0.1
2024-01-26 16:42:12.783 | INFO | ivadomed.utils:display_selected_model_spec:179 - depth: 4
2024-01-26 16:42:12.783 | INFO | ivadomed.utils:display_selected_model_spec:179 - is_2d: True
2024-01-26 16:42:12.784 | INFO | ivadomed.utils:display_selected_model_spec:179 - final_activation: sigmoid
2024-01-26 16:42:12.784 | INFO | ivadomed.utils:display_selected_model_spec:179 - length_2D: [256, 256]
2024-01-26 16:42:12.784 | INFO | ivadomed.utils:display_selected_model_spec:179 - stride_2D: [244, 244]
2024-01-26 16:42:12.785 | INFO | ivadomed.utils:display_selected_model_spec:179 - folder_name: model_seg_rat_axon-myelin_sem
2024-01-26 16:42:12.785 | INFO | ivadomed.utils:display_selected_model_spec:179 - in_channel: 1
2024-01-26 16:42:12.785 | INFO | ivadomed.utils:display_selected_model_spec:179 - out_channel: 3
Traceback (most recent call last):
File "/Users/bensonhsueh/miniconda/envs/ads_venv/bin/ivadomed", line 8, in
sys.exit(run_main())
File "/Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/ivadomed/main.py", line 623, in run_main
run_command(context=context,
File "/Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/ivadomed/main.py", line 398, in run_command
bids_df = BidsDataframe(loader_params, path_output, derivatives=True,
File "/Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/ivadomed/loader/bids_dataframe.py", line 72, in init
self.create_bids_dataframe()
File "/Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/ivadomed/loader/bids_dataframe.py", line 107, in create_bids_dataframe
layout = pybids.BIDSLayout(str(path_data), config=self.bids_config, indexer=indexer,
File "/Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/bids/layout/layout.py", line 169, in init
self.connection_manager = ConnectionManager(
File "/Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/bids/layout/db.py", line 45, in init
self.reset_database(init_args, config)
File "/Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/bids/layout/db.py", line 110, in reset_database
config = [Config.load(c, session=self.session) for c in listify(config)]
File "/Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/bids/layout/db.py", line 110, in
config = [Config.load(c, session=self.session) for c in listify(config)]
File "/Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/bids/layout/models.py", line 165, in load
raise ValueError("{} is not a valid path.".format(config))
ValueError: ivadomed/config/config_bids.json is not a valid path.

[mathieuboudreau]

Hi @Benson-Hsueh - just from a quick look, it appears you're in the AxonDeepSeg virtual environment - was IVADOMED installed in this same env? If you want to train data, it may be best to download IVADOMED directly (not through AxonDeepSeg) via their instructions, because the installation done through ADS may not have all of the acoompanying files / they may not get downloaded where it can "see" them.

Alternatively, you could try to find the full path of the config file, which appears to be /Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/ivadomed/config/config_bids.json or /Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/ivadomed/ivadomed/config/config_bids.json, an then use that full path in your command ivadomed -c (full path to config file)

[Benson-Hsueh]

Hi @mathieuboudreau , thanks for your response! Yes, I installed IVADOMED under the same virtual environment as AxonDeepSeg.
I've tried "ivadomed -c /Users/bensonhsueh/miniconda/envs/ads_venv/lib/python3.8/site-packages/ivadomed/config/config_bids.json" and it still shows errors at the end. I attached the screenshot here.

I saw the IVADOMED instruction website you provided, mentioning it requires Python >= 3.7 and < 3.10 virtual environment. Could you show me how to install it?
I copied and pasted the coding they provided, python -m venv ivadomed_env, source ivadomed_env/bin/activate, and it said command not found in the terminal window.

Thank you so much!

Benson

[Benson-Hsueh]

Hi @mathieuboudreau , I have installed Python and PyTorch as it requested on the website..
After I installed ivadomed, I was trying to download dataset using code "ivadomed_download_data -d data_axondeepseg_sem" and it shoed ModuleNotFoundError: No module named 'torch._six'.

like this

Thanks a lot!
Benson

[mathieuboudreau]

Hi @Benson-Hsueh !

Though their instructions recommend using python -m venv to make the virtual environment, in my own experience using IVADOMED I've had an easier time using it with a conda env, and actually made one today and installed it from scratch for another user issue. Since you already were using AxonDeepSeg and followed those installation instructions, I'm going to assume you already have miniconda or anaconda installed.

Here are the steps I would recommend:

conda create -n ivadomed_venv python=3.9
conda activate ivadomed_venv
git clone https://github.com/neuropoly/ivadomed.git
cd ivadomed
pip install -e .

And follow the prompts (eg type yes or y) when asked.

These are the steps that worked for me today on my local computer, give that a try and let me know if it worked for you!

[Benson-Hsueh]

Hi @mathieuboudreau , many thanks!
Yes, the steps you provided work for me. I appreciate it.
Then, I assume that I need to download the dataset and train the model.
However, when I used "ivadomed_download_data -d data_axondeepseg_sem" in the terminal, it showed command not found.

What would you suggest me to do?
Thanks!

Benson

[mathieuboudreau]

That's odd, so pip install -e . completed succesfully, but ivadomed_download_data isn't a command available post the pip install? Typically, pip install -e . would create that command (as with more); though, they'll only "exist" within the virtual environment. If you opened a new terminal, you'd have to do conda activate ivadomed_venv again before ivadomed_download_data would work; was the virtual environment activated when you tried that command?

[mathieuboudreau]

HI @Benson-Hsueh - just wanted to quickly follow up with you to see if you're still experiencing an issue. Also, I wanted to let you know there another user was recently trying to train a model on IVADOMED using the dataset and experienced an issue, we've fixed it and pushed the changes yesterday (see ivadomed/ivadomed#1313) so make sure you pull or download the latest IVADOMED version from the master branch so that you don't encounter this issue as well.

[Benson-Hsueh]

Hi @mathieuboudreau thanks for the notice. I just re-did the process you suggested.

conda create -n ivadomed_venv python=3.9
conda activate ivadomed_venv
git clone https://github.com/neuropoly/ivadomed.git
cd ivadomed
pip install -e .

The process seems to be successful. What would you suggest me to do next?
Should I follow the steps in this website https://ivadomed.org/tutorials/two_class_microscopy_seg_2d_unet.html and download the dataset using this code?

ivadomed_download_data -d data_axondeepseg_sem

Thanks!

Besnon

[hermancollin]

@Benson-Hsueh I understand you are trying to train a model on your own data and ground truth masks. I remember you opened a discussion a few months ago with your BF images. How did that work out?

Most of our images are from TEM and bright field.

So if I understand correctly you have manually corrected segmentation masks for both TEM and BF images, and would like to train your own model for a better performance on your data, is that correct? If this is the case, ivadomed might not be the best way to go for training. If you are interested, I can provide details about more efficient alternatives.

Would you mind sharing some more information on your project? (e.g. how many segmentation masks you have, why you decided to train a model, etc). If you have a lot of images, we could even assist you with training (e.g. using our computer infrastructures).

[hermancollin]

@Benson-Hsueh thank you for this detailed response. So you have manually corrected masks for these 2 images and intended to use them for training, correct? If that's the case, I'm afraid 2 images might not be enough. Deep learning models are known as "data-hungry". I trained a lot of models on bigger and smaller datasets, and the performance is often limited when we don't have a lot of masks for training.

We could apply our new model we are currently developing and see if it works better than the current ADS models. What do you think @ArthurBoschet @Nish228? It could be a good way to evaluate it on real world data and it might help him. Also, what do you guys think of this user's situation in terms of training?

Related to axondeepseg/model_seg_generalist#11

[Benson-Hsueh]

@hermancollin

So you have manually corrected masks for these 2 images and intended to use them for training, correct?

Correct!

If that's the case, I'm afraid 2 images might not be enough. Deep learning models are known as "data-hungry".

I can provide you with more images for the training in the future.
Thanks for stepping into our situation.

Benson

[hermancollin]

@Benson-Hsueh if you were able to provide more images / corrected masks, we could help you get better results using our new experimental model (which should be publicly available within 1-2 months). If you are interested, please contact us by email at [email protected] so we can discuss this matter further.

[Benson-Hsueh]

@hermancollin thanks for the information. I don't have more TEM images in hand now, but I will talk with my PI to provide more images to you. I'll get back to you as soon as I get more info from my PI. I appreciate it!

Best, Benson

[Benson-Hsueh]

Dear @hermancollin we have collected some TEM images for you to train the model. I will send google drive link to you with this email [email protected] Thank you so much!