Memory Error during train

[martirius]

Hi, thanks for your amazing work!

I'm facing an issue when trying to train.
I'm training on 99 classes with a total of 18000 samples but i'm facing a Memory Error problem.
Here is the traceback:
Using TensorFlow backend. class_names = ['Acoustic_guitar', 'Airplane', 'Applause', 'Bark', 'Bass_drum', 'Bass_guitar', 'Breathing', 'Brushing_teeth', 'Burping_or_eructation', 'Bus', 'Can_opening', 'Car_horn', 'Cat', 'Cello', 'Chainsaw', 'Chime', 'Chirping_birds', 'Church_bells', 'Clapping', 'Clarinet', 'Clock_alarm', 'Clock_tick', 'Coin_(dropping)', 'Computer_keyboard', 'Cough', 'Coughing', 'Cow', 'Cowbell', 'Crackling_fire', 'Crash_cymbal', 'Crickets', 'Crow', 'Crying_baby', 'Dishes_and_pots_and_pans', 'Dog', 'Door_wood_creaks', 'Door_wood_knock', 'Double_bass', 'Drawer_open_or_close', 'Drinking_sipping', 'Electric_piano', 'Engine', 'Fart', 'Finger_snapping', 'Fire', 'Fireworks', 'Flute', 'Footsteps', 'Frog', 'Glass', 'Glass_breaking', 'Glockenspiel', 'Gong', 'Gunshot', 'Hand_saw', 'Harmonica', 'Helicopter', 'Hen', 'Hi-hat', 'Insects', 'Keyboard_typing', 'Keys_jangling', 'Knock', 'Laughing', 'Laughter', 'Meow', 'Microwave_oven', 'Mouse_click', 'Oboe', 'Piano', 'Pig', 'Pouring_water', 'Rain', 'Rooster', 'Saxophone', 'Scissors', 'Sea_waves', 'Shatter', 'Sheep', 'Siren', 'Slam', 'Snare_drum', 'Sneezing', 'Snoring', 'Squeak', 'Tambourine', 'Tearing', 'Telephone', 'Thunderstorm', 'Toilet_flush', 'Train', 'Trumpet', 'Vacuum_cleaner', 'Violin_or_fiddle', 'Walk_or_footsteps', 'Washing_machine', 'Water_drops', 'Wind', 'Writing'] total files = 18001 , going to load total_load = 18000 total files = 18001 , going to load total_load = 18000 get_sample_dimensions: 41514.wav.npz: melgram.shape = (1, 96, 2586, 2) melgram dimensions: (1, 96, 2586, 2) Traceback (most recent call last): File "/home/guardian/.vscode/extensions/ms-python.python-2019.5.18875/pythonFiles/ptvsd_launcher.py", line 43, in <module> main(ptvsdArgs) File "/home/guardian/.vscode/extensions/ms-python.python-2019.5.18875/pythonFiles/lib/python/ptvsd/__main__.py", line 434, in main run() File "/home/guardian/.vscode/extensions/ms-python.python-2019.5.18875/pythonFiles/lib/python/ptvsd/__main__.py", line 312, in run_file runpy.run_path(target, run_name='__main__') File "/usr/lib/python3.6/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/usr/lib/python3.6/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/usr/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/guardian/Desktop/audio-classification/panotti/train_network.py", line 62, in <module> val_split=args.val, tile=args.tile) File "/home/guardian/Desktop/audio-classification/panotti/train_network.py", line 29, in train_network X_train, Y_train, paths_train, class_names = build_dataset(path=classpath, batch_size=batch_size, tile=tile) File "/home/guardian/Desktop/audio-classification/panotti/panotti/datautils.py", line 217, in build_dataset X = np.zeros((total_load, mel_dims[1], mel_dims[2], mel_dims[3])) MemoryError

Does samples have to be of a maximum length( i.e 5 seconds)?

[drscotthawley]

Hi. That "MemoryError" is not very descriptive. It could be an "Out of Memory" error, although usually the message is explicit. How much VRAM does your GPU have? I've generally only used the code with at least 8GB; but the limiting factor is the size of your dataset, which is a combination of how many files you have and how long each clip is. There is a command-line flag (try running with --help) for clipping the audio to a certain duration; the default is 3 seconds.
Perhaps you should try it with dataset of half that size (either half the number of files or half the duration) and see if it goes through ok?

[martirius]

Thanks for the reply, at the moment i'm trying to trainig without a GPU, only on CPU with 16 GB of RAM.
I think probably the problem is related to the size of dataset (and length of clips) because I debugged the execution of the program and stopped it before the crashing statement and
np.zeros(.....) is the indicted instruction because the space to allocate was enormous (something like 500GB!!)
I will try your suggestion and make the clips all of same length, thanks!