Getting "out of CUDA memory error" just for some videos with wierd _frames folder behavior

[anlutfi]

--EDIT-- Running nvidia-smi I can see the GPU indeed running out of memory. Is there a way to free RAM when it's no longer needed? I'm understanding that the frames are all being loaded into the memory eventually

Hello,

I am running into a weird error while running i3d feature extraction. Here is the command:

python main.py \
--feature_type i3d \
--on_extraction save_numpy \
--device_ids 0
--extraction_fps 25
--video_paths ../../../tmp/a73fd64a-10ae-44f6-ae03-ea95f910833e_0.mp4
--output_path ../../../tmp

And here is the error:

CUDA out of memory. Tried to allocate 746.00 MiB (GPU 0; 7.79 GiB total capacity; 2.06 GiB already allocated; 687.19 MiB free; 1.63 GiB cached)
Extraction failed at: ../../../tmp/a73fd64a-10ae-44f6-ae03-ea95f910833e_0.mp4. Continuing extraction

So, there is two notes here:
1 - The math does not add up, 7.79 GB - 2.06 already allocated - 1.63 cached = way more than the 746 MB needed.

2 - And this is the weird part. As you can see, the name of the video ends with a _0, as it's a segment of an original video. For two of those segments, i3d works normally. For these same two, there is NO _frames folder under BMT/submodules/video_features/tmp. For every failed one there is.

Can you guys help me? Tell me if I'm being too vague

Thanks so much for your time
Antonio

[v-iashin]

Hi,

Thanks for providing the details.

1. CUDA out of memory. is a PyTorch error, not BMT's. I tried to google for "cuda out of memory doesn't add up": here are some links (1, 2). If I would see this error, I would think that my GPU doesn't have enough memory to run the code.
2. I am not sure what you mean. The code extracts frames for every video before feeding them into I3D, then, these frames are removed. See https://github.com/v-iashin/video_features/blob/4fa02bd5c5b8c34081dcfb609e2bcd5a973eaab2/models/i3d/extract_i3d.py#L84-L98. If I3D fails to extract features, they are never removed.

It seems like you are splitting the video into segments which might indeed be useful but not in this implementation. The code processes stacks of 64 RGB frames (65 for the flow). It doesn't load the whole video into the memory – only these stacks. Therefore, your GPU should be able to digest at least 64 frames regardless of the video length.

In my case, the feature extraction allocates 7.96 GB of memory for 64 frames. I am guessing your GPU doesn't have enough memory, hence, the error, and I am sorry if it is. You may try to adjust the stack_size parameter to e.g. 32 and it will fit into your memory.

[anlutfi]

Hi, thanks for the reply!

I understood what was happening. I made an object to run multiple videos and it kept the feature extraction/captioning models loaded, so GPU memory was being hoarded. Even though your code only runs 64 frames at a time, pytorch doesn't free the memory in a perfectly efficient way. So, I could see that after 3 or 4 batches of 64 frames, the process failed with the mentioned error.

As for number 2, yeah that makes perfect sense, thank you.

Thanks again for your time and the work available here

[v-iashin]

Why can’t you make an executable which would run the shell command for feature extraction first, then the next shell command will produce the captioning based on the saved features. Then repeat for the next video.

I am not sure which batches you are referring to. In my case, the mem footprint is fixed after the first batch and doesn't accumulate regardless of the video length

[jandog8990]

Hi, so what was the fix for out of memory GPU error? Do we have to reduce the 64 frames? And if so where in the code do we do that? Maybe that's something that should be configurable?

[jandog8990]

If anyone needs help with this, set your stack_size and step_size to half the default of 64 like so:

--stack_size 32 --step_size 32