seconds/iteration is fast in first epoch, gets slower every subsequent epoch

[angadkalra]

I'm training a resnet101 3D model on grayscale images, 224x224x320, 16-bit precision, on VM with 4xV100 GPUs, using DDP and num_workers = 4. Batch size is 8 (2 per GPU). My first epoch goes very fast with 2.5s/it, but every epoch after that gets slower and slower and at epoch 6 I'm getting 6.5s/it. Any idea why this is happening or tips to speed up?

Thanks!

[ananthsub]

Do you have any data cached across epochs that you need to reset? Does the performance dramatically drop off at each epoch boundary, or does it gradually slow down during training execution?

[tchaton]

Dear @angadkalra,

Is your code open source and could be replicated, even with mocked data ?

Best,
T.C

[angadkalra]

No data that is cached across epochs. I'm using the metrics package to keep track of preds and targets for AUROC calculation but that gets reset at end of every epoch. Once the first epoch ends, the second one just starts very slow and gets to about 5s/it after 30-50 iterations, and then ends around the same value. Then every epoch after that is the same or slower.

@tchaton yes I can try and setup a colab for people to try...

I'm wondering in general if this is common or should the iteration speed be consistent across epochs?

[tchaton]

Yes, it should be consistent. A Colab would definitely be useful.

[complete-dope]

@tchaton Any updates on this ? what probably is causing this error ?
I am using pytorch lightning 1.9.1

[angadkalra]

For me it was a memory issue. I recommend the following:

• Upgrade to torch >= 2.0x and pytorch-lightning >= 2.1.x. Follow the upgrade guide in the docs.
• Check your system memory and cache during training. All of mine was being used up so when I increased my VM memory the problem went away. I get consistent training across epochs.
• Make sure that whatever values you collect during train/val, you delete at the end of epoch.
• Follow all the best training practices guide in the lightning docs for multi-gpu/multi-node training.

[AndreyStille]

@tchaton Are you interested at all in improving your shitty lib? Are you satisfied ignoring critical dummy issues ?

[angadkalra]

This is not helpful. The library is great and all of these ppl are helping for free. If you can't figure your problem out it's not their fault.

[AndreyStille]

This is not helpful. The library is great and all of these ppl are helping for free. If you can't figure your problem out it's not their fault.

Absolutely useless, only when I need to test some open-source code. What the point of making wrapper around wrapper instead of writing 10-20 more rows of code where you can observe all inner processes and have adequate error handling.

[AndreyStille]

@angadkalra Hello. Have you found out what was the cause of the issue?

[angadkalra]

See above.