LR Scheduler and Optimizer

[lukas-blecher]

Ideas for training speed up by choosing optimal optimization algorithms.

As optimizer I only ever tried Adam and AdamW. They seem to perform quite well but I had the feeling that AdamW is a better fit.

Now for the LR Scheduler.
It has a quite big effect on the training progress. I've mostly used OneCycleLR until now. But the loss either stagnated or got even worse after some time. That's why I continued the training after a couple of epochs with a "fresh" OneCycle.
Maybe using a cyclic scheduler from the start would be the way to go. Something like CosineAnnealingLR.
Does anybody have experience with other schedulers/optimizers?

[jalola]

I am current training as well, I got BLEU=0.89 1 time, but then it fluctuated between 0.7 to 0.8x all the time.
I used different optimizer (AdamW) with very small lr (0.0001). When I use Adam, the accuracy fluctuate much more (0.4-0.7)

I guess there is something to do with the lr as you mentioned. I am planning to make it faster if possible for deployment.

[lukas-blecher]

Thank you for trying to improve the training. I'm very interested in the progress.
Once I have the resources again I will try out what you are suggesting.

Please keep me posted if you find something!

[oakmoos]

Excuse me, if you used a self-generated dataset or the dataset provided by this project for training? I trained using the dataset provided by the project and found that progress was very slow without loading any checkpoints/(ㄒoㄒ)/~~. Thank you!

[lukas-blecher]

I've managed to achieve a good performance and loss without restarts by replacing the OneCycleLR with the StepLR.
Commit coming soon

[tuiiitendinh]

Could you please show us the configuration of OneCycleLR? I have encountered the error as the picture below

[lukas-blecher]

I don't use OneCycleLR anymore and it's not supported without changing the code, sorry.