Explosive gradients for 40B tokens

[egor-2]

Thanks Andrej for your tutorial! It's amazing to learn so many things from just one YouTube video

I tried to rerun you experiment with 8GPUs and 4 x 10B tokens with shuffled dataset before sharding. The training dynamics looks good up until approximately 60K step. Then the norm of gradient starts growing and the performance degrades quickly. See picture attached.

Was wondering, whether it is normal, and whether it is possible to somehow escape such point without retraining from scratch again?

[pavanpreet-gandhi]

Interesting, I'm not sure why this would happen - but if you save your model state and optimizer state along the way, maybe you could continue training from the most recent save?

[egor-2]

Indeed seems like the standard practice, but I couldn't find much details in the papers. Only in the Meta's OPT paper they mention briefly loss divergence and they just roll back to previous checkpoint. A colleague of mine also pointed out the mention of "loss spikes" in Reka AI report "Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models".

[MasterSkepticista]

Do you mind sharing your training config and L2 norm curve?

[egor-2]

log (1).txt
Not sure about config. I saved the log file but the norm is not there

[MasterSkepticista]

Modify your summary logging somewhere like this and plot the norm stream:

train_logs.append((step, loss_accum, norm))
if (step % 100 == 0) and master_process and args.workdir:
     with open(logfile, "a") as f:
          for (s, tl, l2_n) in train_logs:
              f.write(f"{s},train,{tl}\n")
              f.write(f"{s},norm,{l2_n}\n")

By training config, I meant your settings for batch_size, learning_rate, warmup etc.