Loss Distribution at 400k and 600k steps - OWT - 124M model

[MarkoKarbevski]

What is a reasonable interval of the losses at 400k and 600k steps for the default ~124M model of the repo on OWT?

My val loss is around 2.95 at 400k and 2.90 at 600k steps.
Should I be concerned that I might be doing something wrong, or is this reasonable?

I am asking because the plotted val loss is around 2.905 in Karpathy's run at 400k steps.

[SrijanSriv211]

It depends on the quality of dataset you are using, however it seems reasonable to be to be honest. As models train more and more the jump in loss gets smaller and smaller so I'm honestly not surprised if it took 200k steps to go from 2.95 to 2.90, although I would say that does seem a bit much but again it depends on the dataset and your model size. I'll say there's nothing wrong with it.

[SrijanSriv211]

Should I be concerned that I might be doing something wrong, or is this reasonable?

I am asking because the plotted val loss is around 2.905 in Karpathy's run at 400k steps.

No you shouldn't be concerned because it is generally due to initialization of the weights. If you are using same seed as Karpathy did then it could due to either due to the tokenization, dataset or some hyperparameters.

To put it simply there's nothing wrong, it's totally fine and reasonable. Also everything should be fine until your model is generating coherent enough text. If not then there's some problem.

[MarkoKarbevski]

Should I be concerned that I might be doing something wrong, or is this reasonable?
I am asking because the plotted val loss is around 2.905 in Karpathy's run at 400k steps.

No you shouldn't be concerned because it is generally due to initialization of the weights. If you are using same seed as Karpathy did then it could due to either due to the tokenization, dataset or some hyperparameters.

To put it simply there's nothing wrong, it's totally fine and reasonable. Also everything should be fine until your model is generating coherent enough text. If not then there's some problem.

I am training it (against another variant) for scientific purposes, but was wondering if I may have made a mistake to get that type of a loss

[SrijanSriv211]

Should I be concerned that I might be doing something wrong, or is this reasonable?
I am asking because the plotted val loss is around 2.905 in Karpathy's run at 400k steps.

No you shouldn't be concerned because it is generally due to initialization of the weights. If you are using same seed as Karpathy did then it could due to either due to the tokenization, dataset or some hyperparameters.
To put it simply there's nothing wrong, it's totally fine and reasonable. Also everything should be fine until your model is generating coherent enough text. If not then there's some problem.

I am training it (against another variant) for scientific purposes, but was wondering if I may have made a mistake to get that type of a loss

Nope, I don't think you made any mistake. It's normal to get that type of loss. Good luck with your project.