Restart from ckpt: set training data

[Doris2022]

Hi~ Great job!(^_^)
I have a few questions about reproducing your results. After about 200 steps, the program crashed because of OOM. Do you know if I want to restart from the last ckpt, where can I set the training data——not using previews training data in the 200 steps? Is that possible for the current implementation?
Thank you very much for your help!!!

[michaelzhiluo]

Not currently possible in this current implementation; however, the main branch in VERL should have some code that can be ported over for saved optimizer and data loader states.

Our internal version is up to date with verl, watch out for our future releases ;)

[Doris2022]

Not currently possible in this current implementation; however, the main branch in VERL should have some code that can be ported over for saved optimizer and data loader states.

Our internal version is up to date with verl, watch out for our future releases ;)

Thank you ! Looking forward to that!

[SwordFaith]

Is their any easy way to merge verl subdirectory of deepscaler with verl new release ?