continue training and longer length

[ypwang61]

Hi, thanks for your really great work! Have you tried on continue training (like on 24k context length) or using longer context length (like 36k or higher)? I just wonder when will this crazy thing saturates, or actually it's still quite powerful if giving more computation budget

[michaelzhiluo]

We're unsure about the limit of 1.5B, watch out for our future work! There are some new challenges we've encountered ;)

[ypwang61]

very cool! looking forward to your future works!