[bug] icemix.py port: Optimizer not handling weight decay for cls_token

[timinar]

The DeepIce model contains a method called no_weight_decay() which is intended to specify that the cls_token parameter should not be subject to weight decay during training:

@torch.jit.ignore
def no_weight_decay(self) -> Set:
    """cls_tocken should not be subject to weight decay during training."""
    return {"cls_token"}

However, optimizer_grouped_parameters are not specified during training, so this method has no effect.
I believe that in the original 2nd place code, FastAI's wrapper around AdamW handled this automatically.

[chenlinear]

Yes, this method def no_weight_decay() is called in BEiT-2 when training. However, I searched for it and it is not called in the original 2nd place solution or fastai.

Also, I can find optimizer_grouped_parameters in BEiT-2, but not in the original 2nd place solution or fastai.

The 2nd place solution uses fastai.vision.all.OptimWrapper. But fastai.vision.all.OptimWrapper or its base class do not contain this method or call this method.

I find this to be a historical reason by BEiT-2, and maybe this method def no_weight_decay() can be removed.