AIMV2 as the encoder, unfreezing it and setting the learning rate to 2e-6 results in the LLaVA-NEXT model achieving a loss of 0

[1359347500cwc]

When using AIMV2 as the encoder, unfreezing it and setting the learning rate to 2e-6 leads to the LLaVA-NEXT model reaching a loss of 0 after 3000-4000 steps. The original paper kept the encoder frozen. Why is it not recommended to unfreeze it for training? If I decide to unfreeze it, what learning rate should I set?