Starts a new training from last model with changing training parameters.

[youngjae142]

Hello! I'm enjoying using NequIP these days.
There is one idea for NequIP: It would be awesome if there is an option to stop updating the learning rate at some point while the training keeps going.

As I know, there is an option for the early stop of training, namely, early_stopping_lower_bounds.
With this option, however, the training should stop even though I want more with a fixed learning rate.
I'm not sure but I cannot continue the training by changing the options like learning_rate and lr_scheduler_factor, or can I?
So I'm thinking that an option for discarding the learning rate scheduler at some point would do great!

[Linux-cpp-lisp]

hi @ssrokyz ,

the easiest way to go about this is to do two trainings: one with early_stopping_lower_bounds, and then a second with a fixed learning rate that starts from the state of the first.

For the second training, you want to duplicate the config of your first one but to change:

1. remove the LR early stopping condition
2. remove the LR scheduler
3. set the LR to the low LR you want to stay on
4. add the following:

model_builders:
- SimpleIrrepsConfig
- EnergyModel
- PerSpeciesRescale
- ForceOutput
- RescaleEnergyEtc
- initialize_from_state

initial_model_state: /my/previous/early-stopped/training/run/last_model.pth

The model_builders given are the default model builders, with the addition of initialize_from_state, which initializes the model when it is first built using weights loaded from the model state file pointed to by initial_model_state, which you should set to the path to the last_model.pth of the run that got early stopped by the LR dropping.

Let me know if this works for you!

[youngjae142]

This is exactly what I needed!
Training seems continuous from the previous model since the loss value is continuous.
By changing run_name, it independently starts from a new folder.
Thx again @Linux-cpp-lisp ! 😊

[CalvinGe]

I am doing the same thing and something weird happened. The loss for my validation is continous with my previous run, but when it starts training, the loss goes up obviously high immediately. Do you know what might be the cause of this issue? @Linux-cpp-lisp