RFC: Add early stopping for all LR schedulers based on validation score improvements

[mcognetta]

This is an example implementation to address #215 , which allows early stopping based on validation metric improvements regardless of the learning rate scheduler used. Prior to this, only the plateau scheduler could trigger early stopping, and that was based on it reducing the learning rate of the optimizer to below the min_lr setting. This change allows for specifying a threshold of successive validation runs, which if no improvement on the validation score is observed, ends training early.

An example training config is also included (configs/iwslt14_deen_bpe_early_stopping.yaml).

Note: I manually copied this from a personal fork that has a number of other changes, so I haven't been able to fully test it yet.

[may-]

Awesome!

Should we maybe save the value of self.stats.validation_steps_since_last_improvement in the state_dict to make it resumable, when someone want to continue training from a saved ckpt? just like self.stats.best_ckpt_score?