Using validation set for early stopping

[sivomke]

Hi, everyone!

I have created a benchmark with dataset_benchmark that contains 3 experiences, and then have added a validation set to this benchmark with benchmark_with_validation_stream. I am trying to use Early Stopping with validation set: for each experience, I would like to use loss on the validation set of the corresponding experience. I have indicated metric_name = 'Loss_Exp' and val_stream_name='valid' , however, it still uses Loss_Exp/eval_phase/train_stream of the corresponding experience instead. Could you please tell me what would be the correct way to use validation set for early stopping?

Here is the script

Thank you!

[vlomonaco]

Hi @sivomke! I had a quick look into your script and I didn't find any issue. This may be a bug. We will look into it asap!

[AntonioCarta]

Thanks. This looks like a bug. If you open an issue with a minimal example it can help to fix it quickly.

[sivomke]

I have created minimal example:
python min-reproducable-example.py --early_stopping --training_batch_size 8 --eval_batch_size 1 --epochs 25 --lr 2e-4
For first experience, we have the first 75 samples with target 'zeros', and last 25 samples with target 'ones', in benchmark_with_validation_stream we create 25% validation split, and by setting shuffle=False we use the last 25 samples of the first experience as a validation set, hence for the first experience the error on validation set will increase.

[vlomonaco]

I created #904 to track the issue!

[sivomke]

Hello!

I have found a workaround that works for my application, however, it may not be the most general way to do this. Since the problem was that the full name of the metric also contains the experience, and for each experience, I evaluate on the validation set of the corresponding experience (res = strategy.train(experience, eval_streams=[benchmark.valid_stream[i]])), instead of trying to retrieve the metric by the key (in the example Loss_Exp/eval_phase/valid_stream, which caused the problem since the full name of the computed metric contains the experience and task name), I retrieve the last computed metric that starts with the key :

res = strategy.evaluator.get_last_metrics()
full_name =[k for k in res.keys() if k.startswith(self.metric_key)][-1]
val_acc = res.get(full_name)

Hence in my application, during the training on the first experience, the retrieved value would be Loss_Exp/eval_phase/valid_stream/Task000/Exp000, while for the second experience it would be Loss_Exp/eval_phase/valid_stream/Task000/Exp001.

[vlomonaco]

Thanks @sivomke for reporting your fix (that may be useful to others) and for your patience, we will try to solve the bug soon.