New feature: adding a parameter to control the number of pseudo-validation cases #2061
Conversation
|
Hey, I just did a quick benchmark of the time the validation takes:
Validation in its default setting is less than 10% of the runtime, which seems very reasonable to me. Do you really think that this is an urgently needed feature? I would like to avoid changing the parametrization if the trainers init as much as possible because then everyone would have to go through all their code and modify that in their custom trainers. That would affect hundrets of trainers in our case |
No, I would not say this is urgently needed. It would allow customizing the number of pseudo-validation steps, but it's not something critical. Several trainers (e.g.: StuNet v1) already use 1 validation step for training.
This is not always the case. When using HDD or when having other I/O bottleneck, the validation takes a lot of time.
Yes, I agree with this, that's why I added the new parameter at the end, with a default value. By doing this I expect that all trainers inheriting from already existing nnUnet trainers would use the default value and further configurations after the |
This feature allows to do fewer iterations of pseudo-validation instead of 50, allowing for faster training or more confident pseudo-validation score, depending on the needs of each experiment. It also works together with #2053 by allowing no additional processes for pseudo-validation if
-val_itersis set to 1.While this value could be adjusted by creating a custom trainer, it does not influence the training results, therefore there's no need to create a custom trainer for it.