Logging wall time during training

[apoletayev]

This may be a dumb question, but I have not been able to find a way to log simple wall time during training in 0.7.0 and above. Prior to 0.7.0 there was cumulative_wall to log. I have tried with callbacks.Timer with weights and biases, but did not get it to work for anything other than overriding the max training time, which is of course the purpose of the timer. In principle, wandb has its own Timer class, so I imagine I could write a wrapper around a LightningModule to add it. But is there a simpler way to log the wall time during training?

[cw-tan]

Hi @apoletayev ,

Definitely not a dumb question -- it's not obvious at all. For weights and biases, I usually make a manual plot using _runtime on the y-axis and trainer/global_step on the x-axis. e.g.

Does this help for your use case? I imagine that we could also write custom callbacks or maybe PyTorch Lightning has some available for reporting more intricate timing details. If you are looking into writing custom code for custom functionality, we have an extension package docs page to help you get started. The NequIP framework is designed to be flexible enough for other developers to write extra code for custom functionality without having to fork a copy of nequip itself just to make changes necessary for your objectives. Of course, if you need functionality that can only be achieved through a change/addition in nequip, let us know and we can figure something out.

[apoletayev]

Yup _runtime is what I was looking for. Thank you for the answer!
It is very cool to see the extensions people have written already, the LES from Berkeley is on my to-try list.