Autograd/autocast error when training with OE #555
-
|
I have updated to 0.15.0 from 0.7.1 to try training with OpenEquivariance as described here: https://nequip.readthedocs.io/en/latest/guide/accelerations/openequivariance.html |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 5 replies
-
|
Hi @apoletayev , I'm guessing that PyTorch 2.6 might not have Sorry for the inconvenience, are there specific barriers to compiling more recent PyTorch versions from source (if you're already compiling PyTorch from source anyway)? Another thing to try is perhaps to downgrade the OEQ version, but I recall making some changes to our OEQ integration over the past few NequIP versions for compatibility. Potentially one thing for us to do is nail down maybe a NequIP vs OEQ vs PyTorch version compatibility table or something that may be helpful for users. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the response! Building torch from scratch is not very fun, in part because the whole toolchain through to LAMMPS needs to be checked / redone. But once there are (unfortunately cluster-specific) build scripts, everything becomes a little easier the second time around 😅 I'll give 2.7 a go and see what happens. |
Beta Was this translation helpful? Give feedback.
-
|
I'm guessing you're limited by your cluster's CUDA drivers? Otherwise you could just pip install latest PyTorch. If you're building from source anyway, I'd suggest using PyTorch 2.8, just because it's more recent. PyTorch 2.9 will be released this month, and that will allow OpenEquivariance to be used seamlessly with AOT Inductor compilation for inference. Otherwise, one just has to modify a line or so in the PyTorch code to get the OpenEquivariance AOTI compilation working in pre-2.9 torch versions. Note that training with OpenEquivariance should be fine with say PyTorch 2.7 to 2.8. |
Beta Was this translation helpful? Give feedback.
-
|
It's CUDA drivers and minimizing dependencies on cluster modules that insert themselves into path variables for builds. For example, I use PLUMED extensively, but building that is its own can of worms. So PLUMED comes from a module that is loaded when building LAMMPS, but that module brings its own dependencies, and those mess with cmake finding things. In the end, I more or less duplicate everything from gcc/g++ compilers onwards in a conda env, and that has seemed to work so far. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry about this, yes, this seems to be a PyTorch versioning issue; PT2.7 should work fine. The API evolves so rapidly that supporting multiple PyTorch versions quickly spirals into a big engineering effort. We might take a look at adding a guard for this in our code for PT2.6. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you both for the responses! I admit building pytorch was not my favorite part of updating, but I'll give 2.7 a shot and see how that goes. |
Beta Was this translation helpful? Give feedback.
-
|
I was able to train models with everything reconfigured around nequip=0.15.0 and pytorch=2.8.0, thank you for the pointer to update. However, I did get new errors when trying to run those ML-IAP models with LAMMPS built in these new configuration, so I started a different discussion: #556 |
Beta Was this translation helpful? Give feedback.
Sorry about this, yes, this seems to be a PyTorch versioning issue; PT2.7 should work fine. The API evolves so rapidly that supporting multiple PyTorch versions quickly spirals into a big engineering effort.
We might take a look at adding a guard for this in our code for PT2.6.