OSError: /PATH/TO/nvrtc_jit.so: cannot open shared object file: No such file or directory

[christfreund]

Hello,

I am using your SamplesLoss in a project on a NVIDIA GeForce RTX 2080 Ti and often-times I run into this error:

OSError: /PATH/TO/.cache/keops2.2.3/Linux_pc145_6.8.0-56-generic_p3.12.8/nvrtc_jit.so: cannot open shared object file: No such file or directory

The call stack is pretty long, so I attached it in a separate file.

error.txt

I use the loss to approximate the Wasserstein-distance between image-patches and observed, that this error occurs, whenever the two patches are larger than about 70x70 pixels while using the "sinkhorn" loss with default parameters. I noticed though, that the GPU is not nearly out of memory in this case. Now I started to try out the multiscale backend but in this case I always run into this error, no matter how large the input is. So I suppose it might have something to do with the sparse implementation? Also resetting the pykeops cache didn't solve the issue so far.

When importing geomloss from SamplesLoss, I also always get the two errors/warnings:

[KeOps] Compiling cuda jit compiler engine ... 
[KeOps] Warning : There were warnings or errors :
/usr/bin/ld: cannot find -lnvrtc: No such file or directory
collect2: error: ld returned 1 exit status

OK
[pyKeOps] Compiling nvrtc binder for python ... 
[KeOps] Warning : There were warnings or errors :
/usr/bin/ld: cannot find -lnvrtc: No such file or directory
collect2: error: ld returned 1 exit status

OK

Against this, I tried to reinstall KeOps, and torch and also set the environment variables as indicated here

export CODA_PATH="/usr/local/cuda"
export CUDA_HOME="/usr/local/cuda"
export CPLUS_INCLUDE_PATH="$CPLUS_INCLUDE_PATH:/usr/local/cuda/include"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"

but unfortunately that also didn't solve the issue. But as they are both about nvrtc, it seems like the two issues are related.

nvcc --version and nvidia-smi both indicate CUDA version 12.8.

Thanks in advance for any help you can offer! :)

[bcharlier]

Hi @christfreund , the new v2.3 release includes a method to display the path detected on your system.

pykeops.check_health()

It could be helpful to check if KeOps is able to find everything needed to compile formulas.