libnvrtc not found, v13

[ccoulombe]

Description

When trying to show config or running the example below, Cupy fails to find libnvrtc:

RuntimeError: CuPy failed to load libnvrtc.so.12: OSError: libnvrtc.so.12: cannot open shared object file: No such file or directory

On our HPC systems, cuda libraries are installed on non-standard location.
This was never an issue until v13 (worked in previous versions).

To Reproduce

import numpy as np
import cupy as cp
a = np.zeros((10,10))
d = cp.asarray(a)
d += 1.  ## fails

Installation

Built from GitHub source

Environment

Linux, Centos, HPC system, python 3.11, cuda 12.2.
Cupy v13.0.0

Cuda libraries are not installed in standard location.

Additional Information

The rpaths are present and correct :

$ patchelf --print-rpath $VIRTUAL_ENV/lib/python3.11/site-packages/cupy_backends/cuda/libs/nvrtc.cpython-311-x86_64-linux-gnu.so 
/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Core/cudacore/12.2.2/lib64:/cvmfs/soft.computecanada.ca/easybuild/software/2023/x86-64-v3/Core/cudacore/12.2.2/lib

and librairies are found.

But nvrtc library is not needed :

$ patchelf --print-needed $VIRTUAL_ENV/lib/python3.11/site-packages/cupy_backends/cuda/libs/nvrtc.cpython-311-x86_64-linux-gnu.so 
libcublas.so.12
libcufft.so.11
libcurand.so.10
libcusparse.so.12
libstdc++.so.6
libm.so.6
libgcc_s.so.1
libc.so.6

A quick fix: add libnvrtc as needed :

$ patchelf --add-needed libnvrtc.so.12 $VIRTUAL_ENV/lib/python3.11/site-packages/cupy_backends/cuda/libs/nvrtc.cpython-311-x86_64-linux-gnu.so

[leofang]

NVRTC is now dlopen'd

cupy/cupy_backends/cuda/libs/_cnvrtc.pxi

Line 143 in a384b1a

return SoftLink(libname, prefix, mandatory=True)

it was part of v13 efforts (#7620) to help CuPy packages stay minimal. This means you need to make it searchable by the dynamic linker (say, set LD_LIBRARY_PATH). Could you give it a shot?

[ccoulombe]

Setting LD_LIBRARY_PATH is a workaround but we generally avoid using it. It works, but is not a solution unfortunately as we cannot tell thousands of users to set LD_LIBRARY_PATH for cupy to works. In my opinion, it is best to fail early if cupy depends on cuda.

ctypes find_library has a bug since forever where it does not return the full path of the library, hence not founding said library when they are installed in non standard location, such as on HPC systems. https://bugs.python.org/issue21042
We hacked ctypes with our own version to palliate for this issue, but even that does not work:

[coulombc@ng10403 ~]$ module load gcc python/3.11 cuda/12 custom-ctypes
[coulombc@ng10403 ~]$ makevenv 
(15787) [coulombc@ng10403 ~]$ pip install --no-index cupy-13.0.0+computecanada-cp311-cp311-linux_x86_64.whl 
(15787) [coulombc@ng10403 ~]$ python -c "import cupy; cupy.show_config()"

RuntimeError: CuPy failed to load libcudart.so.12: OSError: libcudart.so.12: cannot open shared object file: No such file or directory

I'll dig a bit more into why it did not work with our custom-ctypes, but since cupy links on cuda, would it be possible to offer distributors the option to soft or hard link on cuda libraries? Soft linking does not work for many of us unfortunately.

I could also patch cupy to add needed libs, but I'd prefer to avoid doing that systematically for every release now on.

[kmaehashi]

Thanks for the feedback @ccoulombe, this is a thing I overlooked when moving to ctypes-based loading. I think we can fix it on our side as follows:

1. Add a new extension module cupy_backends.cuda.libs._cuda_deps, which has libnvrtc.so.12 as NEEDED.
2. During import cupy, do import cupy_backends.cuda.libs._cuda_deps while ignoring any errors raised.

If libnvrtc.so.12 can be discovered in step 2, ctypes.CDLL("libnvrtc.so.12") will use that shared library. This should allow RPATH set during the build to be searched first.

[leofang]

I am still confused, though. @ccoulombe I agree no one wants users to set LD_LIBRARY_PATH, but isn't this HPC system admin's job to ensure there's a mechanism for putting DSOs in a linker-searchable path via some administrated mechanisms (ex: module load)? If the admins set up a proper mechanism and the user follows, ideally the DSOs would be discoverable at run time by the linker, and users don't have do anything weird. (And you don't have to build CuPy from source to hack RPATH.)

To make RPATH work, we still need the DSOs to be locatable by the linker, so I don't know what problem it'd solve?

[leofang]

@ccoulombe gentle ping, could you help us understand better?

[ccoulombe]

@leofang Sorry for the delay, thanks for the ping! :)
I'm out of office and want to take the time to provide a detailed response, I'll add it shortly!