You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
$ singularity --version
singularity-ce version 3.11.1
Expected behavior
Having OpenCL GPU capabilities available inside a sif file with a proprietary simulation software Yasara just like directly executed on the host machine or also successfully in a Singularity v3.5.2 image few years back (on a host system without custom CUDA path).
Actual behavior
The proprietary software throws OpenCL compiler encountered an error, unfortunately without further specification.
CUDA 12.1 installed from run file at: /usr/local/cuda-12.1 (requires driver >=525.60.13)
nvidia-smi executed on host:
$ nvidia-smi
Mon May 8 11:56:31 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03 Driver Version: 530.41.03 CUDA Version: 12.1 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce GTX 1050 Off| 00000000:01:00.0 Off | N/A |
| N/A 65C P0 N/A / N/A| 1350MiB / 4096MiB | 51% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1150 G /usr/lib/xorg/Xorg 258MiB |
| 0 N/A N/A 2720 G cinnamon 51MiB |
| 0 N/A N/A 2776 G /usr/lib/firefox/firefox 162MiB |
| 0 N/A N/A 2792 G /usr/lib/thunderbird/thunderbird 114MiB |
| 0 N/A N/A 12429 C+G ./yasara 756MiB |
+---------------------------------------------------------------------------------------+
Inside Singularity container:
> nvidia-smi executed in Singularity
Mon May 8 12:06:18 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03 Driver Version: 530.41.03 CUDA Version: 12.1 |
|-----------------------------------------+----------------------+----------------------+
...
Detailed description
I'm assuming an issue with duplicate occurrences of OpenCL relevant files, since 3 libraries listed in nvliblist.conf are present twice on this host system:
Since repository-based OpenCL libraries are needed by few tools installed from the linux repos, there must be a way to tell Singularity, which library versions to use for OpenCL/GPU calculations inside the container. Using the --nv flag and the absolute path /usr/local/cuda-12.1/targets/x86_64-linux/lib/libOpenCL.so in nvliblist.conf is obviously not sufficient.
Executed directly on my host (where the GPU is utilized properly) clinfo gives me lots of output, but interestingly, without any ICD related lines at the end. However, there's a note saying: "NOTE: your OpenCL library only supports OpenCL 2.2":
Number of platforms 1
Platform Name NVIDIA CUDA
Platform Vendor NVIDIA Corporation
Platform Version OpenCL 3.0 CUDA 12.1.98
Platform Profile FULL_PROFILE
...
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) No platform
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) No platform
clCreateContext(NULL, ...) [default] No platform
clCreateContext(NULL, ...) [other] Success [NV]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) Invalid device type for platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform
NOTE: your OpenCL library only supports OpenCL 2.2,
but some installed platforms support OpenCL 3.0.
Programs using 3.0 features may crash
or behave unexpectedly
Executed inside Singularity, there ICD related lines but the final note mentions OpenCL version 2.1:
singularity exec --cleanenv --nv --bind /etc/OpenCL --overlay image-overlay.img image.sif /bin/bash
Singularity> clinfo
Number of platforms 1
Platform Name NVIDIA CUDA
Platform Vendor NVIDIA Corporation
Platform Version OpenCL 3.0 CUDA 12.1.98
Platform Profile FULL_PROFILE
...
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) NVIDIA CUDA
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) Success [NV]
clCreateContext(NULL, ...) [default] Success [NV]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) Invalid device type for platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform
ICD loader properties
ICD loader Name OpenCL ICD Loader
ICD loader Vendor OCL Icd free software
ICD loader Version 2.2.11
ICD loader Profile OpenCL 2.1
NOTE: your OpenCL library only supports OpenCL 2.1,
but some installed platforms support OpenCL 3.0.
Programs using 3.0 features may crash
or behave unexpectedly
And finally, if I in addition bind the entire CUDA lib path to /nvlib (added to LD_LIBRARY_PATH inside the container), I get the same clinfo output as on the host itself (no ICD lines, OpenCL 2.2):
singularity exec --cleanenv --nv --bind /etc/OpenCL --bind /usr/local/cuda-12.1/targets/x86_64-linux/lib:/nvlib --overlay image-overlay.img image.sif /bin/bash
Singularity> clinfo
Number of platforms 1
Platform Name NVIDIA CUDA
Platform Vendor NVIDIA Corporation
Platform Version OpenCL 3.0 CUDA 12.1.98
Platform Profile FULL_PROFILE
...
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) No platform
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) No platform
clCreateContext(NULL, ...) [default] No platform
clCreateContext(NULL, ...) [other] Success [NV]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) Invalid device type for platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform
NOTE: your OpenCL library only supports OpenCL 2.2,
but some installed platforms support OpenCL 3.0.
Programs using 3.0 features may crash
or behave unexpectedly
However, the issue in Yasara execution remains. The singularity installation instructions for OpenCL usage are not straight-forward enough at least for me. Can someone please shed light on which libraries needs to come from which tool or might be missing/perturbing in my singularity system?
The text was updated successfully, but these errors were encountered:
Version of Singularity:
Expected behavior
Having OpenCL GPU capabilities available inside a sif file with a proprietary simulation software Yasara just like directly executed on the host machine or also successfully in a Singularity v3.5.2 image few years back (on a host system without custom CUDA path).
Actual behavior
The proprietary software throws
OpenCL compiler encountered an error
, unfortunately without further specification.What OS/distro are you running
How did you install Singularity
By source.
Installed NVIDIA/CUDA versions
/usr/local/cuda-12.1
(requires driver >=525.60.13)nvidia-smi executed on host:
Inside Singularity container:
Detailed description
I'm assuming an issue with duplicate occurrences of OpenCL relevant files, since 3 libraries listed in
nvliblist.conf
are present twice on this host system:Since repository-based OpenCL libraries are needed by few tools installed from the linux repos, there must be a way to tell Singularity, which library versions to use for OpenCL/GPU calculations inside the container. Using the
--nv
flag and the absolute path/usr/local/cuda-12.1/targets/x86_64-linux/lib/libOpenCL.so
innvliblist.conf
is obviously not sufficient.Executed directly on my host (where the GPU is utilized properly)
clinfo
gives me lots of output, but interestingly, without any ICD related lines at the end. However, there's a note saying: "NOTE: your OpenCL library only supports OpenCL 2.2":Executed inside Singularity, there ICD related lines but the final note mentions OpenCL version 2.1:
And finally, if I in addition bind the entire CUDA lib path to
/nvlib
(added toLD_LIBRARY_PATH
inside the container), I get the sameclinfo
output as on the host itself (no ICD lines, OpenCL 2.2):However, the issue in Yasara execution remains. The singularity installation instructions for OpenCL usage are not straight-forward enough at least for me. Can someone please shed light on which libraries needs to come from which tool or might be missing/perturbing in my singularity system?
The text was updated successfully, but these errors were encountered: