GPU Partial Usage and Detection Issues on HPC

[PandoraMagis]

Hi,

I'm installing ReLERNN on an HPC and need to set it up within a Conda environment that includes a specific CUDA version.

However, every execution or environment change I've made results in only partial GPU utilization. The GPU cards are requested multiple times—sometimes they are detected but followed by an error (see below), and other times they are not detected at all within the same execution.

Even more frustrating, the process appears in the NVIDIA-SMI panel, consuming around 4GB of VRAM.

I do install TensorFlow 2.15 and TensorFlow RT using pip.

If you need additional error logs, I can provide them.

Thanks for your help!

GPU found but with some errors that i still struggle to fix - mid of execution

2025-03-26 15:05:07.401896: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different 
computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 15:05:07.403892: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 15:05:07.440774: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 15:05:07.441209: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-03-26 15:05:07.934751: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2025-03-26 15:05:08.450044: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA n
ode, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2025-03-26 15:05:08.487091: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if 
you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

GPU not found - start and end of execution

2025-03-26 15:03:56.835916: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different 
computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-03-26 15:03:56.837956: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 15:03:56.873827: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used.
2025-03-26 15:03:56.874272: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-03-26 15:03:57.352307: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

[andrewkern]

can you give some more details about the conda environment? the output of conda list should be a good start

[PandoraMagis]

Hi the list is joined as a file in this message,
It can be resumed by :

conda create -n ReLERNN-1.0.0-MagisCarpe -c conda-forge -c nvidia -c HCC cudatoolkit=12.5 cuda-version=12.5 cuDNN=8.9 cuda-libraries-dev cuda-nvcc cuda-cupti cuda-nvtx keras msprime scikit-learn matplotlib scikit-allel python=3.10
# then pip of tensor flow/rt ; then relerne 
pip install tensorflow==2.15
pip install tenrort # I tried with and without tensorrt without any best result

But i did test many version combining,
What i tried as first, was more in touch with the git mentioned version, was :

conda create -n ReLERNN-1.0.0-MagisCarpe cudatoolkit-10.1.243 cudnn-7.6.5.32 python=3.10 msprime scikit-learn matplotlib scikit-allel 
pip install tensorflow==2.15

For other environment details : the HPC is running on Ubuntu 20.04, with some Nvidia L40s on Driver Version: 550.54.14 and CUDA Version: 12.4.

ReLERNNconda_list.txt

[andrewkern]

So looking over things it looks like the correct libs are being installed, but given that you are getting warnings like:

W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if 
you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...

I'm guessing that there is still a driver mismatch for the hardware on your system. Can you open your conda env on a compute node on your system tell me what the output the following looks like?

nvcc --version

also the output from

nvidia-smi

[andrewkern]

by the way this has nothing to do with ReLERNN per se-- this is about how your HPC is managing their nvidia setup. You might just reach out to them and ask what they recommend for using tensorflow 2.15

[OliverPStuart]

I'm having this issue as well. I installed all prerequisites following the mamba method described in the readme.

When I run nvcc --version the output is:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Jan_15_19:20:09_PST_2025
Cuda compilation tools, release 12.8, V12.8.61
Build cuda_12.8.r12.8/compiler.35404655_0

I have exactly the same issue as @PandoraMagis. It seems as if the GPU is being recognised, when I run nvidia-smi the output is:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A30                     On  | 00000000:21:00.0 Off |                    0 |
| N/A   31C    P0              31W / 165W |    237MiB / 24576MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A30                     On  | 00000000:E2:00.0 Off |                    0 |
| N/A   26C    P0              26W / 165W |      7MiB / 24576MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A   2371331      C   /opt/mambaforge/bin/python3.10              224MiB |
+---------------------------------------------------------------------------------------+

However, the model training is still slow, projected to take more than two days on the example data. I get the same stderr message:

2025-08-04 22:28:42.440669: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2256] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.

I've reached out to the sysadmin for help but just documenting things here as well.

[OliverPStuart]

Sys admin and I have both tried to debug it without much luck. We used the following slightly modified installation with mamba:

# Step 1: Create the Conda environment with required packages
mamba create -n relearnn-1.0.0-v2 -c conda-forge python=3.10 

# Step 2: Activate the environment
mamba activate relearnn-1.0.0-v2

mamba install tensorflow=2.15.0 cuda-toolkit h5py

# Step 4: Clone and install ReLERNN
git clone https://github.com/kr-colab/ReLERNN.git
cd ReLERNN
pip install .

The conda environment looks like this: conda_env.txt

Then, we run a test script to check that tensorflow is installed and can perform GPU operations:

import tensorflow as tf

# Create a simple tensor and perform an operation
a = tf.constant([[1.0, 2.0], [3.0, 4.0]])
b = tf.constant([[1.0, 1.0], [0.0, 1.0]])
c = tf.matmul(a, b)

print("\nResult of matrix multiplication:")
print(c)

# To explicitly check where a tensor is placed:
with tf.device('/GPU:0'): # Or /GPU:1 to test the second one
    d = tf.constant([[5.0, 6.0]])
    e = tf.matmul(d, a)
    print("\nResult of explicit GPU operation:")
    print(e)

Result:

Result of matrix multiplication:
tf.Tensor(
[[1. 3.]
 [3. 7.]], shape=(2, 2), dtype=float32)

Result of explicit GPU operation:
tf.Tensor([[23. 34.]], shape=(1, 2), dtype=float32)

After that, we run the example_pipeline_pool.sh script inside the conda environment (this is the pipeline I plan to use). It proceeds as expected and completes the simulation steps:

2025-08-06 18:08:11.415428: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-08-06 18:08:11.415530: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-08-06 18:08:11.416249: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-08-06 18:08:11.420944: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Warning: no demographic history file found. All training data will be simulated under demographic equilibrium.
Split chromosome: 2L...
Split chromosome: 2R...
Split chromosome: 3L...
Split chromosome: X...
Split chromosome: 3R...
poolFILE: ./example_output_pool/splitPOOLs/example_2L:0-840000.pool
poolFILE: ./example_output_pool/splitPOOLs/example_2R:0-1669000.pool
poolFILE: ./example_output_pool/splitPOOLs/example_3L:0-742000.pool
poolFILE: ./example_output_pool/splitPOOLs/example_3R:0-1963000.pool
poolFILE: ./example_output_pool/splitPOOLs/example_X:0-1250000.pool

Accessibility mask found: calculating the proportion of the genome that is masked...
0.9% of genome inaccessible

Simulating with window size = 136200 bp.

Training set:
Simulate...
Validation set:
Simulate...
Test set:
Simulate...

SIMULATIONS FINISHED!

SANITY CHECK
====================
numSegSites                     Min     Mean    Max
Simulated:                      447     1338    3917
Input 2L:0-840000:              234     1318    1752
Input 2R:0-1669000:             164     1336    1752
Input 3L:0-742000:              474     1340    1752
Input 3R:0-1963000:             156     1335    1752
Input X:0-1250000:              331     1339    1752


***ReLERNN_SIMULATE_POOL.py FINISHED!***

2025-08-06 18:08:32.231351: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-08-06 18:08:32.231434: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-08-06 18:08:32.232335: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-08-06 18:08:32.237169: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: SSE4.1 SSE4.2 AVX AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Warning: training data to be treated as if generated by pool-seq
2025-08-06 18:08:33.838542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 22288 MB memory:  -> device: 0, name: NVIDIA A30, pci bus id: 0000:21:00.0, compute capability: 8.0
2025-08-06 18:08:42.652147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1929] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 22288 MB memory:  -> device: 0, name: NVIDIA A30, pci bus id: 0000:21:00.0, compute capability: 8.0
Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 input_1 (InputLayer)        [(None, 3927, 2)]         0

 bidirectional (Bidirection  (None, 168)               44352
 al)

 dense (Dense)               (None, 256)               43264

 dropout (Dropout)           (None, 256)               0

 dense_1 (Dense)             (None, 1)                 257

=================================================================
Total params: 87873 (343.25 KB)
Trainable params: 87873 (343.25 KB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
Epoch 1/1000
2025-08-06 18:09:16.220534: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:454] Loaded cuDNN version 8907
2025-08-06 18:09:17.551220: I external/local_xla/xla/service/service.cc:168] XLA service 0x7f425bb9e8e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2025-08-06 18:09:17.551285: I external/local_xla/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA A30, Compute Capability 8.0
2025-08-06 18:09:17.556364: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1754467757.675074 2533755 device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
 12/100 [==>...........................] - ETA: 28s - loss: 1.0303

But it is unbearably slow, even though tensorflow seems to be able to access the GPU:

$ nvidia-smi
Wed Aug  6 18:10:32 2025
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.86.10              Driver Version: 535.86.10    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A30                     On  | 00000000:21:00.0 Off |                    0 |
| N/A   36C    P0              49W / 165W |   2347MiB / 24576MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A30                     On  | 00000000:E2:00.0 Off |                    0 |
| N/A   31C    P0              46W / 165W |      7MiB / 24576MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A   2531998      C   ...vs/relearnn-1.0.0-v2/bin/python3.10     2334MiB |
+---------------------------------------------------------------------------------------+

We also had to increase the open file limit first ulimit -n 5000 before the script would run successfully at all. Is that expected? I don't see it listed anywhere in the documentation.

[andrewkern]

this looks like it is running on your GPU @OliverPStuart. Can you confirm that the GPU is seeing load? A good tool for this is nvitop

as for the ulimit that's about your system and not the software

[OliverPStuart]

Looks like no, nvitop tells me that the average CPU usage is about 40 % while the GPU usage is 0%.

[andrewkern]

hmmm... you are still having issues that are related to your compute setup. i'm guessing there are duplicate versions of tensorflow/cuda throughout the system.

First please run this and report the output:

import tensorflow as tf
print(tf.__version__)
print(tf.sysconfig.get_build_info().get("cuda_version"))
print(tf.sysconfig.get_build_info().get("cudnn_version"))

then check the LD_LIBRARY_PATH:

echo "LD_LIBRARY_PATH=$LD_LIBRARY_PATH"

# Run a tiny GPU op and keep the process alive briefly:
python - <<'PY'
import os, time, tensorflow as tf
print("LD_LIBRARY_PATH:", os.environ.get("LD_LIBRARY_PATH"))
with tf.device("/GPU:0"):
    x = tf.random.normal([2048,2048])
    y = tf.matmul(x, x)
print("ok"); time.sleep(30)  # so we can inspect /proc
PY

# In another shell, inspect loaded CUDA libs (paths should be from ONE place)
pid=$(pgrep -n -f "python -")
grep -E "libcudnn|libcublas|libcufft|libcusolver|libcuda" /proc/$pid/maps | awk '{print $6}' | sort -u

we're looking for a mix of system level and user level installs

[OliverPStuart]

Thanks for your help.

First step:

>>> import tensorflow as tf
2025-08-14 01:58:06.789906: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-08-14 01:58:06.790026: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-08-14 01:58:06.791603: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-08-14 01:58:06.799879: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-08-14 01:58:07.664399: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
>>> print(tf.__version__)
2.15.0
>>> print(tf.sysconfig.get_build_info().get("cuda_version"))
12.2
>>> print(tf.sysconfig.get_build_info().get("cudnn_version"))
8

Second step:

/mnt/data/dayhoff/home/u6905905/.local/lib/python3.10/site-packages/nvidia/cuda_runtime/lib/libcudart.so.12
/usr/lib/x86_64-linux-gnu/libcuda.so.535.86.10

Exactly as you've said! I've no idea how to solve this, though.

[andrewkern]

okay you've got a mixed environment issue. the key here is to unset your LD_LIBRARY_PATH or specify it explicitly.

if you run the following, does it relieve the “factory already registered” lines?

# 1) One-off test with a clean loader environment
env -u LD_LIBRARY_PATH python - <<'PY'
import tensorflow as tf, os
print("TF", tf.__version__)
print("LD_LIBRARY_PATH", os.environ.get("LD_LIBRARY_PATH"))
print("Build CUDA/cuDNN", tf.sysconfig.get_build_info().get("cuda_version"),
      tf.sysconfig.get_build_info().get("cudnn_version"))
with tf.device('/GPU:0'):
    x = tf.random.normal([4096,4096]); y = tf.matmul(x, x)
print("ok")
PY

[OliverPStuart]

Nope, the result is the same.

2025-08-14 02:39:07.989550: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-08-14 02:39:07.989669: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-08-14 02:39:07.991193: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-08-14 02:39:07.999409: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-08-14 02:39:09.869546: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
TF 2.15.0
LD_LIBRARY_PATH None
Build CUDA/cuDNN 12.2 8
2025-08-14 02:39:11.768054: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:274] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2025-08-14 02:39:11.768134: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:129] retrieving CUDA diagnostic information for host: dayhoff.uds.anu.edu.au
2025-08-14 02:39:11.768150: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:136] hostname: dayhoff.uds.anu.edu.au
2025-08-14 02:39:11.768313: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:159] libcuda reported version is: 535.86.10
2025-08-14 02:39:11.768371: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:163] kernel reported version is: 535.86.10
2025-08-14 02:39:11.768386: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:241] kernel version seems to match DSO: 535.86.10
ok

[andrewkern]

hmm... out of interest are you running this on a cluster? did you just run that test on a head node that doesn't have GPUs exposed?

[OliverPStuart]

This is all on a cluster, yes. I ran these in an interactive SLURM job on the GPU partition:
salloc --nodes=1 --ntasks=1 --mem=56G --partition=GPU --gres=gpu:0 --time=00:20:00 srun --pty bash -i