Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc) #61

Open
mayanka98 opened this issue Aug 31, 2023 · 1 comment
Open

RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc) #61

mayanka98 opened this issue Aug 31, 2023 · 1 comment

Comments

@mayanka98
Copy link

I am facing the above error while running the training script. Can someone let me know how to solve it?

My GPU specifications are-
image

And the packages in conda are-
# Name Version Build Channel
_libgcc_mutex 0.1 main conda-forge
_pytorch_select 0.2 gpu_0 anaconda
absl-py 0.8.1 py37_0 conda-forge
asn1crypto 1.3.0 py37_0 conda-forge
blas 1.0 mkl conda-forge
ca-certificates 2020.1.1 0 anaconda
certifi 2020.4.5.1 py37_0 anaconda
cffi 1.14.0 py37h2e261b9_0 anaconda
chardet 3.0.4 py37_1003 conda-forge
cryptography 2.8 py37h1ba5d50_0 anaconda
cuda-command-line-tools 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0
cuda-compiler 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0
cuda-cudart 11.3.58 hc1aae59_0 nvidia/label/cuda-11.3.0
cuda-cuobjdump 11.3.58 hc78e225_0 nvidia/label/cuda-11.3.0
cuda-cupti 11.3.58 h9a3dd33_0 nvidia/label/cuda-11.3.0
cuda-cuxxfilt 11.3.58 he670d9e_0 nvidia/label/cuda-11.3.0
cuda-gdb 11.3.58 h531059a_0 nvidia/label/cuda-11.3.0
cuda-libraries 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0
cuda-libraries-dev 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0
cuda-memcheck 11.3.58 h8711ecb_0 nvidia/label/cuda-11.3.0
cuda-nvcc 11.3.58 h2467b9f_0 nvidia/label/cuda-11.3.0
cuda-nvdisasm 11.3.58 hd2ea46e_0 nvidia/label/cuda-11.3.0
cuda-nvml-dev 12.2.128 0 nvidia
cuda-nvprof 11.3.58 h860cd9e_0 nvidia/label/cuda-11.3.0
cuda-nvprune 11.3.58 hb917323_0 nvidia/label/cuda-11.3.0
cuda-nvrtc 11.3.58 he300756_0 nvidia/label/cuda-11.3.0
cuda-nvtx 11.3.58 h3fa534a_0 nvidia/label/cuda-11.3.0
cuda-nvvp 11.3.58 hd16380c_0 nvidia/label/cuda-11.3.0
cuda-samples 11.6.101 h8efea70_0 nvidia
cuda-sanitizer-api 11.3.58 h58da6c8_0 nvidia/label/cuda-11.3.0
cuda-thrust 11.3.58 h7b74f08_0 nvidia/label/cuda-11.3.0
cuda-toolkit 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0
cuda-tools 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0
cuda-visual-tools 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0
cudatoolkit 10.0.130 0 anaconda
cudnn 7.6.5.32 ha8d7eb6_1 conda-forge
cymem 1.31.2 py37h6bb024c_0 anaconda
cytoolz 0.9.0.1 py37h14c3975_1 anaconda
dill 0.2.9 py37_0 conda-forge
en-core-web-sm 2.0.0 pypi_0 pypi
future 0.17.1 py37_0 anaconda
idna 2.9 py_1 conda-forge
intel-openmp 2020.0 166 anaconda
joblib 0.14.1 py_0 conda-forge
ld_impl_linux-64 2.33.1 h53a641e_7 conda-forge
libcublas 11.4.2.10064 h8a72295_0 nvidia/label/cuda-11.3.0
libcufft 10.4.2.58 h58ccd86_0 nvidia/label/cuda-11.3.0
libcurand 10.2.4.58 h99380db_0 nvidia/label/cuda-11.3.0
libcusolver 11.1.1.58 hec68242_0 nvidia/label/cuda-11.3.0
libcusparse 11.5.0.58 hf5aa513_0 nvidia/label/cuda-11.3.0
libedit 3.1.20181209 hc058e9b_0 anaconda
libffi 3.2.1 hd88cf55_4
libgcc-ng 9.1.0 hdf63c60_0 anaconda
libgfortran-ng 7.3.0 hdf63c60_0 anaconda
libnpp 11.3.3.44 h8df316f_0 nvidia/label/cuda-11.3.0
libnvjpeg 11.4.1.58 h3d06750_0 nvidia/label/cuda-11.3.0
libprotobuf 3.11.4 h8b12597_0 conda-forge
libstdcxx-ng 9.1.0 hdf63c60_0 anaconda
markdown 3.2.1 py_0 conda-forge
mkl 2020.0 166 anaconda
mkl-service 2.3.0 py37he904b0f_0
mkl_fft 1.0.15 py37ha843d7b_0
mkl_random 1.1.0 py37hd6b4f25_0
msgpack-numpy 0.4.4.3 py_0 conda-forge
msgpack-python 0.5.6 py37h6bb024c_1 anaconda
murmurhash 0.28.0 py37hf484d3e_0 anaconda
ncurses 6.2 he6710b0_1 anaconda
ninja 1.9.0 py37hfd86e86_0 anaconda
numpy 1.15.4 py37h7e9f1db_0
numpy-base 1.15.4 py37hde5b4d6_0
openjdk 8.0.152 h7b6447c_3 anaconda
openssl 1.1.1g h7b6447c_0 anaconda
pandas 0.24.2 py37he6710b0_0 anaconda
pip 20.0.2 py37_1 conda-forge
plac 0.9.6 py37_0 anaconda
preshed 1.0.1 py37he6710b0_0 anaconda
protobuf 3.11.4 py37h3340039_1 conda-forge
pycparser 2.20 py_0 conda-forge
pyopenssl 19.1.0 py37_0 conda-forge
pysocks 1.7.1 py37_0 conda-forge
python 3.7.7 hcf32534_0_cpython anaconda
python-dateutil 2.8.1 py_0 conda-forge
python_abi 3.7 1_cp37m conda-forge
pytorch 1.2.0 cuda100py37h938c94c_0
pytz 2020.1 py_0 anaconda
readline 8.0 h7b6447c_0 anaconda
regex 2018.07.11 py37h14c3975_0 anaconda
requests 2.23.0 py37_0 conda-forge
scikit-learn 0.22.1 py37hd81dba3_0
scipy 1.3.1 py37h7c811a0_0
setuptools 46.1.3 py37_0 anaconda
six 1.14.0 py37_0 conda-forge
spacy 2.0.12 py37h962f231_0 anaconda
sqlite 3.31.1 h62c20be_1 anaconda
tensorboard 1.14.0 py37_0 conda-forge
termcolor 1.1.0 py37_1 anaconda
thinc 6.10.3 py37h962f231_0 anaconda
tk 8.6.8 hbc83047_0 anaconda
toolz 0.10.0 py_0 conda-forge
torchtext 0.3.1 pypi_0 pypi
tqdm 4.46.0 py_0 anaconda
ujson 2.0.3 py37he6710b0_0 anaconda
urllib3 1.25.8 py37_0 anaconda
werkzeug 1.0.1 pyh9f0ad1d_0 conda-forge
wheel 0.34.2 py37_0 conda-forge
wrapt 1.10.11 py37h14c3975_2 anaconda
xz 5.2.5 h7b6447c_0 anaconda
zlib 1.2.11 h7b6447c_3 anaconda
@threegold116
Copy link

I am facing the above error while running the training script. Can someone let me know how to solve it?

My GPU specifications are- image

And the packages in conda are- # Name Version Build Channel _libgcc_mutex 0.1 main conda-forge _pytorch_select 0.2 gpu_0 anaconda absl-py 0.8.1 py37_0 conda-forge asn1crypto 1.3.0 py37_0 conda-forge blas 1.0 mkl conda-forge ca-certificates 2020.1.1 0 anaconda certifi 2020.4.5.1 py37_0 anaconda cffi 1.14.0 py37h2e261b9_0 anaconda chardet 3.0.4 py37_1003 conda-forge cryptography 2.8 py37h1ba5d50_0 anaconda cuda-command-line-tools 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0 cuda-compiler 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0 cuda-cudart 11.3.58 hc1aae59_0 nvidia/label/cuda-11.3.0 cuda-cuobjdump 11.3.58 hc78e225_0 nvidia/label/cuda-11.3.0 cuda-cupti 11.3.58 h9a3dd33_0 nvidia/label/cuda-11.3.0 cuda-cuxxfilt 11.3.58 he670d9e_0 nvidia/label/cuda-11.3.0 cuda-gdb 11.3.58 h531059a_0 nvidia/label/cuda-11.3.0 cuda-libraries 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0 cuda-libraries-dev 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0 cuda-memcheck 11.3.58 h8711ecb_0 nvidia/label/cuda-11.3.0 cuda-nvcc 11.3.58 h2467b9f_0 nvidia/label/cuda-11.3.0 cuda-nvdisasm 11.3.58 hd2ea46e_0 nvidia/label/cuda-11.3.0 cuda-nvml-dev 12.2.128 0 nvidia cuda-nvprof 11.3.58 h860cd9e_0 nvidia/label/cuda-11.3.0 cuda-nvprune 11.3.58 hb917323_0 nvidia/label/cuda-11.3.0 cuda-nvrtc 11.3.58 he300756_0 nvidia/label/cuda-11.3.0 cuda-nvtx 11.3.58 h3fa534a_0 nvidia/label/cuda-11.3.0 cuda-nvvp 11.3.58 hd16380c_0 nvidia/label/cuda-11.3.0 cuda-samples 11.6.101 h8efea70_0 nvidia cuda-sanitizer-api 11.3.58 h58da6c8_0 nvidia/label/cuda-11.3.0 cuda-thrust 11.3.58 h7b74f08_0 nvidia/label/cuda-11.3.0 cuda-toolkit 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0 cuda-tools 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0 cuda-visual-tools 11.3.0 h3b286be_0 nvidia/label/cuda-11.3.0 cudatoolkit 10.0.130 0 anaconda cudnn 7.6.5.32 ha8d7eb6_1 conda-forge cymem 1.31.2 py37h6bb024c_0 anaconda cytoolz 0.9.0.1 py37h14c3975_1 anaconda dill 0.2.9 py37_0 conda-forge en-core-web-sm 2.0.0 pypi_0 pypi future 0.17.1 py37_0 anaconda idna 2.9 py_1 conda-forge intel-openmp 2020.0 166 anaconda joblib 0.14.1 py_0 conda-forge ld_impl_linux-64 2.33.1 h53a641e_7 conda-forge libcublas 11.4.2.10064 h8a72295_0 nvidia/label/cuda-11.3.0 libcufft 10.4.2.58 h58ccd86_0 nvidia/label/cuda-11.3.0 libcurand 10.2.4.58 h99380db_0 nvidia/label/cuda-11.3.0 libcusolver 11.1.1.58 hec68242_0 nvidia/label/cuda-11.3.0 libcusparse 11.5.0.58 hf5aa513_0 nvidia/label/cuda-11.3.0 libedit 3.1.20181209 hc058e9b_0 anaconda libffi 3.2.1 hd88cf55_4 libgcc-ng 9.1.0 hdf63c60_0 anaconda libgfortran-ng 7.3.0 hdf63c60_0 anaconda libnpp 11.3.3.44 h8df316f_0 nvidia/label/cuda-11.3.0 libnvjpeg 11.4.1.58 h3d06750_0 nvidia/label/cuda-11.3.0 libprotobuf 3.11.4 h8b12597_0 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0 anaconda markdown 3.2.1 py_0 conda-forge mkl 2020.0 166 anaconda mkl-service 2.3.0 py37he904b0f_0 mkl_fft 1.0.15 py37ha843d7b_0 mkl_random 1.1.0 py37hd6b4f25_0 msgpack-numpy 0.4.4.3 py_0 conda-forge msgpack-python 0.5.6 py37h6bb024c_1 anaconda murmurhash 0.28.0 py37hf484d3e_0 anaconda ncurses 6.2 he6710b0_1 anaconda ninja 1.9.0 py37hfd86e86_0 anaconda numpy 1.15.4 py37h7e9f1db_0 numpy-base 1.15.4 py37hde5b4d6_0 openjdk 8.0.152 h7b6447c_3 anaconda openssl 1.1.1g h7b6447c_0 anaconda pandas 0.24.2 py37he6710b0_0 anaconda pip 20.0.2 py37_1 conda-forge plac 0.9.6 py37_0 anaconda preshed 1.0.1 py37he6710b0_0 anaconda protobuf 3.11.4 py37h3340039_1 conda-forge pycparser 2.20 py_0 conda-forge pyopenssl 19.1.0 py37_0 conda-forge pysocks 1.7.1 py37_0 conda-forge python 3.7.7 hcf32534_0_cpython anaconda python-dateutil 2.8.1 py_0 conda-forge python_abi 3.7 1_cp37m conda-forge pytorch 1.2.0 cuda100py37h938c94c_0 pytz 2020.1 py_0 anaconda readline 8.0 h7b6447c_0 anaconda regex 2018.07.11 py37h14c3975_0 anaconda requests 2.23.0 py37_0 conda-forge scikit-learn 0.22.1 py37hd81dba3_0 scipy 1.3.1 py37h7c811a0_0 setuptools 46.1.3 py37_0 anaconda six 1.14.0 py37_0 conda-forge spacy 2.0.12 py37h962f231_0 anaconda sqlite 3.31.1 h62c20be_1 anaconda tensorboard 1.14.0 py37_0 conda-forge termcolor 1.1.0 py37_1 anaconda thinc 6.10.3 py37h962f231_0 anaconda tk 8.6.8 hbc83047_0 anaconda toolz 0.10.0 py_0 conda-forge torchtext 0.3.1 pypi_0 pypi tqdm 4.46.0 py_0 anaconda ujson 2.0.3 py37he6710b0_0 anaconda urllib3 1.25.8 py37_0 anaconda werkzeug 1.0.1 pyh9f0ad1d_0 conda-forge wheel 0.34.2 py37_0 conda-forge wrapt 1.10.11 py37h14c3975_2 anaconda xz 5.2.5 h7b6447c_0 anaconda zlib 1.2.11 h7b6447c_3 anaconda

bro, you should check the verison between cuda (nvcc-V) and torch version (nvcc-V), and the version of numpy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@threegold116 @mayanka98