GPU Processing Issue after latest driver updates

[Sveeggen]

Apologies in advance, my coding experience is virtually nonexistent, but I can no longer get SRRF to run.

The error:

EM com.amd.aparapi.internal.kernel.KernelRunner executeOpenCL
WARNING: ### CL exec seems to have failed. Trying to revert to Java ###

The last time I used SRRF, I was able to blaze through a .nji file in under 30 seconds, but now it's claiming to require an hour at 0 FPS. Based on my reading, it sounds like the program is no longer able to utilize my GPU? This is frustrating because this plug-in was giving us great data with zero issues prior to now!

Since I last used it about a month ago, I believe there have been updates to FIJI, Java, and my NVIDIA drivers. As a habit I updated these all automatically, but now I can't find old files to roll back to and prove it's the new drivers causing issues. Regarding the NVIDIA drivers, I have tried both the gaming and studio drivers to no avail.

My computer info:

Operating System: Windows 10 Home 64-bit
Card name: NVIDIA GeForce GTX 1060
Card driver: 441.20 (11/12/2019)
Java Version: 8 Update 231
FIJI Version: 1.52r
SRRF Version: 1.14 Stable 1

Any ideas? This really is an amazing plug-in that our lab would hate to not be able to use.

Edit: Update

I have also checked OpenCL in ImageJ and tested that OpenCL is able to detect my graphics card as successfully test it with Mandelbrot. So ImageJ and OpenCL can recognize my graphics card but for some reason SRRF can no longer use OpenCL?

[XarlesSta]

Dear Sveeggen, were you able to resolve this issue? I am in the same rabbit hole and I have tried every driver there is. Any of them works. I know my Fiji is also communicating with my GPU. For example, CLIJ works but NanoJ does not.

[Sveeggen]

Dear Sveeggen, were you able to resolve this issue? I am in the same rabbit hole and I have tried every driver there is. Any of them works. I know my Fiji is also communicating with my GPU. For example, CLIJ works but NanoJ does not.

Sadly I am still having trouble. We have ended up using an older computer to run it - though it takes around 3 minutes per image, that's at least manageable. I worry there needs to be an update or something to work with newer computers, but I have no idea how to go about that.

[joaomamede]

Congrats Dr. Henriques in starting your lab at IGC!!

I have the same error and message about openCL:
EM com.amd.aparapi.internal.kernel.KernelRunner executeOpenCL
WARNING: ### CL exec seems to have failed. Trying to revert to Java ###

I tested the openCL with the NanoJ-core test for openCL and the animation and test runs properly.
When I try to run NanoJ-SRRF I have the same error as you.

I have tried in my Lenovo Legion Y540 with a Geforce RTX 2060 and in my lab's microscope computer HP 4 with a Geforce Quadro .

I have tried in windows 10 and in Ubuntu 20.04.
Operating Systems : Ubuntu 20.04 + Windows 10 Home 64-bit + Windows 10 Pro 64-bits
Card name: NVIDIA GeForce RTX 2060 / Nvidia Quadro
Card driver: 440.82-0ubuntu0~0.20.04.1 (Ubuntu) + 451.22 (windows)
FIJI Version: 1.52p
Java Running: 1.8.0_172(64-bits)
NanoJ-Core: 2.1RC1
SRRF Version: 1.14 Stable1
I have CUDA libraries installed in both my Ubuntu 20.04 and W10 in my lenovo.

From my Ubuntu install NanoJ-Core tests.
"
Running Device Tests
NVIDIA Corporation_GPU
Identity of device used: - NVIDIA Corporation_GPU
global:589824 local:(derived)1024
Execution mode=GPU
Executed at 748.0FPS
Best Device: - NVIDIA Corporation_GPU
"

I know that in linux the CUDA components changed recently. Would this be connected to it?
My camera is 2048x2044 so using CPU in a 100 frame image is basically 1h+.
I could use CPU in a small ROI sucessfully.

[XarlesSta]

This config works for me:

1. Open NVIDIA Control Panel --> Manage 3D settings.
2. Go to Global Settings tab.
3. In the setting window, change the 'Power Management Mode' to 'Prefer Maximum Performance'.
4. Also check below settings too,
   • Power management mode – prefer maximum performance
   • Ambient Occlusion – Off
   • Triple buffering – off
   • Threaded optimization - ON
   • Vertical Sync – Adaptive
5. Click on Save and restart the Computer.
6. Run the program and check.

Please let me know if it works for you.

[joaomamede]

Did not help me (windows). Still drops to CPU mode.
Linux doesn't have any of those options in nvidia-settings.

[joaomamede]

Okay. I think it's something to do with the 440+ drivers.
As I got my laptop from my new workplace today I tested it right away, and I made a lucky mistake of installing old drivers.

It did work.
System:

Red hat enterprise Linux 8
Quadro T2000
NVIDIA-SMI 430.26 Driver Version: 430.26 CUDA Version: 10.2

we are actually getting an EMCCD with SRRF, but I'd like to also do SRRF with my CMOS as well.
Any way we can make the plugin more verbose to see if there's a library problem or something like that?

I'm installing the most recent 440 right now to see if I can reproduce the error.

EDIT1:
after installing 440.100 it fails with the opencl error. I also tried 450 beta from the rhel cuda repository and that resulted in crashing imageJ

Outputs from the app nvidia-smi tool in linux (to show versions)
Works:
NVIDIA-SMI 430.26 Driver Version: 430.26 CUDA Version: 10.2

Fail:
NVIDIA-SMI 440.100 Driver Version: 440.100 CUDA Version: 10.2
NVIDIA-SMI 450.36.06 Driver Version: 450.36.06 CUDA Version: 11.0

EDIT2:
Windows 10 64bits with Quadro RTX 4000
Works : 423.28-quadro-desktop-notebook
Fail: 436.30-quadro-desktop-notebook ; 451.48-quadro-desktop-notebook

[DSbioimaging]

I am also experiencing this problem on a 2070S under Windows 10

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 451.82 Driver Version: 451.82 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+

[cilia-etc]

I was having the same error (...CL exec seems to have failed...) and I was able to resolve it on a Windows 10 machine with an NVIDIA Quadro P4000. To resolve the error, I reverted back to the original drivers from March, 16 2018. The driver version according to the Windows Device Manager is: 23.21.13.9125.

To revert back to these drivers, I went in through:
Device Manager -> Update Driver ->Browse for Driver -> Let me pick..

The earliest driver was from March 16, 2018 (presumably from when the system was built). So, I picked that driver and restarted the system.

SRRF is now working at GPU speeds after doing the above.

I am not the most savvy GPU person and so I don't know the difference between the Windows Device Manager version numbers and the 450, 440, 430 numbers listed on the Nvidia downloads page. However, I was not able to get SRRF to work with either the 440 or 430 driver installers that I downloaded from the Nvidia webpage. It was only when I went in through Device Manager that SRRF started working again.

I hope this helps someone.

[DSbioimaging]

I was having the same error (...CL exec seems to have failed...) and I was able to resolve it on a Windows 10 machine with an NVIDIA Quadro P4000. To resolve the error, I reverted back to the original drivers from March, 16 2018. The driver version according to the Windows Device Manager is: 23.21.13.9125.

To revert back to these drivers, I went in through:
Device Manager -> Update Driver ->Browse for Driver -> Let me pick..

The earliest driver was from March 16, 2018 (presumably from when the system was built). So, I picked that driver and restarted the system.

SRRF is now working at GPU speeds after doing the above.

I am not the most savvy GPU person and so I don't know the difference between the Windows Device Manager version numbers and the 450, 440, 430 numbers listed on the Nvidia downloads page. However, I was not able to get SRRF to work with either the 440 or 430 driver installers that I downloaded from the Nvidia webpage. It was only when I went in through Device Manager that SRRF started working again.

I hope this helps someone.

Thank you for replying, unfortunately the newer graphic card models (like my 2070S and the upcoming 3XXX models) do not have the option of rolling back to older drivers, the earliest driver supporting my graphic card happens to have a higher version number than the one you are using.
I will try to post on Nvidia forums but given the love they have previously shown for OpenCL, I am less than hopeful that they will do anything about it.

Repacking the project with a newer Aparapi version could fix the issue, but I lack the skills to do so. The whole project feels abandoned (Especially the Bitbucket dead links).

[joaomamede]

I tried to open the Nano-Updater.jar, and just replace the naparapi .so files inside to see if that would work. Doing so breaks NanoJ.

The package that needs the update is in NanoJ-core (updater). The .dll and .so files are in that github, and I believe they come from aparapi-jni

[joaomamede]

I'm trying to get this recompiled with more recent aparapi.
The reason why we can't just change the libraries in updater is that before it was
"import com.amd.aparapi"
and with more recent versions it's:
"import com.aparapi"

I am trying to recompile everything and I spent a fair amount of time finding maven libraries.
I got stuck here (when compiling the module NanoJ-Core it can't find libraries within its own src directory.

I'm a python guy, so I have never done this.
Any tips?

[joaomamede]

I managed to make a jar with the latest aparapi.

I have this problem in Fiji. If it fixed the nvidia driver issue,I will share the jars and try to fork the updates I did to nanoj-core into github.

There was a problem with the class org.apache.bcel.Repository which can be found here:
/home/jmamede/Downloads/fiji-linux64(1)/Fiji.app/jars/xalan-2.7.2.jar
/home/jmamede/Downloads/fiji-linux64(1)/Fiji.app/plugins/NanoJ-Core.jar
/home/jmamede/Downloads/fiji-linux64(1)/Fiji.app/plugins/NanoJ-Updater.jar

WARNING: multiple locations found!
java.lang.NoSuchMethodError: org.apache.bcel.Repository.lookupClass(Ljava/lang/Class;)Lorg/apache/bcel/classfile/JavaClass;
	at com.aparapi.internal.model.ClassModel.parse(ClassModel.java:2842)
	at com.aparapi.internal.model.ClassModel.<init>(ClassModel.java:187)
	at com.aparapi.internal.model.ClassModel.createClassModelInternal(ClassModel.java:676)
	at com.aparapi.internal.model.ClassModel.access$300(ClassModel.java:88)
	at com.aparapi.internal.model.ClassModel$4.compute(ClassModel.java:671)
	at com.aparapi.internal.model.ClassModel$4.compute(ClassModel.java:668)
	at com.aparapi.internal.model.ValueCache.computeIfAbsent(ValueCache.java:50)
	at com.aparapi.internal.model.ClassModel.createClassModel(ClassModel.java:682)
	at com.aparapi.internal.kernel.KernelRunner.executeInternalInner(KernelRunner.java:1474)
	at com.aparapi.internal.kernel.KernelRunner.executeInternalOuter(KernelRunner.java:1383)
	at com.aparapi.internal.kernel.KernelRunner.execute(KernelRunner.java:1374)
	at com.aparapi.Kernel.execute(Kernel.java:2897)
	at com.aparapi.Kernel.execute(Kernel.java:2854)
	at com.aparapi.Kernel.execute(Kernel.java:2829)
	at nanoj.updater.java.InstallNanoJDependencies.<init>(InstallNanoJDependencies.java:36)
	at nanoj.core.java.gui._BaseDialog_.<clinit>(_BaseDialog_.java:37)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at ij.IJ.runUserPlugIn(IJ.java:233)
	at ij.IJ.runPlugIn(IJ.java:198)
	at ij.Executer.runCommand(Executer.java:150)
	at ij.Executer.run(Executer.java:68)
	at java.lang.Thread.run(Thread.java:748)

[ERROR] java.lang.NoSuchMethodException: Could not find method org.apache.bcel.Repository.lookupClass(Ljava/lang/Class;)Lorg/apache/bcel/classfile/JavaClass;
There was a problem with the class org.apache.bcel.Repository which can be found here:
/home/jmamede/Downloads/fiji-linux64(1)/Fiji.app/jars/xalan-2.7.2.jar
/home/jmamede/Downloads/fiji-linux64(1)/Fiji.app/plugins/NanoJ-Core.jar
/home/jmamede/Downloads/fiji-linux64(1)/Fiji.app/plugins/NanoJ-Updater.jar

WARNING: multiple locations found!
java.lang.NoSuchMethodError: org.apache.bcel.Repository.lookupClass(Ljava/lang/Class;)Lorg/apache/bcel/classfile/JavaClass;
	at com.aparapi.internal.model.ClassModel.parse(ClassModel.java:2842)
	at com.aparapi.internal.model.ClassModel.<init>(ClassModel.java:187)
	at com.aparapi.internal.model.ClassModel.createClassModelInternal(ClassModel.java:676)
	at com.aparapi.internal.model.ClassModel.access$300(ClassModel.java:88)
	at com.aparapi.internal.model.ClassModel$4.compute(ClassModel.java:671)
	at com.aparapi.internal.model.ClassModel$4.compute(ClassModel.java:668)
	at com.aparapi.internal.model.ValueCache.computeIfAbsent(ValueCache.java:50)
	at com.aparapi.internal.model.ClassModel.createClassModel(ClassModel.java:682)
	at com.aparapi.internal.kernel.KernelRunner.executeInternalInner(KernelRunner.java:1474)
	at com.aparapi.internal.kernel.KernelRunner.executeInternalOuter(KernelRunner.java:1383)
	at com.aparapi.internal.kernel.KernelRunner.execute(KernelRunner.java:1374)
	at com.aparapi.Kernel.execute(Kernel.java:2897)
	at com.aparapi.Kernel.execute(Kernel.java:2854)
	at com.aparapi.Kernel.execute(Kernel.java:2829)
	at nanoj.updater.java.InstallNanoJDependencies.<init>(InstallNanoJDependencies.java:36)
	at nanoj.core.java.gui._BaseDialog_.<clinit>(_BaseDialog_.java:37)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at ij.IJ.runUserPlugIn(IJ.java:233)
	at ij.IJ.runPlugIn(IJ.java:198)
	at ij.Executer.runCommand(Executer.java:150)
	at ij.Executer.run(Executer.java:68)
	at java.lang.Thread.run(Thread.java:748)

	at com.aparapi.internal.model.ClassModel.parse(ClassModel.java:2842)
	at com.aparapi.internal.model.ClassModel.<init>(ClassModel.java:187)
	at com.aparapi.internal.model.ClassModel.createClassModelInternal(ClassModel.java:676)
	at com.aparapi.internal.model.ClassModel.access$300(ClassModel.java:88)
	at com.aparapi.internal.model.ClassModel$4.compute(ClassModel.java:671)
	at com.aparapi.internal.model.ClassModel$4.compute(ClassModel.java:668)
	at com.aparapi.internal.model.ValueCache.computeIfAbsent(ValueCache.java:50)
	at com.aparapi.internal.model.ClassModel.createClassModel(ClassModel.java:682)
	at com.aparapi.internal.kernel.KernelRunner.executeInternalInner(KernelRunner.java:1474)
	at com.aparapi.internal.kernel.KernelRunner.executeInternalOuter(KernelRunner.java:1383)
	at com.aparapi.internal.kernel.KernelRunner.execute(KernelRunner.java:1374)
	at com.aparapi.Kernel.execute(Kernel.java:2897)
	at com.aparapi.Kernel.execute(Kernel.java:2854)
	at com.aparapi.Kernel.execute(Kernel.java:2829)
	at nanoj.updater.java.InstallNanoJDependencies.<init>(InstallNanoJDependencies.java:36)
	at nanoj.core.java.gui._BaseDialog_.<clinit>(_BaseDialog_.java:37)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at ij.IJ.runUserPlugIn(IJ.java:233)
	at ij.IJ.runPlugIn(IJ.java:198)
	at ij.Executer.runCommand(Executer.java:150)
	at ij.Executer.run(Executer.java:68)
	at java.lang.Thread.run(Thread.java:748)

[pawlowska]

@joaomamede have you managed to get a working version for the plugin? I belong to this unlucky group whose drivers are too new for the official version

[joaomamede]

I did not. I compiled it but i have never done Java or fiji plug-ins. Some of the libs should come from fiji, some from maven and things got messed up and it does not run.

[joaomamede]

Old bitbucket stuff can be found here:

https://web.archive.org/web/20200624140819/https://bitbucket.org/rhenriqueslab/nanoj-srrf/wiki/Getting%20set%20up%20with%20SRRF

https://web.archive.org/web/20200622021429/https://bitbucket.org/rhenriqueslab/nanoj-core/wiki/Home

Something interesting is, that nano-core always picks the best card.
We can probably unload the nvidia module if we have a prime laptop and run in the Intel HD, but we can't select Intel HD because the internal nanoj-core code picks the best card and only list that one. (talking about linux btw)