[FreeBSD] mmul and Execution error (UnsatisfiedLinkError) at org.jblas.NativeBlas/dgemm

[lungsi]

I am aware this seems to be a recurring issue but after spending couple of days trying most of the proposed solutions (here in Wiki and issues as well as in other forums) I have not been able to fix the problem.

Before I describe the problem here is some context:

OS: FreeBSD 12.1-RELEASE r354233 GENERIC amd64
jblas: 1.2.5

What/How I am using jblas?

Clojure is the primary language. I am calling jblas functions via Clojure's Java Interop

Error using mmul

I can successfully (without error) use the matrix multiplication function mmul when I am multiplying a row vector (1 x n) and a column vector (n x 1).

But when I use mmul to multiply a column vector (m x 1) and a row vector (1 x n) or among two compatible matrices I get

Execution error (UnsatisfiedLinkError) at org.jblas.NativeBlas/dgemm (NativeBlas.java:-2).
org.jblas.NativeBlas.dgemm(CCIIID[DII[DIID[DII)V

Because row * column can be performed using mmul I can use this to fill out the product (matrix) for column * row and matrix * matrix. But I'd prefer not to do this. Can somebody help me how to fix this dependency in FreeBSD?

Since jblas 1.2.5 has moved to openblas I have installed this and other potential dependencies

• pkg install openblas
• pkg install gcc
• pkg install linux-c7-libgfortran

The installed openblas libraries (pkg info -lx openblas) are in

openblas-0.3.10,1:
        /usr/local/include/cblas.h
        /usr/local/include/f77blas.h
        /usr/local/include/lapack.h
        /usr/local/include/lapacke.h
        /usr/local/include/lapacke_config.h
        /usr/local/include/lapacke_mangling.h
        /usr/local/include/lapacke_utils.h
        /usr/local/include/openblas_config.h
        /usr/local/lib/cmake/openblas/OpenBLASConfig.cmake
        /usr/local/lib/cmake/openblas/OpenBLASConfigVersion.cmake
        /usr/local/lib/libopenblas.a
        /usr/local/lib/libopenblas.so
        /usr/local/lib/libopenblas.so.0
        /usr/local/lib/libopenblasp-r0.3.10.a
        /usr/local/lib/libopenblasp-r0.3.10.so
        /usr/local/libdata/pkgconfig/openblas.pc
        /usr/local/share/licenses/openblas-0.3.10,1/BSD3CLAUSE
        /usr/local/share/licenses/openblas-0.3.10,1/LICENSE
        /usr/local/share/licenses/openblas-0.3.10,1/catalog.mk

Looking at the source code ~/.m2/repository/org/jblas/jblas/1.2.5/jblas-1.2.3.jar/org//jblas/NativeBlastLibraryLoader.class I am wondering if I need to change this to fix the problem (I would prefer not to use this approach).

package org.jblas;

import org.jblas.exceptions.UnsupportedArchitectureException;
import org.jblas.util.LibraryLoader;
import org.jblas.util.Logger;

class NativeBlasLibraryLoader {
    NativeBlasLibraryLoader() {
    }

    static void loadLibraryAndCheckErrors() {
        try {
            try {
                System.loadLibrary("jblas");
            } catch (UnsatisfiedLinkError var3) {
                Logger.getLogger().config("BLAS native library not found in path. Copying native library from the archive. Consider installing the library somewhere in the path (for Windows: PATH, for Linux: LD_LIBRARY_PATH).");
                loadDependentLibraries();
                (new LibraryLoader()).loadLibrary("jblas", true);
            }

            double[] a = new double[1];
            NativeBlas.dgemm('N', 'N', 1, 1, 1, 1.0D, a, 0, 1, a, 0, 1, 1.0D, a, 0, 1);
        } catch (UnsatisfiedLinkError var4) {
            String arch = System.getProperty("os.arch");
            String name = System.getProperty("os.name");
            if (name.startsWith("Windows") && var4.getMessage().contains("Can't find dependent libraries")) {
                System.err.println("On Windows, you need some additional support libraries.\nFor example, you can install the two packages in cygwin:\n\n   mingw64-x86_64-gcc-core   mingw64-x86_64-gfortran\n\nand add the directory <cygwin-home>\\usr\\x86_64-w64-mingw32\\sys-root\\mingw\\bin to your path.\n\nFor more information, see http://github.com/mikiobraun/jblas/wiki/Missing-Libraries");
            }
        } catch (UnsupportedArchitectureException var5) {
            System.err.println(var5.getMessage());
        }

    }

    public static void loadDependentLibraries() {
        String arch = System.getProperty("os.arch");
        String name = System.getProperty("os.name");
        LibraryLoader loader = new LibraryLoader();
        if (name.startsWith("Windows") && arch.equals("amd64")) {
            loader.loadLibrary("libgcc_s_sjlj-1", false);
            loader.loadLibrary("libgfortran-3", false);
        } else if (name.startsWith("Windows") && arch.equals("x86")) {
            loader.loadLibrary("libgcc_s_dw2-1", false);
            loader.loadLibrary("libgfortran-3", false);
        } else if (name.equals("Linux") && arch.equals("amd64")) {
            loader.loadLibrary("quadmath-0", false);
            loader.loadLibrary("gfortran-4", false);
        }

    }
}

[lungsi]

Using ldd the library shared by the installed OpenBLAS were shown as

$ ldd /usr/local/lib/libopenblas.so
/usr/local/lib/libopenblas.so:
        libthr.so.3 => /lib/libthr.so.3 (0x80067d000)
        libgfortran.so.5 => /usr/local/lib/gcc9/libgfortran.so.5 (0x802e00000)
        libgomp.so.1 => /usr/local/lib/gcc9/libgomp.so.1 (0x803295000)
        libc.so.7 => /lib/libc.so.7 (0x80024a000)
        libquadmath.so.0 => /usr/local/lib/gcc9/libquadmath.so.0 (0x8034c9000)
        libm.so.5 => /lib/libm.so.5 (0x8006aa000)
        libgcc_s.so.1 => /usr/local/lib/gcc9/libgcc_s.so.1 (0x80370f000)
        libdl.so.1 => /usr/lib/libdl.so.1 (0x8006dc000)

In modifying the NativeBlasLibraryLoader.class with the additional conditions

        ...
        } else if (name.equals("FreeBSD") && arch.equals("amd64")) {
            loader.loadLibrary(libname, false);
            loader.loadLibrary(libname, false);
        }

I can't figure out what libname I should use as the arguments.

Should I use "libquadmath" and "libgfortran"?
That is,

        ...
        } else if (name.equals("FreeBSD") && arch.equals("amd64")) {
            loader.loadLibrary("libquadmath", false);
            loader.loadLibrary("libgfortran", false);
        }

PS: Prior to attempting the modification, after locating the shared libraries using the ldd command I have copied the libraries to /compat/linux and then created symbolic link as instructed in the FreeBSD Handbook.

[mikiobraun]

Hey there,

that seems odd indeed. Do you know what the corresponding freebsd docker base image is so that I can try to reproduce this, and can you print the whole error message? IIRC it also gives more hints to what is actually raising the issue.

[lungsi]

I am not using docker or any kind of virtualization (well, disregarding JVM).

My OS is FreeBSD and I am using Clojure to call jblas.

jblas as a dependency is defined in Clojures' project.clj - which is basically a .jar file pulled from maven locally into ~/.m2/repository/org/jblas/jblas/1.2.5. The definition is

  :dependencies [[org.clojure/clojure "1.10.0"]
                 [org.jblas/jblas "1.2.5"]]

[mikiobraun]

Hi lungsi,

sorry, I may not have explained it well enough. I was not implying you use docker, but for me to reproduce the problem it would be helpful to have a freebsd in a docker (the alternative would be to install freebsd in a virtualvm or something or install it on a free machine, which I don't have).

Can you paste in the full error message?

Just for context why I am asking: In the error messages I'm looking for more information on which symbol exactly was the culprit and which library. One thing that jblas does is to have the native libraries in the jar. I've tried to link any dependency statically, but unfortunately, there are a few that I could not (for example, because they weren't compiled in a way you can just add them), or because they are very basic. I am suspecting that probably one of the basic libraries that jblas still depends on are different on FreeBSD and would need to get a better understanding what exactly is happening there.

Hope this clarifies it a bit.

[mikiobraun]

Thanks, @brada4 for crosslinking. I see that you actually cannot run freebsd in a docker (now that I think about it, it makes sense tbh), so I guess one path forward for me would be to install freebsd in virtualvm. I don't have any freebsd experience, though, so not I will get somewhere quickly.

[brada4]

If there is a chance to build something off /usr/include and -lblas -llapack. The magic with prebuilt ATLAS does not seem to work on other systems.

[mikiobraun]

Prebuilt ATLAS is very OS specific, yes.

If you run configure with --dynamic-libs it should do a more normal build. You can also always edit the flags in configure.out and try more sane things when doing make all.

[brada4]

Been there already, Linux x86_64 - Unsupported flags dynamic_libs - thats why said it is not a trivial gig.

[mikiobraun]

Ah wait, that's a bug. There was a typo in the definition of supported flags. I just pushed a commit that fixes that to main. 🤦🏻‍♂️

[lungsi]

Thank you @mikiobraun and @brada4

Based on

In principle, all you need is the jblas-1.2.5.jar in your classpath. jblas-1.2.5.jar will then automagically extract your platform dependent native library to a tempfile and load it from there. You can also put that file somewhere in your load path ($LD_LIBRARY_PATH for Linux, %PATH for Windows).

In my system path

# echo $PATH
/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin

after adding the following paths of the installed OpenBLAS

# pkg info -lx openblas
openblas-0.3.10,1:
        /usr/local/include/cblas.h
        /usr/local/include/f77blas.h
        /usr/local/include/lapack.h
        /usr/local/include/lapacke.h
        /usr/local/include/lapacke_config.h
        /usr/local/include/lapacke_mangling.h
        /usr/local/include/lapacke_utils.h
        /usr/local/include/openblas_config.h
        /usr/local/lib/cmake/openblas/OpenBLASConfig.cmake
        /usr/local/lib/cmake/openblas/OpenBLASConfigVersion.cmake
        /usr/local/lib/libopenblas.a
        /usr/local/lib/libopenblas.so
        /usr/local/lib/libopenblas.so.0
        /usr/local/lib/libopenblasp-r0.3.10.a
        /usr/local/lib/libopenblasp-r0.3.10.so
        /usr/local/libdata/pkgconfig/openblas.pc
        /usr/local/share/licenses/openblas-0.3.10,1/BSD3CLAUSE
        /usr/local/share/licenses/openblas-0.3.10,1/LICENSE
        /usr/local/share/licenses/openblas-0.3.10,1/catalog.mk

such that,

# echo $PATH
/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin:/usr/local/include:/usr/local/lib

I am still getting the same error when the mmul operation is used for two matrices

Execution error (UnsatisfiedLinkError) at org.jblas.NativeBlas/dgemm (NativeBlas.java:-2).
org.jblas.NativeBlas.dgemm(CCIIID[DII[DIID[DII)V

[mikiobraun]

Thanks for sharing that. I would like to see more of the very last error message, not just this one line but ideally all the rest (of course remove everything you don't want to share).

Alternatively, can you run java -jar jblas-1.2.5.jar and show me all the output it sends?

[lungsi]

Actually this is all the output. My console does not show any other error message.

[mikiobraun]

Oh. I see... sorry for my insistence ;)

Yeah unfortunately I am not really familiar with freebsd. I pushed a fix that re-enables the --dynamic-libs feature which just "normally" links jblas.so against existing libraries. Maybe with that you or @brada4 have more luck getting at least that to compile.

[lungsi]

No worries @mikiobraun

Here's the output from running java -jar jblas-1.2.5.jar

$ java -jar jblas-1.2.5.jar
-- org.jblas INFO jblas version is 1.2.4
Simple benchmark for jblas

Running sanity benchmarks.

checking vector addition... ok
-- org.jblas CONFIG BLAS native library not found in path. Copying native library from the archive. Consider installing the library somewhere in the path (for Windows: PATH, for Linux: LD_LIBRARY_PATH).
-- org.jblas CONFIG ArchFlavor native library not found in path. Copying native library libjblas_arch_flavor from the archive. Consider installing the library somewhere in the path (for Windows: PATH, for Linux: LD_LIBRARY_PATH).
Exception in thread "main" java.lang.UnsatisfiedLinkError: org.jblas.NativeBlas.dgemm(CCIIID[DII[DIID[DII)V
        at org.jblas.NativeBlas.dgemm(Native Method)
        at org.jblas.SimpleBlas.gemm(SimpleBlas.java:247)
        at org.jblas.DoubleMatrix.mmuli(DoubleMatrix.java:1793)
        at org.jblas.DoubleMatrix.mmul(DoubleMatrix.java:3166)
        at org.jblas.util.SanityChecks.checkMatrixMultiplication(SanityChecks.java:91)
        at org.jblas.util.SanityChecks.main(SanityChecks.java:182)
        at org.jblas.benchmark.Main.main(Main.java:114)
-- org.jblas INFO Deleting /tmp/jblas6495989346745663149

[mikiobraun]

OK, thanks!

Aha, I've just been re-reading the code in NativeBlasLibraryLoader and my guess is that because os.arch is an unknown value, it just falls through, doesn't actually load anything and then it crashes when trying to load a library. I see that you already tried to add another branch there, which is good.

I think the easiest way forward would be to try to make --dynamic-libs work. Default is that it tries to link against openblas. You might want to specify --libpath= if it cannot find it. The Makefile should then be smart enough to copy the libjblas.so to the right place in src/main/resources.

[lungsi]

So I guess adding the following lines in the NativeBlasLibraryLoader.class file

        ...
        } else if (name.equals("FreeBSD") && arch.equals("amd64")) {
            loader.loadLibrary(libname, false);
            loader.loadLibrary(libname, false);
        }

and recompiling the jar will not solve the above "paths not found" error.

[mikiobraun]

So that path that is referred to is the library path, not the path with which the shell looks for executables. On Linux it is LD_LIBRARY_PATH, not sure what it is under FreeBSD. That message is okay, as long as the libjblas.so has been compiled and copied into the appropriate directory under src/main/resources/lib/static/... For Linux, for example, you have src/main/resources/lib/static/Linux/amd64/sse3/libjblas.so

The method loadDependentLibraries() is only if you need to load additional libraries which are not pre-installed.

The function where the main libjblas.so is loaded is in LibraryLoader.loadLibrary(). That automatically constructs paths looking at the operating system name, so no additional changes are necessary if you do the dyanmic build.

Sorry about all the mess, I think I really need to clean up the whole thing a bit 😅

[mikiobraun]

So yeah, thinking about it, my best recommendation would be to try to build it with ./configure --dynamic-libs (and additional parameters), make all and then it "should" work without additional changes.

Changes to dependent libraries etc. are just necessary if you want to really have everything in your jar file. But I think in your case you just want it to work, so you could postpone that.

[lungsi]

Yes you are exactly right. I just want it to work :-)

I will therefore go ahead with your suggestion to build jblas with respective parameters.

I guess this also means I would not need to define the jblas dependency [org.jblas/jblas "1.2.5"] in my Clojure project.clj

  :dependencies [[org.clojure/clojure "1.10.0"]
                 ;[org.jblas/jblas "1.2.5"] No longer needed to define here
                 ]

[mikiobraun]

Good luck!

Regarding clojure, you still need to make your own jar file available to the project. Not sure exactly how that works, dependencies would try to download it from the internet I guess.

Sorry, not sure how to do that with lein. Maybe do a "mvn install" in the jblas directory and then add a dependency to the version that that writes. Or you need to copy the jar file somewhere where your JVM can find it?

[lungsi]

Just to be sure I am following the steps correctly.

Can you confirm if these are?

# git clone https://github.com/jblas-project/jblas.git
# cd jblas
# ./configure --dynamic-libs
# make
# ant jblasFreeBSD-jar

[mikiobraun]

The high level steps are:

$ git clone https://github.com/jblas-project/jblas.git
$ cd jblas
$ ./configure --dynamic-libs  # this might require more tweaking, in particular add --libpath=... with the path to your openblas libraries
$ make all
$ mvn package

After that you have your own jar in the target directory. You need to figure out how to use that in your clojure project.

Hope this helps!

[lungsi]

Thanks so much @mikiobraun

[mikiobraun]

You're very welcome, and thanks for putting in so much work to make jblas work for you!