Skip to content

Regression in object detection #3840

Open
#3841
Open
Regression in object detection#3840
#3841
Labels
bugSomething isn't working
@alanocallaghan

Description

@alanocallaghan
alanocallaghan
opened on Mar 2, 2026

Description

A regression seems to have been introduced in at least the SSD model code meaning that detected objects are always tiny and located in the far upper left of the image. I suspect (based purely on inspecting the outputs) that this is due to double normalization of coordinates, or possible using normalized coordinates when un-normalized are expected.

The example is described here: https://github.com/deepjavalibrary/djl/blob/master/examples/docs/train_pikachu_ssd.md

I selected v0.23.0 as a "known good" version as this seemed to be roughly when the SSD example was added, or at least modified last.

Expected Behavior

The previous version of object detection, where objects were correctly detected at normal sizes and across the image.

Error Message

N/A

How to Reproduce?

Steps to reproduce

Expected behaviour

  1. Check out v0.23.0, disable nightly requirement in TrainPikachuTest::testDetection (unsure if this is required but it seemed to be to me)
  2. Run SSD example as follows:
cd examples
./gradlew clean
./gradlew test --tests "ai.djl.examples.training.TrainPikachuTest"
  1. Observe expected and correct output:
Image

Current behaviour

  1. Checkout master, disable nightly requirement in TrainPikachuTest::testDetection, also allow training on CPU if no GPU is available (if so, also remove expectedLoss check).
  2. Run SSD example with
cd examples
./gradlew clean
./gradlew test --tests "ai.djl.examples.training.TrainPikachuTest"
  1. Observe incorrect/deviant output:
Image

What have you tried to solve it?

  1. I tried on a local branch to change the SSD loss to accept coordinates in the form xmin,ymin,xmax,ymax as some losses accept, but this seems insufficient, eg:
@ api/src/main/java/ai/djl/training/loss/SingleShotDetectionLoss.java:54 @ public class SingleShotDetectionLoss extends AbstractCompositeLoss {
            int componentIndex, NDList labels, NDList predictions) {
        NDArray anchors = predictions.get(0);
        NDArray classPredictions = predictions.get(1);
        NDArray gt = labels.head();
        NDArray xmin = gt.get(":,:,1");
        NDArray ymin = gt.get(":,:,2");
        NDArray width = gt.get(":,:,3");
        NDArray height = gt.get(":,:,4");
        NDArray xmax = xmin.add(width);
        NDArray ymax = ymin.add(height);
        NDArray newGt = NDArrays.stack(
                new NDList(
                        gt.get(":,:,0"),   // class
                        xmin,
                        ymin,
                        xmax,
                        ymax
                ),
                -1
        );
        NDList targets =
                multiBoxTarget.target(
                        new NDList(anchors, labels.head(), classPredictions.transpose(0, 2, 1)));
                        new NDList(anchors, newGt, classPredictions.transpose(0, 2, 1)));

It's not clear to me the exact code path that's happening here and what format different methods expect.
2. I attempted to run the YOLO pikachu model with current master in ai.djl.examples.training.TrainPikachuWithYOLOV3 to see if the issue was related specifically to SSD, or object detection more broadly. However, I consistently run into divergence:

Exception in thread "main" ai.djl.TrainingDivergedException: The Loss became NaN, try reduce learning rate,add clipGradient option to your optimizer, check input data and loss calculation.
	at ai.djl.training.listener.DivergenceCheckTrainingListener.onTrainingBatch(DivergenceCheckTrainingListener.java:27)
	at ai.djl.training.EasyTrain.lambda$trainBatch$2(EasyTrain.java:115)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at ai.djl.training.Trainer.notifyListeners(Trainer.java:285)
	at ai.djl.training.EasyTrain.trainBatch(EasyTrain.java:115)
	at ai.djl.training.EasyTrain.fit(EasyTrain.java:58)
	at ai.djl.examples.training.TrainPikachuWithYOLOV3.runExample(TrainPikachuWithYOLOV3.java:65)
	at ai.djl.examples.training.TrainPikachuWithYOLOV3.main(TrainPikachuWithYOLOV3.java:43)
  1. I had a look at the diff between v0.23.0 and current master, but with 1.2k files changed I haven't managed to find anything relevant yet

Environment Info

v0.23.0

----------- System Properties -----------
sun.desktop: gnome
awt.toolkit: sun.awt.X11.XToolkit
java.specification.version: 11
sun.cpu.isalist: 
sun.jnu.encoding: UTF-8
java.class.path: /home/alan/Documents/github/imaging/djl/integration/build/classes/java/main:/home/alan/Documents/github/imaging/djl/integration/build/resources/main:/home/alan/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.5.0/dc98be5d5390230684a092589d70ea76a147925c/commons-cli-1.5.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.20.0/7ab4f082fd162f60afcaf2b8744a3d959feab3e8/log4j-slf4j-impl-2.20.0.jar:/home/alan/Documents/github/imaging/djl/basicdataset/build/libs/basicdataset-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/model-zoo/build/libs/model-zoo-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/testing/build/libs/testing-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/mxnet/mxnet-model-zoo/build/libs/mxnet-model-zoo-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/pytorch/pytorch-model-zoo/build/libs/pytorch-model-zoo-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/pytorch/pytorch-jni/build/libs/pytorch-jni-1.13.1-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/tensorflow/tensorflow-model-zoo/build/libs/tensorflow-model-zoo-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/ml/xgboost/build/libs/xgboost-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/ml/lightgbm/build/libs/lightgbm-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/onnxruntime/onnxruntime-engine/build/libs/onnxruntime-engine-0.23.0-SNAPSHOT.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.20.0/eb2a9a47b1396e00b5eee1264296729a70565cc0/log4j-core-2.20.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.20.0/1fe6082e660daf07c689a89c94dc0f49c26b44bb/log4j-api-2.20.0.jar:/home/alan/Documents/github/imaging/djl/engines/mxnet/mxnet-engine/build/libs/mxnet-engine-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/pytorch/pytorch-engine/build/libs/pytorch-engine-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/tensorflow/tensorflow-engine/build/libs/tensorflow-engine-0.23.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/api/build/libs/api-0.23.0-SNAPSHOT.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.testng/testng/7.8.0/90ff6902a350432ce23ef209b2f109bcf587069c/testng-7.8.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.36/6c62681a2f655b49963a5983b8b0950a6120ae14/slf4j-api-1.7.36.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-csv/1.10.0/8669bee353424c3223c93723291b5c3753260c1c/commons-csv-1.10.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/ml.dmlc/xgboost4j_2.12/1.7.5/5f074b329677d4e23222668597eb82de4b845a4b/xgboost4j_2.12-1.7.5.jar:/home/alan/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.2/4bfc12adfe4842bf07b657f0369c4cb522955686/commons-logging-1.2.jar:/home/alan/.gradle/caches/modules-2/files-2.1/com.microsoft.ml.lightgbm/lightgbmlib/3.2.110/f6c85e5d7cc44d49c4544240ea5c96004680007b/lightgbmlib-3.2.110.jar:/home/alan/.gradle/caches/modules-2/files-2.1/com.microsoft.onnxruntime/onnxruntime/1.15.0/6db39caba947384ce09c3071e84cb73437a77e74/onnxruntime-1.15.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/com.google.code.gson/gson/2.10.1/b3add478d4382b78ea20b1671390a858002feb6c/gson-2.10.1.jar:/home/alan/.gradle/caches/modules-2/files-2.1/net.java.dev.jna/jna/5.13.0/1200e7ebeedbe0d10062093f32925a912020e747/jna-5.13.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-compress/1.23.0/4af2060ea9b0c8b74f1854c6cafe4d43cfc161fc/commons-compress-1.23.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/com.beust/jcommander/1.82/a7c5fef184d238065de38f81bbc6ee50cca2e21/jcommander-1.82.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.webjars/jquery/3.6.1/d08df6250157cd2db3d9b01b11b76e9b7225083a/jquery-3.6.1.jar:/home/alan/Documents/github/imaging/djl/engines/tensorflow/tensorflow-api/build/libs/tensorflow-api-0.23.0-SNAPSHOT.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.tensorflow/tensorflow-core-api/0.5.0/6dfb7f13a9d96e6c4bd0705f122bd00d3b596b0d/tensorflow-core-api-0.5.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.bytedeco/javacpp/1.5.9/bee92b783ea619381df7577527f8739f778cf2f6/javacpp-1.5.9.jar:/home/alan/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protobuf-java/3.23.3/6ea96f2109fb6cf8f827aa58eebf784c4708d01f/protobuf-java-3.23.3.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.tensorflow/ndarray/0.4.0/7ab74f002dbec93944b7feb38de013afe8d4e8de/ndarray-0.4.0.jar
java.vm.vendor: Ubuntu
sun.arch.data.model: 64
user.variant: 
java.vendor.url: https://ubuntu.com/
user.timezone: Europe/London
os.name: Linux
java.vm.specification.version: 11
sun.java.launcher: SUN_STANDARD
user.country: GB
sun.boot.library.path: /usr/lib/jvm/java-11-openjdk-amd64/lib:/usr/lib/jvm/java-11-openjdk-amd64/lib
sun.java.command: ai.djl.integration.util.DebugEnvironment
jdk.debug: release
sun.cpu.endian: little
user.home: /home/alan
org.gradle.appname: gradlew
user.language: en
java.specification.vendor: Oracle Corporation
java.version.date: 2026-01-20
java.home: /usr/lib/jvm/java-11-openjdk-amd64
ai.djl.logging.level: debug
org.gradle.internal.http.connectionTimeout: 60000
file.separator: /
java.vm.compressedOopsMode: Zero based
line.separator: 

java.specification.name: Java Platform API Specification
java.vm.specification.vendor: Oracle Corporation
java.awt.graphicsenv: sun.awt.X11GraphicsEnvironment
sun.management.compiler: HotSpot 64-Bit Tiered Compilers
java.runtime.version: 11.0.30+7-post-Ubuntu-1ubuntu122.04
user.name: alan
path.separator: :
os.version: 6.17.9-76061709-generic
java.runtime.name: OpenJDK Runtime Environment
file.encoding: UTF-8
java.vm.name: OpenJDK 64-Bit Server VM
java.vendor.url.bug: https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
java.io.tmpdir: /tmp
org.gradle.internal.http.socketTimeout: 120000
java.version: 11.0.30
user.dir: /home/alan/Documents/github/imaging/djl/integration
os.arch: amd64
java.vm.specification.name: Java Virtual Machine Specification
java.awt.printerjob: sun.print.PSPrinterJob
sun.os.patch.level: unknown
java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
java.vm.info: mixed mode, sharing
java.vendor: Ubuntu
java.vm.version: 11.0.30+7-post-Ubuntu-1ubuntu122.04
java.specification.maintenance.version: 3
sun.io.unicode.encoding: UnicodeLittle
library.jansi.path: /home/alan/.gradle/native/jansi/1.18/linux64
java.class.version: 55.0
org.gradle.internal.publish.checksums.insecure: true

--------- Environment Variables ---------
PATH: /home/alan/.rbenv/shims:/home/alan/.local/bin:/home/alan/bin:/home/alan/.pyenv/shims:/home/alan/.pyenv/bin:/home/alan/miniconda3/bin:/usr/local/go/bin:/home/alan/.yarn/bin:/home/alan/.config/yarn/global/node_modules/.bin:/home/alan/.rbenv/shims:/home/alan/.pyenv/bin:/home/alan/miniconda3/bin:/usr/local/go/bin:/home/alan/.yarn/bin:/home/alan/.config/yarn/global/node_modules/.bin:/home/alan/micromamba/condabin:/home/alan/.nvm/versions/node/v25.2.1/bin:/home/alan/.rbenv/shims:/home/alan/.rbenv/bin:/home/alan/.juliaup/bin:/home/alan/.cargo/bin:/home/alan/.local/bin:/home/alan/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/usr/local/texlive/2025/bin/x86_64-linux:/home/alan/.local/bin:/home/alan/go/bin:/home/alan/Documents/github/shell_scripts/:/opt/nvim-linux-x86_64/bin:/usr/local/texlive/2025/bin/x86_64-linux:/home/alan/.local/bin:/home/alan/go/bin:/home/alan/Documents/github/shell_scripts/:/opt/nvim-linux-x86_64/bin:/home/alan/.local/share/JetBrains/Toolbox/scripts
PYENV_SHELL: bash
XAUTHORITY: /run/user/1000/gdm/Xauthority
XMODIFIERS: @im=ibus
XDG_DATA_DIRS: /usr/share/pop:/usr/share/gnome:/home/alan/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
GDMSESSION: pop
GTK_IM_MODULE: ibus
MAMBA_EXE: /home/alan/bin/micromamba
MAMBA_ROOT_PREFIX: /home/alan/micromamba
RBENV_SHELL: bash
DBUS_SESSION_BUS_ADDRESS: unix:path=/run/user/1000/bus
XDG_CURRENT_DESKTOP: pop:GNOME
COLORTERM: truecolor
SESSION_MANAGER: local/pop-os:@/tmp/.ICE-unix/4014,unix/pop-os:/tmp/.ICE-unix/4014
USERNAME: alan
LOGNAME: alan
PWD: /home/alan/Documents/github/imaging/djl
NVM_CD_FLAGS: 
SHELL: /bin/bash
LESSOPEN: | /usr/bin/lesspipe %s
RIG_PLATFORM: ubuntu-22.04
PYENV_ROOT: /home/alan/.pyenv
R_MAX_NUM_DLLS: 1000
GOPATH: /home/alan/go
OLDPWD: /home/alan/Documents/github/imaging/djl
GNOME_DESKTOP_SESSION_ID: this-is-deprecated
GNOME_TERMINAL_SCREEN: /org/gnome/Terminal/screen/26aaff53_d421_44a9_a2b8_434856d9f320
GTK_MODULES: gail:atk-bridge
SYSTEMD_EXEC_PID: 37585
LS_COLORS: fi=0;37:rs=0:di=01;94:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
XDG_SESSION_DESKTOP: pop
SSH_AGENT_LAUNCHER: gnome-keyring
SHLVL: 1
LESSCLOSE: /usr/bin/lesspipe %s %s
BROWSER: firefox
QT_IM_MODULE: ibus
TERM: xterm-256color
XDG_CONFIG_DIRS: /etc/xdg/xdg-pop:/etc/xdg
GNOME_TERMINAL_SERVICE: :1.129
LANG: en_GB.UTF-8
XDG_SESSION_TYPE: x11
DISPLAY: :1
NVM_INC: /home/alan/.nvm/versions/node/v25.2.1/include/node
CONDA_SHLVL: 0
XDG_SESSION_CLASS: user
_: ./gradlew
NVM_DIR: /home/alan/.nvm
GPG_AGENT_INFO: /run/user/1000/gnupg/S.gpg-agent:0:1
DESKTOP_SESSION: pop
SYNAPSE_PAT: eyJ0eXAiOiJKV1QiLCJraWQiOiJXN05OOldMSlQ6SjVSSzpMN1RMOlQ3TDc6M1ZYNjpKRU9VOjY0NFI6VTNJWDo1S1oyOjdaQ0s6RlBUSCIsImFsZyI6IlJTMjU2In0.eyJhY2Nlc3MiOnsic2NvcGUiOlsidmlldyIsImRvd25sb2FkIl0sIm9pZGNfY2xhaW1zIjp7fX0sInRva2VuX3R5cGUiOiJQRVJTT05BTF9BQ0NFU1NfVE9LRU4iLCJpc3MiOiJodHRwczovL3JlcG8tcHJvZC5wcm9kLnNhZ2ViYXNlLm9yZy9hdXRoL3YxIiwiYXVkIjoiMCIsIm5iZiI6MTc2OTY4MTg3OSwiaWF0IjoxNzY5NjgxODc5LCJqdGkiOiIzMTY4NCIsInN1YiI6IjM0OTQ3MTgifQ.aXXt3HtzEZI7dEqLwswhK-gnnwDCDOSWJ-_g84FgNhFuEAbL-o5sX0xjhXLnUhUc1J7FnfU2kJARtS0cIpUERAMZ2vR2Q1pzzKYInZF_6QP71rMl8UGH7aoPhDUm8KlpPTvLbSAe-SFzpi4BJquQvAF7R_2NNGVpgdhMwqFa-HgyChBSaFR50L-m974oj3I6ESUUQrZEK5NxDATvAUJl0iccc4vGnPlt3OlWbNmpKJdlr-PVDlOsf2MmHOTy-xJN7izy2d8xp4R2pI3Pi1B33--VLKdx17Z4iB51XLfuuFxEzo8P2BrNJdixY6AVmS7euu4TsRW6aYTq5sDEeWADAA
USER: alan
XDG_MENU_PREFIX: gnome-
VTE_VERSION: 6800
QT_ACCESSIBILITY: 1
WINDOWPATH: 2
SSH_AUTH_SOCK: /run/user/1000/keyring/ssh
EDITOR: nano
GNOME_SHELL_SESSION_MODE: pop
XDG_RUNTIME_DIR: /run/user/1000
NVM_BIN: /home/alan/.nvm/versions/node/v25.2.1/bin
HOME: /home/alan

-------------- Directories --------------
temp directory: /tmp
DJL cache directory: /home/alan/.djl.ai
Engine cache directory: /home/alan/.djl.ai

------------------ CUDA -----------------
GPU Count: 1
CUDA: 131
ARCH: 120
GPU(0) memory used: 1400373248 bytes

----------------- Engines ---------------
DJL version: 0.23.0-SNAPSHOT
[WARN ] - No matching cuda flavor for linux found: cu131mkl/sm_120.
[DEBUG] - Loading mxnet library from: /home/alan/.djl.ai/mxnet/1.9.1-mkl-linux-x86_64/libmxnet.so
[WARN ] - No matching cuda flavor for linux found: cu131mkl/sm_120.
Default Engine: MXNet:1.9.0, capabilities: [
        CPU_SSE,
        SIGNAL_HANDLER,
        LAPACK,
        BLAS_OPEN,
        CPU_SSE2,
        DIST_KVSTORE,
        CPU_SSE3,
        OPENMP,
        OPENCV,
        MKLDNN,
]
MXNet Library: /home/alan/.djl.ai/mxnet/1.9.1-mkl-linux-x86_64/libmxnet.so
Default Device: cpu()
PyTorch: 2
MXNet: 0
XGBoost: 10
LightGBM: 10
OnnxRuntime: 10
TensorFlow: 3

--------------- Hardware --------------
Available processors (cores): 24
Byte Order: LITTLE_ENDIAN
Free memory (bytes): 1027336304
Maximum memory (bytes): 16617832448
Total memory available to JVM (bytes): 1048576000
Heap committed: 1048576000
Heap nonCommitted: 33423360
GCC: 
gcc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Latest master:

----------- System Properties -----------
java.specification.version: 25
sun.jnu.encoding: UTF-8
java.class.path: /home/alan/Documents/github/imaging/djl/integration/build/classes/java/main:/home/alan/Documents/github/imaging/djl/integration/build/resources/main:/home/alan/.gradle/caches/modules-2/files-2.1/commons-cli/commons-cli/1.9.0/e1cdfa8bf40ccbb7440b2d1232f9f45bb20a1844/commons-cli-1.9.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j2-impl/2.24.3/c3bda2dc612bfc780ad06c5acf5e0ffd55b770e5/log4j-slf4j2-impl-2.24.3.jar:/home/alan/Documents/github/imaging/djl/basicdataset/build/libs/basicdataset-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/model-zoo/build/libs/model-zoo-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/testing/build/libs/testing-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/mxnet/mxnet-model-zoo/build/libs/mxnet-model-zoo-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/pytorch/pytorch-model-zoo/build/libs/pytorch-model-zoo-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/pytorch/pytorch-jni/build/libs/pytorch-jni-2.7.1-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/tensorflow/tensorflow-model-zoo/build/libs/tensorflow-model-zoo-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/ml/xgboost/build/libs/xgboost-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/ml/lightgbm/build/libs/lightgbm-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/onnxruntime/onnxruntime-engine/build/libs/onnxruntime-engine-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/extensions/tokenizers/build/libs/tokenizers-0.37.0-SNAPSHOT.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.24.3/7f6a261243ca767c7f38fd4b542bcde626c8894e/log4j-core-2.24.3.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.24.3/b02c125db8b6d295adf72ae6e71af5d83bce2370/log4j-api-2.24.3.jar:/home/alan/Documents/github/imaging/djl/engines/mxnet/mxnet-engine/build/libs/mxnet-engine-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/pytorch/pytorch-engine/build/libs/pytorch-engine-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/engines/tensorflow/tensorflow-engine/build/libs/tensorflow-engine-0.37.0-SNAPSHOT.jar:/home/alan/Documents/github/imaging/djl/api/build/libs/api-0.37.0-SNAPSHOT.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.testng/testng/7.11.0/c75ee639e6fc9c0a406d32608d532d9ee9341941/testng-7.11.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/2.0.17/d9e58ac9c7779ba3bf8142aff6c830617a7fe60f/slf4j-api-2.0.17.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-csv/1.14.1/467354980fade8528b6a9f9bdf7f4f19ab9b3373/commons-csv-1.14.1.jar:/home/alan/.gradle/caches/modules-2/files-2.1/ml.dmlc/xgboost4j_2.12/3.0.4/99ecd710f723db420736258c6d280791e6af72c/xgboost4j_2.12-3.0.4.jar:/home/alan/.gradle/caches/modules-2/files-2.1/commons-logging/commons-logging/1.3.5/a3fcc5d3c29b2b03433aa2d2f2d2c1b1638924a1/commons-logging-1.3.5.jar:/home/alan/.gradle/caches/modules-2/files-2.1/com.microsoft.ml.lightgbm/lightgbmlib/3.2.110/f6c85e5d7cc44d49c4544240ea5c96004680007b/lightgbmlib-3.2.110.jar:/home/alan/.gradle/caches/modules-2/files-2.1/com.microsoft.onnxruntime/onnxruntime/1.21.1/8c83d6e2120bf4add64082ebe8a2d773619efd40/onnxruntime-1.21.1.jar:/home/alan/.gradle/caches/modules-2/files-2.1/com.google.code.gson/gson/2.13.1/853ce06c11316b33a8eae5e9095da096a9528b8f/gson-2.13.1.jar:/home/alan/.gradle/caches/modules-2/files-2.1/net.java.dev.jna/jna/5.17.0/33d12735bef894440780fce64f9758d420c7bae2/jna-5.17.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.apache.commons/commons-compress/1.27.1/a19151084758e2fbb6b41eddaa88e7b8ff4e6599/commons-compress-1.27.1.jar:/home/alan/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.20.0/36f3474daec2849c149e877614e7f979b2082cd2/commons-io-2.20.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/commons-codec/commons-codec/1.19.0/8c0dbe3ae883fceda9b50a6c76e745e548073388/commons-codec-1.19.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.jcommander/jcommander/1.83/c342a2ad17ec08db105146e27c7ba9c535c9bb46/jcommander-1.83.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.webjars/jquery/3.7.1/42088e652462c40a369b64d87e18e825644acfab/jquery-3.7.1.jar:/home/alan/Documents/github/imaging/djl/engines/tensorflow/tensorflow-api/build/libs/tensorflow-api-0.37.0-SNAPSHOT.jar:/home/alan/.gradle/caches/modules-2/files-2.1/com.google.errorprone/error_prone_annotations/2.38.0/fc0ae991433e8590ba51cd558421478318a74c8c/error_prone_annotations-2.38.0.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.tensorflow/tensorflow-core-api/1.0.0-rc.1/ea1878fb8e289742237e5a0ba6f15398f3e9b7ef/tensorflow-core-api-1.0.0-rc.1.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.tensorflow/tensorflow-core-native/1.0.0-rc.1/62b5fa3283865cc696dfbebf073ca2116b18f327/tensorflow-core-native-1.0.0-rc.1.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.bytedeco/javacpp/1.5.11/76a0197853ab5e2d3643804e62f9018053750ddf/javacpp-1.5.11.jar:/home/alan/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protobuf-java/4.31.1/1828b20315b63d5f71b3c61b094494a8f1acdc5a/protobuf-java-4.31.1.jar:/home/alan/.gradle/caches/modules-2/files-2.1/org.tensorflow/ndarray/1.0.0-rc.1/4a96a398ad87bec32be9177b1441b9880c04d822/ndarray-1.0.0-rc.1.jar
java.vm.vendor: Ubuntu
sun.arch.data.model: 64
user.variant: 
java.vendor.url: https://ubuntu.com/
user.timezone: Europe/London
java.vm.specification.version: 25
os.name: Linux
user.country: GB
sun.java.launcher: SUN_STANDARD
sun.boot.library.path: /usr/lib/jvm/java-25-openjdk-amd64/lib
sun.java.command: ai.djl.integration.util.DebugEnvironment
jdk.debug: release
sun.cpu.endian: little
user.home: /home/alan
user.language: en
java.specification.vendor: Oracle Corporation
java.version.date: 2026-01-20
java.home: /usr/lib/jvm/java-25-openjdk-amd64
file.separator: /
java.vm.compressedOopsMode: Zero based
line.separator: 

java.vm.specification.vendor: Oracle Corporation
java.specification.name: Java Platform API Specification
sun.management.compiler: HotSpot 64-Bit Tiered Compilers
java.runtime.version: 25.0.2+10-Ubuntu-122.04
user.name: alan
stdout.encoding: UTF-8
path.separator: :
os.version: 6.17.9-76061709-generic
java.runtime.name: OpenJDK Runtime Environment
file.encoding: UTF-8
java.vm.name: OpenJDK 64-Bit Server VM
java.vendor.url.bug: https://bugs.launchpad.net/ubuntu/+source/openjdk-25
java.io.tmpdir: /tmp
java.version: 25.0.2
user.dir: /home/alan/Documents/github/imaging/djl/integration
os.arch: amd64
java.vm.specification.name: Java Virtual Machine Specification
native.encoding: UTF-8
java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
java.vm.info: mixed mode, sharing
stderr.encoding: UTF-8
java.vendor: Ubuntu
java.vm.version: 25.0.2+10-Ubuntu-122.04
stdin.encoding: UTF-8
sun.io.unicode.encoding: UnicodeLittle
java.class.version: 69.0

--------- Environment Variables ---------
R_MAX_NUM_DLLS: 1000
WINDOWPATH: 2
GNOME_SHELL_SESSION_MODE: pop
PATH: /home/alan/.rbenv/shims:/home/alan/.local/bin:/home/alan/bin:/home/alan/.pyenv/shims:/home/alan/.pyenv/bin:/home/alan/miniconda3/bin:/usr/local/go/bin:/home/alan/.yarn/bin:/home/alan/.config/yarn/global/node_modules/.bin:/home/alan/.rbenv/shims:/home/alan/.pyenv/bin:/home/alan/miniconda3/bin:/usr/local/go/bin:/home/alan/.yarn/bin:/home/alan/.config/yarn/global/node_modules/.bin:/home/alan/micromamba/condabin:/home/alan/.nvm/versions/node/v25.2.1/bin:/home/alan/.rbenv/shims:/home/alan/.rbenv/bin:/home/alan/.juliaup/bin:/home/alan/.cargo/bin:/home/alan/.local/bin:/home/alan/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin:/usr/local/texlive/2025/bin/x86_64-linux:/home/alan/.local/bin:/home/alan/go/bin:/home/alan/Documents/github/shell_scripts/:/opt/nvim-linux-x86_64/bin:/usr/local/texlive/2025/bin/x86_64-linux:/home/alan/.local/bin:/home/alan/go/bin:/home/alan/Documents/github/shell_scripts/:/opt/nvim-linux-x86_64/bin:/home/alan/.local/share/JetBrains/Toolbox/scripts
XDG_MENU_PREFIX: gnome-
LESSCLOSE: /usr/bin/lesspipe %s %s
LOGNAME: alan
LS_COLORS: fi=0;37:rs=0:di=01;94:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
XDG_CONFIG_DIRS: /etc/xdg/xdg-pop:/etc/xdg
NVM_CD_FLAGS: 
MAMBA_EXE: /home/alan/bin/micromamba
NVM_DIR: /home/alan/.nvm
XAUTHORITY: /run/user/1000/gdm/Xauthority
RIG_PLATFORM: ubuntu-22.04
CONDA_SHLVL: 0
EDITOR: nano
XMODIFIERS: @im=ibus
MAMBA_ROOT_PREFIX: /home/alan/micromamba
QT_ACCESSIBILITY: 1
XDG_SESSION_DESKTOP: pop
NVM_BIN: /home/alan/.nvm/versions/node/v25.2.1/bin
DBUS_SESSION_BUS_ADDRESS: unix:path=/run/user/1000/bus
GOPATH: /home/alan/go
SHLVL: 1
SSH_AGENT_LAUNCHER: gnome-keyring
USERNAME: alan
PYENV_ROOT: /home/alan/.pyenv
SHELL: /bin/bash
XDG_DATA_DIRS: /usr/share/pop:/usr/share/gnome:/home/alan/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
GNOME_TERMINAL_SCREEN: /org/gnome/Terminal/screen/26aaff53_d421_44a9_a2b8_434856d9f320
GTK_IM_MODULE: ibus
XDG_SESSION_CLASS: user
BROWSER: firefox
GTK_MODULES: gail:atk-bridge
SESSION_MANAGER: local/pop-os:@/tmp/.ICE-unix/4014,unix/pop-os:/tmp/.ICE-unix/4014
COLORTERM: truecolor
HOME: /home/alan
DISPLAY: :1
NVM_INC: /home/alan/.nvm/versions/node/v25.2.1/include/node
XDG_CURRENT_DESKTOP: pop:GNOME
TERM: xterm-256color
RBENV_SHELL: bash
QT_IM_MODULE: ibus
GNOME_TERMINAL_SERVICE: :1.129
LANG: en_GB.UTF-8
LESSOPEN: | /usr/bin/lesspipe %s
GDMSESSION: pop
GNOME_DESKTOP_SESSION_ID: this-is-deprecated
SYSTEMD_EXEC_PID: 37585
XDG_RUNTIME_DIR: /run/user/1000
SSH_AUTH_SOCK: /run/user/1000/keyring/ssh
OLDPWD: /home/alan/Documents/github/imaging/djl
PYENV_SHELL: bash
GPG_AGENT_INFO: /run/user/1000/gnupg/S.gpg-agent:0:1
DESKTOP_SESSION: pop
USER: alan
XDG_SESSION_TYPE: x11
PWD: /home/alan/Documents/github/imaging/djl
VTE_VERSION: 6800
_: ./gradlew
SYNAPSE_PAT: eyJ0eXAiOiJKV1QiLCJraWQiOiJXN05OOldMSlQ6SjVSSzpMN1RMOlQ3TDc6M1ZYNjpKRU9VOjY0NFI6VTNJWDo1S1oyOjdaQ0s6RlBUSCIsImFsZyI6IlJTMjU2In0.eyJhY2Nlc3MiOnsic2NvcGUiOlsidmlldyIsImRvd25sb2FkIl0sIm9pZGNfY2xhaW1zIjp7fX0sInRva2VuX3R5cGUiOiJQRVJTT05BTF9BQ0NFU1NfVE9LRU4iLCJpc3MiOiJodHRwczovL3JlcG8tcHJvZC5wcm9kLnNhZ2ViYXNlLm9yZy9hdXRoL3YxIiwiYXVkIjoiMCIsIm5iZiI6MTc2OTY4MTg3OSwiaWF0IjoxNzY5NjgxODc5LCJqdGkiOiIzMTY4NCIsInN1YiI6IjM0OTQ3MTgifQ.aXXt3HtzEZI7dEqLwswhK-gnnwDCDOSWJ-_g84FgNhFuEAbL-o5sX0xjhXLnUhUc1J7FnfU2kJARtS0cIpUERAMZ2vR2Q1pzzKYInZF_6QP71rMl8UGH7aoPhDUm8KlpPTvLbSAe-SFzpi4BJquQvAF7R_2NNGVpgdhMwqFa-HgyChBSaFR50L-m974oj3I6ESUUQrZEK5NxDATvAUJl0iccc4vGnPlt3OlWbNmpKJdlr-PVDlOsf2MmHOTy-xJN7izy2d8xp4R2pI3Pi1B33--VLKdx17Z4iB51XLfuuFxEzo8P2BrNJdixY6AVmS7euu4TsRW6aYTq5sDEeWADAA

-------------- Directories --------------
temp directory: /tmp
DJL cache directory: /home/alan/.djl.ai
Engine cache directory: /home/alan/.djl.ai

------------------ CUDA -----------------
WARNING: A restricted method in java.lang.System has been called
WARNING: java.lang.System::load has been called by com.sun.jna.Native in an unnamed module (file:/home/alan/.gradle/caches/modules-2/files-2.1/net.java.dev.jna/jna/5.17.0/33d12735bef894440780fce64f9758d420c7bae2/jna-5.17.0.jar)
WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this module
WARNING: Restricted methods will be blocked in a future release unless native access is enabled

GPU Count: 1
CUDA: 131
ARCH: 120
GPU(0) memory used: 1393819648 bytes

----------------- Engines ---------------
DJL version: 0.37.0-SNAPSHOT
[WARN ] - No matching cuda flavor for linux found: cu131mkl/sm_120.
[WARN ] - No matching cuda flavor for linux found: cu131mkl/sm_120.
Default Engine: MXNet:1.9.0, capabilities: [
        CPU_SSE,
        SIGNAL_HANDLER,
        LAPACK,
        BLAS_OPEN,
        CPU_SSE2,
        DIST_KVSTORE,
        CPU_SSE3,
        OPENMP,
        OPENCV,
        MKLDNN,
]
MXNet Library: /home/alan/.djl.ai/mxnet/1.9.1-mkl-linux-x86_64/libmxnet.so
Default Device: cpu()
Rust: 4
PyTorch: 2
MXNet: 0
RPC: 15
XGBoost: 10
LightGBM: 10
OnnxRuntime: 10
TensorFlow: 3

--------------- Hardware --------------
Available processors (cores): 24
Byte Order: LITTLE_ENDIAN
Free memory (bytes): 1030941936
Maximum memory (bytes): 16617832448
Total memory available to JVM (bytes): 1048576000
Heap committed: 1048576000
Heap nonCommitted: 30081024
GCC: 
gcc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions