export time - what to do

[nightflasher]

Describe the bug

exporting a handful of images (really not much), it takes ages. i recently exported 50 sony a7 mk2 images and it took about 30+ minutes.

Steps to reproduce

raw chromatic aberrations
lens correction
chromatic aberration
shadows and highlights
demosaic
base curve 'sony alpha like"

this is my preset called "personal defaults", maybe two chromatic aberrations is one no longer needed or obsolete

Expected behavior

export was way faster on my older notebook

Logfile | Screenshot | Screencast

every image export looks exactly like this
640.0299 [dt_dev_load_raw] loading the image. took 0.188 secs (0.677 CPU) 640.1728 [export] creating pixelpipe took 0.125 secs (0.156 CPU) 640.1742 [dev_pixelpipe] took 0.000 secs (0.000 CPU) initing base buffer [export] 640.2422 [dev_pixelpipe] took 0.068 secs (0.010 CPU) [export] processedrawprepare' on GPU, blended on GPU
640.2835 [dev_pixelpipe] took 0.041 secs (0.010 CPU) [export] processed temperature' on GPU, blended on GPU 640.3645 [dev_pixelpipe] took 0.081 secs (0.024 CPU) [export] processed highlights' on GPU, blended on GPU
640.9004 [dev_pixelpipe] took 0.536 secs (2.119 CPU) [export] processed cacorrect' on CPU, blended on CPU 642.4044 [dev_pixelpipe] took 1.504 secs (11.006 CPU) [export] processed demosaic' on CPU, blended on CPU
643.7066 [dev_pixelpipe] took 1.302 secs (2.275 CPU) [export] processed lens' on GPU, blended on GPU 646.4515 [dev_pixelpipe] took 2.745 secs (14.207 CPU) [export] processed cacorrectrgb' on CPU, blended on CPU
646.8414 [dev_pixelpipe] took 0.390 secs (0.031 CPU) [export] processed exposure' on GPU, blended on GPU 647.0055 [dev_pixelpipe] took 0.164 secs (0.052 CPU) [export] processed colorin' on GPU, blended on GPU
647.1679 [dt_ioppr_transform_image_colorspace_cl] IOP_CS_LAB-->IOP_CS_RGB took 0.162 secs (0.002 GPU) [basecurve]
647.4216 [dev_pixelpipe] took 0.416 secs (0.082 CPU) [export] processed basecurve' on GPU, blended on GPU 647.5873 [dt_ioppr_transform_image_colorspace_cl] IOP_CS_RGB-->IOP_CS_LAB took 0.161 secs (0.000 GPU) [shadhi] 660.5432 [dev_pixelpipe] took 13.122 secs (0.125 CPU) [export] processed shadhi' on GPU, blended on GPU
660.7301 [dev_pixelpipe] took 0.187 secs (0.064 CPU) [export] processed colorout' on GPU, blended on GPU 661.0032 [dev_pixelpipe] took 0.273 secs (1.197 CPU) [export] processed gamma' on CPU, blended on CPU
661.0032 [opencl_profiling] profiling device 0 ('Intel(R) OpenCL Graphics Intel(R) UHD Graphics 620'):
661.0032 [opencl_profiling] spent 0.2028 seconds in [Write Image (from host to device)]
661.0032 [opencl_profiling] spent 0.0193 seconds in rawprepare_1f
661.0032 [opencl_profiling] spent 0.0235 seconds in whitebalance_1f
661.0032 [opencl_profiling] spent 0.0137 seconds in highlights_initmask
661.0033 [opencl_profiling] spent 0.0088 seconds in highlights_dilatemask
661.0033 [opencl_profiling] spent 0.1988 seconds in [Write Buffer (from host to device)]
661.0033 [opencl_profiling] spent 0.0062 seconds in highlights_chroma
661.0033 [opencl_profiling] spent 0.0000 seconds in [Read Buffer (from device to host)]
661.0033 [opencl_profiling] spent 0.0232 seconds in highlights_opposed
661.0033 [opencl_profiling] spent 0.2043 seconds in [Read Image (from device to host)]
661.0033 [opencl_profiling] spent 0.1695 seconds in lens_vignette
661.0033 [opencl_profiling] spent 0.2416 seconds in lens_distort_bicubic
661.0033 [opencl_profiling] spent 0.0919 seconds in exposure
661.0033 [opencl_profiling] spent 0.0909 seconds in colorin_unbound
661.0033 [opencl_profiling] spent 0.2689 seconds in [Copy Image (on device)]
661.0033 [opencl_profiling] spent 0.0896 seconds in colorspaces_transform_lab_to_rgb_matrix
661.0033 [opencl_profiling] spent 0.0922 seconds in basecurve_lut
661.0033 [opencl_profiling] spent 0.0893 seconds in colorspaces_transform_rgb_matrix_to_lab
661.0033 [opencl_profiling] spent 0.0000 seconds in zero
661.0033 [opencl_profiling] spent 12.3956 seconds in splat
661.0033 [opencl_profiling] spent 0.0000 seconds in [Copy Buffer to Buffer (on device)]
661.0033 [opencl_profiling] spent 0.0001 seconds in blur_line
661.0033 [opencl_profiling] spent 0.0000 seconds in blur_line_z
661.0034 [opencl_profiling] spent 0.0888 seconds in slice
661.0034 [opencl_profiling] spent 0.1231 seconds in shadows_highlights_mix
661.0034 [opencl_profiling] spent 0.1155 seconds in colorout
661.0034 [opencl_profiling] spent 14.5577 seconds totally in command queue (with 0 events missing)
661.0034 [dev_process_export] pixel pipeline processing took 20.831 secs (31.206 CPU)
662.3904 [export_job] exported to /home/kevin/...

Commit

No response

Where did you obtain darktable from?

downloaded from www.darktable.org

darktable version

5.0.0

What OS are you using?

Linux

What is the version of your OS?

ubuntu 24.04.1

Describe your system?

OS: Ubuntu 24.04.1 LTS x86_64
Host: 81DE Lenovo ideapad 330-15IKB
Kernel: 6.8.0-51-generic
Shell: bash 5.2.21
Resolution: 1920x1080
CPU: Intel i7-8550U (8) @ 4.000GHz
GPU: Intel UHD Graphics 620
GPU: AMD ATI Radeon R7 M260/M265 / M340/M360 / M440/M445 / 530/535 / 620/625 Mobile
Memory: 4169MiB / 19908MiB
GPU Driver: i915
CPU Usage: 47%

^^from neofetch

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

using a script to start darktable with some parameters, i think is needed to use OpenCL and RUSTICL

export RUSTICL_ENABLE=radeonsi
vblank_mode=0 RUSTICL_ENABLE=rusticl DRI_PRIME=1 LIBVA_DRIVER_NAME=radeonsi darktable

[paperdigits]

Your GPU driver being i915 suggests you're using the intel ingetrated GPU. You haven't provided enough log to say what GPU is bening used, and your log message also indicates activity on the GPU. Have you tried disabling OpenCL to see if that improves anything?

[nightflasher]

with this line, i start the application with the AMD gpu driver
export RUSTICL_ENABLE=radeonsi vblank_mode=0 RUSTICL_ENABLE=rusticl DRI_PRIME=1 LIBVA_DRIVER_NAME=radeonsi darktable

exporting 51 images right now, it's much faster after disabling every other OpenCL option than the RUSTICL. But i think it's still slow.

`darktable 5.0.0
Copyright (C) 2012-2024 Johannes Hanika and other contributors.

Compile options:
Bit depth -> 64 bit
Debug -> DISABLED
SSE2 optimizations -> ENABLED
OpenMP -> ENABLED
OpenCL -> ENABLED
Lua -> ENABLED - API version 9.4.0
Colord -> ENABLED
gPhoto2 -> ENABLED
GMIC -> ENABLED - Compressed LUTs are supported
GraphicsMagick -> ENABLED
ImageMagick -> DISABLED
libavif -> ENABLED
libheif -> ENABLED
libjxl -> ENABLED
LibRaw -> ENABLED - Version 0.22.0-Devel202403
OpenJPEG -> ENABLED
OpenEXR -> ENABLED
WebP -> ENABLED

See https://www.darktable.org/resources/ for detailed documentation.
See https://github.com/darktable-org/darktable/issues/new/choose to report bugs.

 0.0002 [dt starting] darktable --library /home/kevin/.config/darktable/library.db.4.8.1 -d opencl -d perf
 0.4906 [dt_get_sysresource_level] switched to 1 as `default'
 0.4906   total mem:       19908MB
 0.4906   mipmap cache:    2488MB
 0.4906   available mem:   9954MB
 0.4906   singlebuff:      155MB
 0.5206 [opencl_init] opencl disabled via darktable preferences
 0.5212 [opencl_init] opencl library 'libOpenCL' found on your system and loaded, preference 'default path'
 1.2328 [opencl_init] found 3 platforms
 1.2328 [check platform] platform 'Intel(R) OpenCL Graphics' with key 'clplatform_intelropenclgraphics' is NOT active
 1.2328 [check platform] platform 'Clover' with key 'clplatform_clover' is NOT active
 1.2328 [opencl_init] no devices found for Mesa/X.org (vendor) - rusticl (name) [opencl_init] found 0 device
 1.2328 [opencl_init] FINALLY: opencl PREFERENCE=OFF is NOT AVAILABLE and NOT ENABLED.`

[nightflasher]

kevin@ideapad330:~$ vblank_mode=0 RUSTICL_ENABLE=rusticl DRI_PRIME=1 LIBVA_DRIVER_NAME=radeonsi glxinfo | grep OpenGL
kevin@ideapad330:~$ vblank_mode=0 RUSTICL_ENABLE=rusticl DRI_PRIME=1 LIBVA_DRIVER_NAME=radeonsi glxinfo | grep OpenGL ATTENTION: default value of option vblank_mode overridden by environment. OpenGL vendor string: AMD OpenGL renderer string: AMD Radeon R5 M445 Series (radeonsi, iceland, ACO, DRM 3.57, 6.8.0-51-generic) OpenGL core profile version string: 4.6 (Core Profile) Mesa 25.0~git2501030600.9033e6~oibaf~n (git-9033e64 2025-01-03 noble-oibaf-ppa) OpenGL core profile shading language version string: 4.60 OpenGL core profile context flags: (none) OpenGL core profile profile mask: core profile OpenGL core profile extensions: OpenGL version string: 4.6 (Compatibility Profile) Mesa 25.0~git2501030600.9033e6~oibaf~n (git-9033e64 2025-01-03 noble-oi OpenGL shading language version string: 4.60 OpenGL context flags: (none) OpenGL profile mask: compatibility profile OpenGL extensions: OpenGL ES profile version string: OpenGL ES 3.2 Mesa 25.0~git2501030600.9033e6~oibaf~n (git-9033e64 2025-01-03 noble-oibaf-ppa) OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20 OpenGL ES profile extensions:

[scorpi11]

1.2328 [opencl_init] no devices found for Mesa/X.org (vendor) - rusticl (name) [opencl_init] found 0 device
1.2328 [opencl_init] FINALLY: opencl PREFERENCE=OFF is NOT AVAILABLE and NOT ENABLED.`

This doesn't look good if you want to use rusticl.

The 'RUSTICL_ENABLE=rusticl' in your command line is bogus, it will overwrite the correct setting at the beginning of the line. Just remove it and try again.

To verify your settings, use "clinfo" at the end of your above command line instead of "darktable". You should see informational output about the OpenCL devices in your setup.

Besides that, when I tried rusticl 11 months ago, it was not that fast on my RX 5600XT. Which is why I'm using ROCm for OpenCL.

Quoting my post on pixls.us:

My setup: AMD RX 5600XT, Mesa 24.0.1, ROCm 6.0.2, darktable 4.6.1

I exported 9 Images. It took 25 seconds with RustiCL and 6 seconds with ROCm.

[nightflasher]

so now my starting line is

vblank_mode=0 DRI_PRIME=1 LIBVA_DRIVER_NAME=radeonsi darktable --library ~/.config/darktable/library.db.4.8.1 -d opencl -d perf

and i got faster export times, but still ~9 sec. for one image.
but now, after closing darktable, it's creating a'segment fault' and creates error log
darktable_bt_5OHC02.txt