Image before export is too red, but after it is too green.

[Maciejka1]

Describe the bug

To be clear my color profiles are configured to sRGB as export and display profile and I tried every solution from the web also I tried 2 different displays, xorg, xwayland, wayland, and different desktop environments.
Every image in darktable is too red. When I enable high quality processing issue dissapears.
Left is image before export and right is after export. This is a huge issue for me since I cannot afford to wait 5 seconds for a image to render every time I change something.
Exported JPEG opened in DT has still different colors than not exported.
Colors don't seem to fix even when every module is off.
Change in color is most noticeable when image is vastly brightened but also can be seen on not brighened images.

Steps to reproduce

1. Open RAW file
2. Export it.
3. Compare exported image which is too green with not exported image opened in DT.

Expected behavior

DT should export image in normal colors.

Logfile | Screenshot | Screencast

Commit

No response

Where did you obtain darktable from?

distro packaging

darktable version

2:4.8.1-2

What OS are you using?

Linux

What is the version of your OS?

Arch Linux x86_64

Describe your system?

My setup:
Kernel: 6.11.5-zen1-1-zen
DE: GNOME 47.1
WM: Mutter
CPU: i5-10400f
GPU: AMD radeon RX 6600 XT
Display: LG ultragear 27GP850P-B 27"
Camera: Sony a6000

Are you using OpenCL GPU in darktable?

Yes

If yes, what is the GPU card and driver?

6600XT amdvlk 2024.Q3.3-1 opencl-amd 1:6.2.1-1

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

[Maciejka1]

Every module that is being used.

[jenshannoschwalm]

You describe something I haven't come across and not reported before. To check we certainly would need a raw file plus the xmp plus setting you are using for exporting.

[Maciejka1]

Yes of course. Could you provide me a list of things that I should send here? As soon as I get home I will get back to you.

[jenshannoschwalm]

As mentioned, raw-file, xmp and export settings :-)

[Maciejka1]

Hey, here is the link to google drive:
https://drive.google.com/drive/folders/1-fTGJzngMPfkGxdCwe9oA8hDAXD1hJ7e?usp=sharing
This includes one raw with xmp, darktable screenshot and exported image that is shown here also one other raw from a6000 but as said before this affects every image. I have included a raw from canon powershot s5 is with mod that allowed saving raws and it is not affected by this issue. One major difference I have noticed is that canon raw in input profile has "embedded color matrix", a6000 does not and has "standard color matrix".

[Maciejka1]

Oh and also here is image when I turn on high quality processing. Before and after.

[jenshannoschwalm]

Aah, very interesting. I'll explain later. Can you confirm this does not happen with low iso images from the same camera? If you have issues then too, provide an other raw please.

[Maciejka1]

As mentioned above it happens with every image. At iso 100 it is not as drastic but still happens. I will provide you a raw as soon as I get home today.

[jenshannoschwalm]

The interesting point for me is, you can reproduce the issue - different output after HQ pipe is toggled - even with only these modules enabled

The chosen demosaicer has some influence but it's even there with the passthru modes.

From my understanding this is another example where the downscaling interpolator is behaving bad because

1. of the high noise on the raw signal
2. there is no filter before downscaling.

BTW i am really surprised about the bad signal/noise ratio in these images. Could it be the camera has a hardware problem or became very hot?

@kofa73 have you seen things like this on your LX7 with high ISO ?

[kofa73]

I don't shoot it at high ISO, because it's very noisy. But I'll test.
Could this be related to what @hanatos posted recently?

[in vkdt] for instance denoise runs before black point subtraction/clipping at 0 so gaussian noise can average to black (not purple)
https://discuss.pixls.us/t/16-inch-laptop-for-photo-editing/45832/138

[kofa73]

Cannot reproduce with the LX7. Minimal stack.

HQ:

LQ:

[jenshannoschwalm]

Would You share that high iso raw?

[kofa73]

https://tech.kovacs-telekes.org/files/2024-11-03-darktable-issue-17758/2024-11-03-20-00-21-P1060032.RW2 (sorry about the cat hair, it's everywhere :-D)

[jenshannoschwalm]

Could you shoot one of colorful content? Bunch of flowers?

[Maciejka1]

BTW i am really surprised about the bad signal/noise ratio in these images. Could it be the camera has a hardware problem or became very hot?
These photos where shot after sunset in really bad light usually 6400 is great (I think)

I have taken a photo in iso 6400. Here it is with only the minimal stack.

I can clearly see the difference when comparing preview in DT and with high quality processing/export even though it is really slight and I am not sure you can see it here. Color very slightly shifts to green just like in title dog photo. All are shot in -3 EV because this is very noticable when image is dark. Changing demosaic helps but it is still not perfect.


Outside the topic is this noise on 6400 acceptable? I have no experience with any other cameras.

different output after HQ pipe is toggled

Yes the issue can be reproduced even on iso 100 images. Not as drastic but colors still shift to green. I will test it on my windows in a minute.

[Maciejka1]

Sorry missclick.

[Maciejka1]

Issue can still be reproduced on windows. I have changed demosaic to amaze + vng4 since that looks a bit better but still image changes drastically when is exported/HQ mode. Worth to mention that this image with minimal stack still vastly changes color.

[jenshannoschwalm]

@kofa73 i can clearly reproduce here with your mouse-pad image.

The reproducer:

1. Just try minimal stack like this
   
2. Toggle HQ-processing button in darkroom. You won't see the color shift described in the original title but you will clearly see a darker vs brighter effect.

Could this be related to what @hanatos posted recently?

I don't think it's this point here. I currently think it's the scaler (see other issues about that #13335 #13682). I have been reading through image-scaling literature recently and from my current understanding the way we downscale by a factor of >>2.0 after demosaicing seems to be wrong to me. See my current PR about demosaicing - that's about refactoring demosaicer code so we could handle this scaling issue in a much safer way.

[jenshannoschwalm]

Outside the topic is this noise on 6400 acceptable? I have no experience with any other cameras.

I am really surprised about the strong noise here. Unfortunately i don't have any images from your camera and can't evaluate further. Not sure is the black/white points are ideal but that doesn't change much in this issue. If you want to do so, you might share some low-iso raw images here so we could inspect. @kofa73 is also very experienced with tracking down issues to the bottom :-)

EDIT: i actually found 4 other ILCE-6000 images in my test-collection, all with ISO 100-200 and all are just "perfect" and don't show the issue in any significant way.

[Maciejka1]

So is there any way to fix this? Or is it an issue on deeper level? For now I will change demosaic since that helps a bit.

[kofa73]

Could you shoot one of colorful content? Bunch of flowers?

I don't know if that was for me or not, but here it goes - no flowers, but a bunch of toys with bold colours:
https://tech.kovacs-telekes.org/files/2024-11-03-darktable-issue-17758/2024-11-04-11-00-04-P1060036.RW2

[jenshannoschwalm]

I don't know if that was for me or not, but here it goes

Actually to both of you :-) Yeah a nice example with lots of noise. Still far less noise than the OP example - due to better exposure.

I tested some more using an experimental filter before demosaic downscaling ... not convinced about my hypothesis any more. Actually i don't know. The big color blotches on the sony image ...

[jenshannoschwalm]

So is there any way to fix this? Or is it an issue on deeper level? For now I will change demosaic since that helps a bit.

I am still interested in one of your low-iso images as the ones i have from the same camera you are using don't show the problem.

Wildly pinging @kmilos @LebedevRI here. Does one of you have any idea about this?

[Maciejka1]

Here are Raws I have on hand from iso 100. Taken with -3 EV because I wanted to test if this issue happens with low iso. If they are bad I will take them in good exposure.
https://drive.google.com/drive/folders/1Leq8i26pv0qnd_OAjT0wLBZG7GeTuzvR?usp=sharing

[jenshannoschwalm]

Thanks, the noise is clearly under control up to ~400ISO. BTW - if you underexpose that much as in all the presented images you will certainly loose > 12dB of signal/noise ratio. Modern sensors can do magic here but it seems you should more stick to ETTR :-)

[kofa73]

Still far less noise than the OP example - due to better exposure.

I can create a more underexposed image, if needed. Or this one (from the D7000) is plenty underexposed: https://discuss.pixls.us/t/night-landing-high-iso/45689

[Maciejka1]

Iso to 1600 is not noiticable on social media or just phone. Right now I want to buy a6700 as my next camera. But let's get back to topic. Is it issue with my camera? Software isn't the newest because usb port is damaged and I cannot update it.

[jenshannoschwalm]

Thanks, same issue as reported.

[jenshannoschwalm]

So is there any way to fix this? Or is it an issue on deeper level? For now I will change demosaic since that helps a bit.

Likely the best option in current dt would be a preset for "raw denoise", you have to push the fine position quite to the top.

Is it issue with my camera? Software isn't the newest because usb port is damaged and I cannot update it.

Currently - after investigating the high-noise images from @kofa73 - i don't think so. For some reason dt has problems with such high noise images. Likely for very long but becoming obvious after people like you starting to use the HQ button making this obvious and not only by comparing exports.

[Maciejka1]

Still it affects image even with minimal stack... Noisy or not. Only thing I can do is use raw denoise?
Edit: For now I will edit noisy pictures with HQ mode enabled.

[kofa73]

@jenshannoschwalm raw denoise also changes colours:

[jenshannoschwalm]

@kofa73 @TurboGit @parafin @ralfbrown Folks i think i got it. I compiled a dt test version with the finalscale module being enforced to be just before colorout, it seems all issues are gone immediately, this can be checked as usual with the HQ button.

So indeed it seems to be the same issue as discussed in the above mentioned issues #13335, #13635 and #13682, down/upscaling done after changing to colorout.

1. Would be fine if others can reproduce this observation
2. Reminds me, we wanted to "fix" this some time ago :-)

How can we have progress here?

[TurboGit]

I'll try to reproduce this issue first.

How can we have progress here?

Do you see all the implication of moving the finalscale just before colorout?

This means that we would ONLY change clahe which will be now after finalscale (and colorout, but I don't see a problem here). This seems an issue to me, no? Having local contrast after the final scale will probably be far more sensitive when working on a scaled down image. I understand that this will be a new order not used for old edit expect if asked, but is local contrast ready to work on down-scaled images?

So we really want to do some testing here.

[TurboGit]

Oh, wait no big deal clahe is the old deprecated local contrast module. So probably no issue, I'll test.

[Maciejka1]

@kofa73 @TurboGit @parafin @ralfbrown Folks i think i got it. I compiled a dt test version with the finalscale module being enforced to be just before colorout, it seems all issues are gone immediately, this can be checked as usual with the HQ button.

Can I compile this version myself? Is there any branch for it?

[jenshannoschwalm]

I'll try to reproduce this issue first.

To reproduce, take any of the noise and underexposed images and switch off all modules that are not technically necessary to make sure it's not some "color magic" stepping in, maybe expose to see things better. (No temp .....) Notice the difference in "brightness" at least ...

[kofa73]

A very similar effect: Gimp uses linear space for scaling, but it seems Krita does not. Exported at full size from darktable, then scaled to width = 500 in both.

When I convert the image to linear space in Krita, before applying the scaling:

[TurboGit]

I have been able to reproduce with the rubix cube at ISO 6400. But when I switch finalscale before colorout I still have the issue:

I have switched retouch/liquify to have the iop-order written into the XMP:

darktable:iop_order_list="rawprepare,0,invert,0,temperature,0,highlights,0,cacorrect,0,hotpixels,0,rawdenoise,0,demosaic,0,denoiseprofile,0,bilateral,0,rotatepixels,0,scalepixels,0,lens,0,cacorrectrgb,0,hazeremoval,0,ashift,0,flip,0,enlargecanvas,0,overlay,0,clipping,0,spots,0,retouch,0,liquify,0,exposure,0,mask_manager,0,tonemap,0,toneequal,0,crop,0,graduatednd,0,profile_gamma,0,equalizer,0,colorin,0,channelmixerrgb,0,diffuse,0,censorize,0,negadoctor,0,blurs,0,primaries,0,nlmeans,0,colorchecker,0,defringe,0,atrous,0,lowpass,0,highpass,0,sharpen,0,colortransfer,0,colormapping,0,channelmixer,0,basicadj,0,colorbalance,0,colorequal,0,colorbalancergb,0,rgbcurve,0,rgblevels,0,basecurve,0,filmic,0,sigmoid,0,filmicrgb,0,lut3d,0,colisa,0,tonecurve,0,levels,0,shadhi,0,zonesystem,0,globaltonemap,0,relight,0,bilat,0,colorcorrection,0,colorcontrast,0,velvia,0,vibrance,0,colorzones,0,bloom,0,colorize,0,lowlight,0,monochrome,0,grain,0,soften,0,splittoning,0,vignette,0,colorreconstruct,0,finalscale,0,colorout,0,clahe,0,overexposed,0,rawoverexposed,0,dither,0,borders,0,watermark,0,gamma,0"

As you can see finalscale is before colorout, but I still have the color change when toggling HQ preview:

HQ off	HQ on

The green color cast when HQ is on is well visible. Did I missed some steps?

[kmilos]

Just some background read of interest: https://usage.imagemagick.org/resize/#resize_colorspace

[TurboGit]

@kmilos : Thanks, interesting reading indeed.

[Maciejka1]

I have compiled @TurboGit commit
Introduce new iop-order v3.1 where finalscale is before colorout.
but I was still able to reproduce the issue.

[TurboGit]

@Maciejka1 : You need to reset image history or select the 3.1 IOP order.

[Maciejka1]

With 3.1 raw input it is better but still not solved. Luminosity change is solved I think. If you zoom in color cast doesn't happen.