Canon EOS D60 files are extremely green

[garrett]

To Reproduce

1. Open a raw file from a Canon EOS D60 (not to be confused with a Canon EOS 60D, which came quite a bit later).
2. Observe wrong colors (it's usually extremely green).

A Canon D60 is a very old camera, but I used it a several years and would love all of my photo library to work in darktable. 😉

Expected behavior
Images should have natural colors. Opening in other raw editors, such as RawTherapee, gives expected results.

It's not a color balance issue, as greens persist in the dark parts of an image even after a color balance attempt to fix.

Also, I should note that I have had a handful of cameras since, both from Canon (5DmkII) and Fuji cameras, and colors are fine from default. And I also know not to expect raws to look like out of camera JPEGs and that different raw processors handle things differently, but darktable with a D60 is something else (clearly a bug somewhere).

Screenshots
darktable, with modern defaults (Filmic & new color module for white balance):

RawTherapee with defaults:

Another one; darktable:

RawTherapee defaults:

Platform

• darktable version : 3.5.0~git2563.319d682233-6084.1, but this happens in 3.4 as well.
• OS : Linux - kernel 5.12.11-300.fc34.x86_64
• Linux - Distro : Fedora 34
• Memory : 16 G
• Graphics card : Intel 620
• Graphics driver : i915
• OpenCL installed : yes, Intel Neo: intel-opencl-21.23.20043-1.fc34.x86_64
• OpenCL activated : yes
• Xorg : XWayland
• Desktop : GNOME
• GTK+ : gtk3-3.24.29-1.fc34.x86_64
• gcc : 11.1.1 20210531 (Red Hat 11.1.1-3)

Additional context

Canon D60 raw files had sidecar JPEG previews when raws were taken. Here's the raw and the ".thm" sidecar:
CRW_1456.zip

And the .thm renamed to a JPEG so GitHub accepts it:

And the same for the bird photo:
CRW_2869.zip

• Can you reproduce with another darktable version(s)? Yes, 3.4. I am pretty sure it worked fine a while ago, but didn't work in 3.4 and possibly not 3.2... but I think it worked before that?

• Can you reproduce with a RAW or Jpeg or both? Most D60 raws have this issue to one degree or another.

• Are the steps above reproducible with a fresh edit (i.e. after discarding history)? Yes

• If the issue is with the output image, attach an XMP file if (you'll have to change the extension to .txt)

• Is the issue still present using an empty/new config-dir (e.g. start darktable with --configdir "/tmp")? Yes

  

• Do you use lua scripts? Yes, but it happens with a fresh config, with no Lua scripts enabled.

Changing the color balance does help somewhat, especially with some photos, but it never fixes the problem. And some photos are worse than others, but underexposed photos or those with details at the dark end are especially affected (even after white balance correction).

---

Additionally, I am happy to license these raw files as CC0 and would cheerfully submit more raw files that were taken under different conditions, if it helps. Unfortunately, I gave this camera to my sister (who used it to shoot in JPEG) many years ago, so I only have my existing collection.

[garrett]

darktable 3.0.1 (on Fedora 32, tested from toolbox... the furthest I can go back w/o installing and using a VM) had the same issue:

(Thumbs look correct, but I didn't edit the photos. Still, they don't have their sidecar JPEG-based THM files, so I wonder if there's an additional JPEG embedded inside the CRW files or if this version of darktable skips the bug when processing thumbnails.)

Difference in rendering (while still wrong) is probably due to a basecurve and old white balance versus filmic rgb and the new color calibration module.

[garrett]

OK, perhaps darktable never handled Canon EOS D60 files properly. I remembered I could run Debian in toolboxes in Fedora, so here's darktable 2.2.1 from Debian 9:

[johnny-bit]

I've checked with sample file available on raw.pixls.us - https://raw.pixls.us/getfile.php/2010/nice/Canon%20-%20EOS%20D60%20-%20RAW%20(3:2).CRW and it doesn't exhibit same problems as your files.

for some reason your files read "as shot" wb coefficients as 1,1,1 which is kinda bonkers. sample file on raw.pixls.us has 2.624,1,1.098 wb coeffs for "as shot"...

I wonder what's more different about your shots than the sample on raw.pixls.us...

Enabling camera reference (d65) mode + color calibration also causes same problem: green everything. but if you choose wb out of say white walls on the house it figures out correct. So there IS a workaround in darktable itself...

[LebedevRI]

I guess a wrong whitebalance field is being read for some reason.

[garrett]

but if you choose wb out of say white walls on the house it figures out correct

Perhaps it might in that photo, but it doesn't really. The dark tones are still heavily shifted green, even after adjusting the white balance. I should look for another photo which shows this more prominently.

I've been setting white balance and adjusting things with color balance to try to overcome it, but it doesn't really work.

An interesting thing is that other raw converters handle the files from my old D60 correctly. (But I don't want to use other raw convertors... I ❤️ darktable and want to use it for all my cameras.)

[johnny-bit]

OK, then we need couple CC0 samples from D60 that exhibit the issue and figure out why the sample from RPU doesn't exhibit the problem :)

[garrett]

I looked quickly for an underexposed raw file and a better exposed one and found these two. Try changing the white balance on the underexposed one. It's still sorely green, and a raw file that's underexposed like that should be salvageable. The other, more properly exposed file is mostly OK, especially after a white balance adjustment. (It still has some issues, but it's subtle.)
d60-castle.zip

License: CC0 for these raw files as well as the ones above.

Here's what the underexposed file looks like with defaults:

I did use the D60 to take a picture of my ColorChecker Passport when I first got one, but I didn't know what I was doing with it so much and also converted it as a DNG (and don't have the CRW, at least easily available... if it still exsits), so it's probably not useful.

I'll look for more examples.

[garrett]

This penguins photo is in daylight and largely monochromatic. While penguins aren't pure black and white, they're close enough. It's exposed about right and adjusting the balance gets it 90% of the way there in this photo. But it's still not right.

darktable default:

darktable white balance only (look at the waveform in the right on the bottom and how the red channel isn't aligned properly):

darktable, with color balance tweaked, with highlights, dark tones, mid tones all selected from the penguins (again, look at the waveform in the top right and compare to the previous waveform):

RawTherapee defaults for comparison (it's a bit darker than darktable, but the color is right from the beginning):

THM sidecar JPEG:

Penguins raw file with XMP and THM sidecars. Two XMP files are included; one for the default, the other with the modification, including the tweaked color balance module:
d60-penguins.zip

[garrett]

Here a few more examples, including one underexposed with blown highlights (and the silver should be mainly neutral, which it isn't after a white balance), one with a grey card in the shade in a forest, and two flowers where the one with a bee is wrong, but somehow, the one by itself (CRW_7289) seems fine with the defaults. Same camera for all.

Preview:

Files:

d60-additional.zip

Again, I'm licensing all of these under CC0. Whichever ones you think are most useful, we can upload to the raw db.

[garrett]

I ran dcraw -i -v on the files to inspect the information and see if there's anything different. The only thing that came back as different are the multipliers, so I filtered for that.

I'm assuming that the example in the DB (the first one in this list) is in daylight. Most of mine are in normal daylight too, although some are in the shade. Shouldn't the multipliers be similar? Could it be that different D60 cameras reported different multipliers?

Is it possible that the multipliers are user-set? And perhaps darktable is or isn't taking these into account, but other raw editors are... or aren't?

Here's the list (daylight is the same for all, but the camera multipliers are different):

Filename: Canon - EOS D60 - RAW (3 2).CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 1989.000000 758.000000 832.000000 756.000000

Filename: CRW_1456.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2695.000000 824.000000 977.000000 817.000000

Filename: CRW_2869.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2603.000000 821.000000 901.000000 815.000000

Filename: d60-additional/CRW_1508.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2070.000000 830.000000 1026.000000 823.000000

Filename: d60-additional/CRW_1551.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2178.000000 830.000000 1118.000000 823.000000

Filename: d60-additional/CRW_7289.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2051.000000 833.000000 1307.000000 826.000000

Filename: d60-penguins/CRW_4919.CRW
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2287.000000 827.000000 1002.000000 820.000000

Filename: d60-castle/crw_6154.crw
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2654.000000 822.000000 915.000000 815.000000

Filename: d60-castle/crw_6155.crw
Daylight multipliers: 2.799998 0.919642 1.004551
Camera multipliers: 2766.000000 821.000000 911.000000 815.000000

[jenshannoschwalm]

I think we already have the underlying problem, as @johnny-bit mentioned, the coefficients should not be 1,1,1 !

So either the decoder reads wrong numbers or the file is wrong. Could it be, you used other software on those files, that might write a modified raw file?

On the phone so I can't test myself but does exiftool give any hint here?

[johnny-bit]

Exiftool output differs - the sample from raw.pixls.us HAS WB RGGB Levels field while @garrett's files don't.

for example here's one of garett's images:

$ exiftool CRW_1456.CRW | grep WB
WB RGGB Levels Auto             : 2178 830 823 1063
WB RGGB Levels Daylight         : 2224 828 821 1012
WB RGGB Levels Shade            : 2433 825 818 935
WB RGGB Levels Cloudy           : 1626 959 951 1829
WB RGGB Levels Tungsten         : 1848 871 864 1424
WB RGGB Levels Fluorescent      : 2520 822 815 909
WB RGGB Levels Flash            : 2695 824 817 977

and here's RPU sample:

$ exiftool Canon\ -\ EOS\ D60\ -\ RAW\ \(3\ 2\).CRW | grep WB
WB RGGB Levels Auto             : 1989 758 756 832
WB RGGB Levels Daylight         : 1856 784 781 945
WB RGGB Levels Shade            : 1976 759 757 848
WB RGGB Levels Cloudy           : 1378 937 933 1774
WB RGGB Levels Tungsten         : 1635 867 864 1439
WB RGGB Levels Fluorescent      : 2004 741 738 804
WB RGGB Levels Flash            : 1742 828 825 1273
WB RGGB Levels                  : 1989 758 756 832

[jenshannoschwalm]

So I guess we know why it's greenish. Maybe there was a camera firmware change later that includes that data?

Or exiftool or other software had been used wrongfully writing crap? I did that a long time ago on some images.

We could ensure data via adobe-coeffs I guess. You can assign me for 3.8 as there is more stuff for that with dng files pending if you want.

[johnny-bit]

The firmware is the same - 1.0.4.

[PeterWem]

Same with my D30 #7366 and if I remember it right, also with my D60. I can take Auto, Daylight, Shade, Cloudy, Tungsten, Fluorescent and Flash with a Spydercheckr and D60 if you want.

[garrett]

@jenshannoschwalm, @johnny-bit: Thank you both so much for looking into this (and for everything you both do in darktable)!

@PeterWem: (an aside) Oh? You have a D60 still? Awesome! Would you mind making a noise profile too? https://pixls.us/articles/how-to-create-camera-noise-profiles-for-darktable/

[PeterWem]

If I remember it right I never succeded to take the noise profile samples for D30 and D60. I couldn't clip all channels in the highlights no matter what (or darktable never showed that all channels were clipped). Especially at higher ISO. I will check again next week and create a new issue.

[johnny-bit]

Don't do noise profiles until the wb problem is fixed - it'll skew results.

@PeterWem - please simply do the wb shots with checker on D60 and licence them CC0. Would probably help in analyzing why the data isn't where rawspeed's looking. Make sure to transfer files directly to PC, so we know it's purely what camera writes to the card.

[garrett]

Aha!

I was looking at the results from the exiftool on one of the problematic images and noticed the following field:

Black Levels                    : 135 130 135 131

But darktable wasn't using these in the raw white/black point module; it was using 206 206 208 208. Once I changed the values manually, the colors were fixed, and so was the "too dark" look with clipped dark tones on some files. I didn't touch the white point.

(This random file happened to be a photo of my sister's graduation, which I'm not sharing here. But the same idea should apply to other files. I'll try this out on the broken ones I posted.)

[garrett]

Using the default in modern white balance mode on an otherwise fresh darktable install, I changed just the black levels.

exiftool reported:

Black Levels                    : 132 127 132 127

I changed them from 169 169 169 169 to 132 127 132 127 in darktable's "raw white/black point" module and it fixed the image. Here's a screenshot, with a comparison (broken above the line, fixed below):

[garrett]

Ah, it doesn't fix it for the penguins in modern mode, unless I manually set the white balance too. So it's indeed both a white balance and a black level issue. But setting the white balance to auto and manually setting the white balance fixes the penguins picture.

[PeterWem]

CC0 https://drive.google.com/file/d/1JrHKJE7lq0Gbv9yUFlAeI18di6xRN7uC/view?usp=sharing
I didn't see any green pictures this time. Not more than the one with custom white balance.
Sorry, no sunlight today for the color checker.

[garrett]

@PeterWem: Interesting. Some of my photos are worse than others, but sometimes they'd look OK. But it's related to most of them not having the "WB RGGB Levels" (without a type) and also darktable not using the same black levels as exiftool reports.

However, I downloaded your files, opened them in darktable and also ran exiftool on them to find the "Black Levels":

File Name                       : CRW_7017.CRW
Black Levels                    : 127 127 127 127

File Name                       : CRW_7018.CRW
Black Levels                    : 127 126 127 126

File Name                       : CRW_7019.CRW
Black Levels                    : 126 127 126 127

File Name                       : CRW_7020.CRW
Black Levels                    : 127 127 127 127

File Name                       : CRW_7021.CRW
Black Levels                    : 127 127 127 127

File Name                       : CRW_7022.CRW
Black Levels                    : 128 126 128 126

File Name                       : CRW_7023.CRW
Black Levels                    : 127 126 127 126

They don't match up between exiftool and darktable, but they aren't as far off as some of my files, so I think your files might be very subtly problematic? What would happen if you intentionally underexposed a photo and push the exposure up in darktable?

Also, I checked for the WB RGGB values:

File Name                       : CRW_7017.CRW
WB RGGB Levels Auto             : 1930 795 784 1023
WB RGGB Levels Daylight         : 1945 783 773 969
WB RGGB Levels Shade            : 2065 759 749 870
WB RGGB Levels Cloudy           : 1469 937 925 1831
WB RGGB Levels Tungsten         : 1687 854 842 1438
WB RGGB Levels Fluorescent      : 2111 748 738 835
WB RGGB Levels Flash            : 1945 783 773 969
WB RGGB Levels                  : 1945 783 773 969

File Name                       : CRW_7018.CRW
WB RGGB Levels Auto             : 1935 795 784 1023
WB RGGB Levels Daylight         : 1945 783 773 969
WB RGGB Levels Shade            : 2065 759 749 870
WB RGGB Levels Cloudy           : 1469 937 925 1831
WB RGGB Levels Tungsten         : 1687 854 842 1438
WB RGGB Levels Fluorescent      : 2111 748 738 835
WB RGGB Levels Flash            : 1945 783 773 969
WB RGGB Levels                  : 1469 937 925 1831

File Name                       : CRW_7019.CRW
WB RGGB Levels Auto             : 1933 794 784 1022
WB RGGB Levels Daylight         : 1945 783 773 969
WB RGGB Levels Shade            : 2065 759 749 870
WB RGGB Levels Cloudy           : 1469 937 925 1831
WB RGGB Levels Tungsten         : 1687 854 842 1438
WB RGGB Levels Fluorescent      : 2111 748 738 835
WB RGGB Levels Flash            : 1945 783 773 969
WB RGGB Levels                  : 1687 854 842 1438

File Name                       : CRW_7020.CRW
WB RGGB Levels Auto             : 1933 794 784 1021
WB RGGB Levels Daylight         : 1945 783 773 969
WB RGGB Levels Shade            : 2065 759 749 870
WB RGGB Levels Cloudy           : 1469 937 925 1831
WB RGGB Levels Tungsten         : 1687 854 842 1438
WB RGGB Levels Fluorescent      : 2111 748 738 835
WB RGGB Levels Flash            : 1945 783 773 969
WB RGGB Levels                  : 2111 748 738 835

File Name                       : CRW_7021.CRW
WB RGGB Levels Auto             : 1930 795 784 1023
WB RGGB Levels Daylight         : 1945 783 773 969
WB RGGB Levels Shade            : 2065 759 749 870
WB RGGB Levels Cloudy           : 1469 937 925 1831
WB RGGB Levels Tungsten         : 1687 854 842 1438
WB RGGB Levels Fluorescent      : 2111 748 738 835
WB RGGB Levels Flash            : 1945 783 773 969
WB RGGB Levels                  : 1945 783 773 969

File Name                       : CRW_7022.CRW
WB RGGB Levels Auto             : 1933 794 784 1024
WB RGGB Levels Daylight         : 1945 783 773 969
WB RGGB Levels Shade            : 2065 759 749 870
WB RGGB Levels Cloudy           : 1469 937 925 1831
WB RGGB Levels Tungsten         : 1687 854 842 1438
WB RGGB Levels Fluorescent      : 2111 748 738 835
WB RGGB Levels Flash            : 1945 783 773 969

File Name                       : CRW_7023.CRW
WB RGGB Levels Auto             : 1935 795 784 1023
WB RGGB Levels Daylight         : 1945 783 773 969
WB RGGB Levels Shade            : 2065 759 749 870
WB RGGB Levels Cloudy           : 1469 937 925 1831
WB RGGB Levels Tungsten         : 1687 854 842 1438
WB RGGB Levels Fluorescent      : 2111 748 738 835
WB RGGB Levels Flash            : 1945 783 773 969
WB RGGB Levels                  : 1935 795 784 1023

It looks like CRW_7022.CRW is missing the "WB RGGB Levels" without a modifier (Auto, Daylight, etc.), like most of my problematic files. Did you do anything special when taking that photo?

Sure enough, that one file has issues in darktable (brand new config with all defaults, made for the directory; legacy whitebalance) here...

[PeterWem]

The green one was custom white balance, so I guess it should be green.

[garrett]

When using my D60, I do remember I would often use a custom white balance based on a grey card, as its auto balance wasn't so great. I guess that could be a reason why a lot of my files have this problem? In my case should've been set to a proper WB and not green, however. (And other raw processors get this right.)

It still has issues with the wrong black/white point outside of the whitebalance, but custom WB is probably part of the mystery behind this bug and why it shows up in some files and not others (such as the example file in the raw repository)?

[PeterWem]

Perhaps my mistake. Check the thumbnail that I included, the THM file. It is not green.
I have been trying to get noise profile raw files for you also, but I can't saturate blue channel at ISO 200 and ISO 400.

[johnny-bit]

@PeterWem - Plz open separate issue with noise profiles. Some cameras can't saturate some colours at super low iso. So it might be the case for that. And rawfiner could look at the profiles, diagrams etc and diagnose out proper codes.

[PeterWem]

CC0 https://drive.google.com/file/d/1s45wL5UBZqjTZxKsteiAiGePu6OMMJNW/view?usp=sharing
Auto WB. ISO 100-1000 and under exposed ISO 100-1000.

[garrett]

Check the thumbnail that I included, the THM file. It is not green.

Yeah, that's consistent. None of my THM files are green either... it's just the raw file in darktable. I'm happy to see it's definitely not just some weird thing with my old specific D60. 😁

[PeterWem]

All custom WB are green. Compare with THM files.

D60-part-3-custom-white-balance.zip

[garrett]

Would it be possible to fix this with two PRs?

1. Fix for the broken reading of the black point
2. Another fix for the white balance change

Manual white balance changing is pretty easy, especially when there's something neutral (or close to it) in the scene. However, black point being incorrect isn't obvious and it is a pain to find the raw file, read the values from exiftool, and manually change the values in darktable to those that exiftool reports.

If we can't get both in for the next version of darktable (as it's late in the cycle), I'd really love the black point reading to be fixed at least, as that's the more difficult one to deal with when editing the files.

[johnny-bit]

@garrett - code freeze is in couple hours for release 3.6, so it's more likely that if the fix will be there, it'll be part of 3.8 release. Or 3.6 point release - you never know.
As for the fix itself - that depends on who does the fix and ultimately rawspeed maintainer.

[garrett]

code freeze is in couple hours for release 3.6

🎉 Awesome to hear! I didn't realize we're so close. I've switched to nightlies lately, and this is going to be an awesome release (as always).

3.6 point release

I hope so! Point releases usually seem to include bugfixes and camera support, and I think (and hope) this qualifies.

[github-actions]

This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[garrett]

Would the fix in #10008 (comment) be similar to the one needed here? It had an incorrect black point as well.

[PeterWem]

Never passed through, but that will solve the green cast and the wrong black level.
darktable-org/rawspeed#402

[kmilos]

Is the black level part of the problem the same as darktable-org/rawspeed#717?

[PeterWem]

The problem with D60 is that the whole left optical black area is used when you only should use around 90% of it. Just decrease the area that is measured and there will be no problem, see the links.

The problem with 2000D is that the left columns that are black in some 2000D models aren't black at all with at least one 2000D. Solution is to not use the most left columns.