SSIM as a loss function and LPIPS input range

[niels-leif-bracher]

Hi Francois Rozet,
Im currently working with your Piqa package, especially the SSIM and LPIPS classes. There are two things I wanted to comment on.

First, I think the correct definition of the SSIM as criterion should be:

criterion = SSIM()
l = -criterion(x, y)

The minus is needed because you don’t have a different score defined with SSIM but a similarity score which is best when it returns 1 right? So with the minus, the returned range of values is [-1, 0] such that the backpropagation used with .backwards() can find the optimal minimum.

Moreover, why do you define SSIM as just the mean part of the SSIM score? So why you ignore the Contrast Sensitivity part of the SSIM when you define l in SSIM to be ssim(…)[0]. Would it be possible to make that an option? And by default return ss*cs ?

Secondly, for the LPIPS package I found in other packages that the input should be in range [-1, 1]. Which input range is needed for your package?

Kind regards

Niels Bracher

[francois-rozet]

Hello Niels 👋

Thanks for your interest!

The minus is needed because you don’t have a different score defined with SSIM but a similarity score [...]

By definition, SSIM is a similarity score, meaning that it is higher when the inputs are closer. I cannot define it otherwise. When a loss function is needed, we usually define a metric SSIMLoss(x, y) that computes 1 - SSIM(x, y) which is a measure of dissimilarity. An easy way to do that with PIQA is

class SSIMLoss(SSIM):
    def forward(*args, **kwargs):
        return 1. - super().forward(*args, **kwargs)

Moreover, why do you define SSIM as just the mean part of the SSIM score? So why you ignore the Contrast Sensitivity part of the SSIM when you define l in SSIM to be ssim(…)[0]. Would it be possible to make that an option? And by default return ss*cs?

The Contrast Sensitivity (CS) is not ignored, it is already in SSIM. This is described in the documentation and the product by cs is in the code:

piqa/piqa/ssim.py

Line 111 in a3ca951

ss = (2. * mu_xy + c1) / (mu_xx + mu_yy + c1) * cs

FYI, the implementations of PIQA are tested against other ones (see benchmark.py) to be sure they are correct.

I found in other packages that the input should be in range [-1, 1]. Which input range is needed for your package?

The short answer is [0, 1].

In the original implementation the inputs were indeed scaled to [-1, 1]. This was quite annoying because it doesn't match the conventions of torchvision. Furthermore, the inputs are normalized w.r.t. ImageNet, making this scaling useless and extra-annoying because the mean and variance of ImageNet are usually given for [0, 1] images. Therefore, I made the choice to use [0, 1] inputs to uniformize all the metrics of PIQA, like a lot of IQA libraries did.

If you have any more questions, feel free to ask 😉

[niels-leif-bracher]

Thanks for your answer. It seems that i didnt have read *cs from line 111 of your code.
Defining SSIM as the criterion and then go from there with l.backwards() seems still confusing and misleading to me. At least on your main page where you have the example with SSIM as loss criterion there a minus or 1- this criterion should be the right way to go :)
And thanks for answering the question about the LPIPS input range. I hoped that it is in line with the torchvision implementation. Great work !

[francois-rozet]

Oh I didn't get that it was the examples that you found misleading! I will change that, thanks!