MSE reconstruction loss is computed in two different ways

[axu-git]

In HVAE, the mse code is:

        if self.model_config.reconstruction_loss == "mse":
            # sigma is taken as I_D
            logp_x_given_z = -0.5 * F.mse_loss(
                recon_x.reshape(x.shape[0], -1),
                x.reshape(x.shape[0], -1),
                reduction="none",
            ).sum(dim=-1)
            # - torch.log(torch.tensor([2 * np.pi]).to(x.device)) \
            #    * np.prod(self.input_dim) / 2

In RHVAE, it is given by:

        if self.model_config.reconstruction_loss == "mse":
            # sigma is taken as I_D
            recon_loss = -0.5 * F.mse_loss(
                recon_x.reshape(x.shape[0], -1),
                x.reshape(x.shape[0], -1),
                reduction="none",
            ).sum(dim=-1)
            -torch.log(torch.tensor([2 * np.pi]).to(x.device)) * np.prod(
                self.input_dim
            ) / 2

Why is there a difference in how the MSE loss is computed for HVAE and RHVAE (i.e., namely in HVAE the following expression is commented out, whereas in RHVAE it is not).

- torch.log(torch.tensor([2 * np.pi]).to(x.device)) \
                * np.prod(self.input_dim) / 2

And what does this term represent?

Thanks!

[clementchadebec]

Hello @axu-git,
Actually, as you see, the difference involves additional constant terms with respect to the input. In particular, these terms will only impact the value of the loss but not its gradients that are needed during training so both losses are actually equivalent. These terms are the normalizing terms of the pdf of the multivariate Gaussian distribution (https://en.wikipedia.org/wiki/Multivariate_normal_distribution).

I hope this helps,

Best,

Clement

[axu-git]

That is super helpful, thank you!