Discussion on GAN tricks on modern methods and issues

[Baran-phys]

Dear all,
I am creating this discussion session for sharing our experiences on different GAN models and the corresponding hyperparameter choices. As you definitely know, GANs are notoriously hard and unstable to train, and as we go further from the Vanilla GAN, the hyperparameters (hp) become more complex. Furthermore, based on the applications and use-cases of GANs on different datasets, the structure and hp's will change, so our solutions to different known problems such as mode collapse or model divergence. The solutions to the problems of GAN are limited to either standard datasets or restricted by the lack of local memory. Moreover, due to the lack of a unified place to see other people's solutions to their GAN problems and discuss new ideas and tricks, I hope this place would be a great one to discuss such ideas. I chose here, as I found StudioGAN a thorough package, thus perhaps we can create a comprehensive solution-based discussion here as well.

[Baran-phys]

As the first question, has anyone used global max pooling or global average pooling, instead of global sum pooling in the discriminator (after all res_blocks)? Do you think the impact will be distinguishable?

[mingukkang]

Perhaps two years ago, I tried to implement SNGAN using TensorFlow.
I used Global Average Pooling unconsciously, but SNGAN counldn't converge evenif I had checked almost all details of the GAN. After changing this with GSP, everything goes well. This is my experience about GAP and GSP.

[Baran-phys]

Interesting. I know from experience that GAP in ResNet-based WGAN-GP (without SN) did a fine job. Also, in another experience of mine, GMP works very well in Conv-based WGAN-GP. Based on the dataset that I have worked on, I would have said that GMP was efficient since the image was gray-scale. But, the GAP successful example contradicts it. For some reason, it seems that SN-based models favor GSP more.

[mingukkang]

Nice heuristic.
Thank you!

[Baran-phys]

@mingukkang In ContraGAN, if the cosine similarity score of the embedding layer does not show the correct diversity ( showing similar scores for many classes although the images are not that similar), does it mean that we will have mode collapse? If so, should we increase the number of embedding dimensions (hypersphere dim.) or add weight decay to the shared embeddings?

[mingukkang]

Hi,

I think this is a class confusion problem of ContraGAN.

The main difference between ContraGAN and ProjGAN is that ContraGAN considers data-to-data relationships while ProjGAN only contemplates data-to-class relations and regards each class as having independent characteristics.

I think that the confusion comes from classes whose features are similar to each other, but 2C_loss can not effectively figure it out.

The solution to this problem is finding good hyperparameters for better optimization.

In my experience, the following settings helped me to train an unconfused ContraGAN.

Higher-dimensional hypersphere,
Positive_collected_numerator: False.

Thank you.

[Baran-phys]

Hello @mingukkang , I tried your remedy. With "Positive_collected_numerator: False", the result got worst, and with a higher-dimensional hypersphere, the result stayed the same. I was wondering since my image dataset is related somehow, and they are not totally distinct images, doing a (multi-head) attention over the embedding layer of the discriminator would help the inter-class awareness for the output of the embedding layer (in which there will be a weighted sum).

As my second question, I remember in the "Generative Dog Images" competition for Kaggle 2 years ago, people repeatedly were saying that large batch size was not of any help and it was even diminishing their performance. And in the end, BigGAN users ended up with bs=16. Well, I know there have been even papers arguing about the importance of batch size, or even in the BigGAN paper, there was huge support for the increment of this hp. I wonder how significant the increase of bs is in total? should it be double the number of classes or one can alleviate this hp by twiddling with another hp?

Also, with that Kaggle competition, there was the idea that increasing the channel size does not help that much either. From yours (or anyone reading this) What is your comments on this?

[PeterouZh]

"should bs be double the number of classes?" Does this idea have references? I found a similar phenomenon for ProjGAN.

[mingukkang]

As my second question, I remember in the "Generative Dog Images" competition for Kaggle 2 years ago, people repeatedly were saying that large batch size was not of any help and it was even diminishing their performance. And in the end, BigGAN users ended up with bs=16. Well, I know there have been even papers arguing about the importance of batch size, or even in the BigGAN paper, there was huge support for the increment of this hp. I wonder how significant the increase of bs is in total? should it be double the number of classes or one can alleviate this hp by twiddling with another hp?
==> I truly agree with you.
Large batch size sometimes deteriorates the training of GAN. Heuristically, I choose a batch size greater than 2*num_classes. Also, I recommend that you set a larger learning rate as you increase the batch size.

I tried your remedy. With "Positive_collected_numerator: False", the result got worst, and with a higher-dimensional hypersphere, the result stayed the same. I was wondering since my image dataset is related somehow, and they are not totally distinct images, doing a (multi-head) attention over the embedding layer of the discriminator would help the inter-class awareness for the output of the embedding layer (in which there will be a weighted sum).
==> Recently, I have found that both ProjGAN and ContraGAN have the class confusion issue when training GANs on fine-grained dataset. I am trying to alleviate the class confusion using a new loss function. I will open implementations I have developed after paper submission (also will add an explanation of why this is happening).
Thanks for your patience.

doing a (multi-head) attention over the embedding layer of the discriminator would help the inter-class awareness for the output of the embedding layer
==> Awesome, Thank you for your insight.

[Baran-phys]

@mingukkang Hi, I was wondering about sth. What happens to the image-to-image similarity term in the 2c loss, if there will be no common class labels in the batch (no class image occurs twice e.g a batch of size 4 with images from classes 1,2,3,4)? So, in this case, the image-to-image term 1_{y_i=y_k}.exp(l(x_i)l(x_k).T) in the numerator becomes 0?