Images are too similar

[sneccc]

Hello i just tested kandinsky and for the same prompt the images are super identical can this be because the model is super overfitted?

There is no variation of place,person,clothes. etc

[seruva19]

Hello. Yes, noticed the same thing, but, honestly, I don't know the cause, need more research to figure it out.

[Acephalia]

This also seems to be more prominent on 2.2

[steto123]

I have used manually generated random seeds for this two pictures.

[Acephalia]

@seruva19 Made some progress on this.

Setting the Prior Scale to 1 fixes the repetitiveness significantly. (I'm still trying to figure out what prior scale actually does.)

Also there is a bug where batch count/size does not change the seed even if it is set to random (-1) which is why a batch looks significantly similar. It just picks one seed and sticks to that for all generations instead of randomising it for each image.

[seruva19]

Thank you for sharing.
Based on the diffusers implementation, when the guidance_scale parameter is set to 1, the classifier-free guidance mechanism during the prior inference phase is effectively disabled. And (I believe?) in this case the model behaves more like SD1.5, directly passing prompt embedding into the UNet, bypassing prior transformer model. So, the output becomes less strictly conditioned on the input prompts.

Regarding prior scale (guidance scale for prior inference phase).
From my understanding, when guidance_scale > 1, the following happens:

• Text encoder used negative_prior_prompt to generate unconditional embeddings
• Both unconditional and text ("positive") embeddings are concatenated into single prompt embeddings tensor (to avoid need for double forward pass)
• Prior transformer predicts CLIP image embedding from prompt embeddings
• CLIP embedding is divided back into two chunks, the first chunk is unconditional image embedding and second chunk is text image embedding
• Result image embedding is constituted as unconditional embedding + guidance_scale * (text embedding - unconditional embedding)
• Notably, if guidance_scale = 1, then image embedding equals text embedding regardless of negative embeddings, that's why no classifier free guidance is used in this case
• Overall idea is to bring the resulting image embedding closer to the positive text embedding while using the unconditional (negative) embeddings as a foundational reference. By increasing prior_scale, we decrease difference between the conditional (text) and the unconditional embeddings (decreasing the influence of unconditional tokens).

Sorry if this explanation was chaotic and no very helpful, I am also in process of understanding how all this works. I hope this is accurate enough, but if someone will correct me, I'll be grateful :)

[seruva19]

About the bug you are mentioned (with wrong seed), could it be reproduced somehow? I did not witness anything like this before...

[Acephalia]

I’m going to have to read that when it’s not 2.30am and with lots of coffee. You are gem of a human being though. I’ll go through it and report back.

As for the bug on my ends it just seems to happen when I set the batch size or count to anything above 1. Basically all the images generated have the same initial seed that’s assigned randomly. For example if I set a prompt and set batch to 4 and click generate a random seed is assigned for the first image. Subsequent images in the batch also use that same seed. So if the initial random seed was 1247382 all images generated in the batch have 1247382 as the seed.

(If I start a new generation batch the seed is randomised again but all the images within that batch will have that same seed.)

Do you have different seeds for each image at your end when generating more than one image with the same settings and prompt in a batch?

[Acephalia]

Ps is Guidance Scale the same as guidance scale? I assumed guidance scale worked like CFG in SD. Unless prior scale = prior guidance scale?

[seruva19]

Ah, I see. I almost never use batch_size > 1, that's why I never encountered this bug) I will check it.

Yes, "CFG scale" = "Classifier Free Guidance" scale, and "prior scale" is actually "prior guidance scale", you're right. I indeed should have the label fixed, it might be confusing.