Question about how to use batches of images in ControlNet+ip-Adapter to generate images separately without the influence of the condition images in one batch #7933
-
Hello everyone, I am using ControlNet+ip-Adapter to generate images about materials (computer graphics, rendering). While trying to generate a material image with conditions of an adapter image and a Control-Net image, it was very successful. However, when turning to batch images, I found that the results were affected by all condition images in one batch. Thanks for everyone's help! The codes I have tested are as follows:
|
Beta Was this translation helpful? Give feedback.
Replies: 4 comments
-
It seems that loading multiple ip-adapters together to deal with a batch in the above way is designed to affect the generated results uniformly. I have changed my mind and wrote the following codes to figure out my problem.
The differences between the two paragraphs of code are mainly in building the prompt_embeds. I find that separately generating the prompt_embeds and cating them up work out. Hope my experience can help you. Or if there is another solution, please write and tell me. Thank you guys! |
Beta Was this translation helpful? Give feedback.
-
I want to understand a bit more of your problem, I'll do a simplified code of what you're doing and without list comprehension: controlnet = ControlNetModel.from_pretrained(...)
pipeline = StableDiffusionControlNetPipeline.from_pretrained(...).to("cuda")
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="models",
weight_name=["ip-adapter_sd15.bin"] * 4,
)
imgs_condition = ["control_image_1.png", "control_image_2.png", "control_image_3.png", "control_image_4.png"]
imgs_adapter = ["ip_image_1.png", "ip_image_2.png", "ip_image_3.png", "ip_image_4.png"]
prompt = [""]*4
output = pipeline(
prompt=prompt,
image=imgs_condition,
ip_adapter_image=imgs_adapter,
negative_prompt=prompt,
num_inference_steps=20,
generator=generator,
) So the problem here is that you want to generate a batch of 4 images, each with a different I tested this and IMO this is a bug, but I don't really use batch generations and don't know which is the real use case for this. @yiyixuxu @fabiorigano WDYT? |
Beta Was this translation helpful? Give feedback.
-
This bug bit me today too. Expected: Got: MultiDiffusion generates a batch of images in parallel with different prompts and then blends the latents together. Extending this to ip_adapters is a natural fit, but unexpectedly discovered these ip_adapters are currently all combined inside the batch. |
Beta Was this translation helpful? Give feedback.
-
Here's a first stab at what a fix might look like. For me this commit results in each ip_adapter getting run once when the number of ip adpaters matches the batch size. The is working for me fine now. |
Beta Was this translation helpful? Give feedback.
It seems that loading multiple ip-adapters together to deal with a batch in the above way is designed to affect the generated results uniformly. I have changed my mind and wrote the following codes to figure out my problem.