The GPU memory usage is too high

[peki12345]

About 60G? This is too scary, can it be optimized?

[8600862]

torch.cuda.OutOfMemoryError: CUDA out of memory.

[microbenh]

How to run pipeline in several GPUs, like 4*4090

[JPlin]

You can try this to in two GPUs:
https://huggingface2.notion.site/How-to-split-Flux-transformer-and-run-inference-aa1583ad23ce47a78589a79bb9309ab0

[c-steve-wang]

You can try this to in two GPUs: https://huggingface2.notion.site/How-to-split-Flux-transformer-and-run-inference-aa1583ad23ce47a78589a79bb9309ab0

Could you kindly provide the script of how this method works with the main.py?

[microbenh]

You can try this to in two GPUs: https://huggingface2.notion.site/How-to-split-Flux-transformer-and-run-inference-aa1583ad23ce47a78589a79bb9309ab0

It do not work. The transformer need 24GB, and controlnet 4GB, they have to be the same gpu.

[Nomination-NRB]

You can try this to in two GPUs: https://huggingface2.notion.site/How-to-split-Flux-transformer-and-run-inference-aa1583ad23ce47a78589a79bb9309ab0

It do not work. The transformer need 24GB, and controlnet 4GB, they have to be the same gpu.

It worked, I used two 3090s to get results, but inpaint was poor, and poorly followed prompt to redraw

[JPlin]

#27 Fixed some bugs, now need 28GB of VRAM.

[xhinker]

I can successfully run it using torchao using about 20G VRAM, the result is great

[shantzhou]

I can successfully run it using torchao using about 20G VRAM, the result is great

can you share the demo with me?

[timmerscher]

I can successfully run it using torchao using about 20G VRAM, the result is great

How? I'm trying quantization with bitsAndBytes, but the it seems that the gradient for the transformer is required, making the int8 version unusable!
I know time is precious, but I would really appreciate a guidance!