Could you add fp16 model version for omnigen?

[herocoming777]

your work is great, however my gpu 2080ti is old, can't support bf16,
if I run it directly in my local computer, it report this error: RuntimeError: cutlassF: no kernel found to launch!
After i changed the dtype of model config file from bfloat16 to float16, and changed the dtype in pipeline.py too, it runs well but ends with black images.
Any ideas for that solution or Could you add fp16 model version for omnigen?
Thank you so much!

[staoxiao]

Thanks for your attention to our work! We will test it with fp16, please stay tuned.

[herocoming777]

Thank you!
that's awesome, look forward to it!

[nitinmukesh]

Thanks for your attention to our work! We will test it with fp16, please stay tuned.

It will be great. Please also publish fp16 models and it will save space as well as processing time. Thank you.

[set-soft]

I found FP8 model here: https://huggingface.co/silveroxides/OmniGen-V1
Works very nice with this fork: https://github.com/newgenai79/OmniGen
@staoxiao: I also modified the code in the fork to detect if a directory called "model" exists, if this is the case the code uses the model from there, instead on relying on the more cryptic huggingface cache. This allowed me to play with FP8 model in a simple way.

[staoxiao]

Hi, @herocoming777 , unfortunately, we found that OmniGen cannot use FP16 for inference due to the use of LLMs (FP16 causes LLMs to crash). If you cannot use BF16, you'll have to use FP32, but be aware that FP32 will result in significant memory consumption and longer inference times.

import torch
images = pipe(
    …,
    dtype=torch.float32
)

We will try to overcome this issue in the next version of OmniGen.

[set-soft]

unfortunately, we found that OmniGen cannot use FP16 for inference due to the use of LLMs (FP16 causes LLMs to crash). If you cannot use BF16, you'll have to use FP32, but be aware that FP32 will result in significant memory consumption and longer inference times.

This fork: https://github.com/newgenai79/OmniGen is using FP8 for inference, the model needs around 6 GiB

[nitinmukesh]

@set-soft

Please create pull request for fp8 as you mention in other topic.

[herocoming777]

@staoxiao thanks for your trying.
yeah, I have tried the fp32, but VRAM couldn't handle it, at least 40G VRAM, not support by consumer GPU.
@set-soft Dear Sir, your FP8 recommend seems nice, is there any Comfyui node for this FP8 model? then I will try it immediately. thanks a lot.

[set-soft]

Hi @nitinmukesh !

Please create pull request for fp8 as you mention in other topic.

I'm not sure about it. @newgenai79 did the patch and I'm more interested on a Comfy_UI node, so I'm dedicating time to the node adaptation. If you want to try @staoxiao work, just to play with install the https://github.com/newgenai79/OmniGen repo. It works quite well, and you can even try disabling the quantization, is configurable.

[set-soft]

Hi @herocoming777 !

Dear Sir, your FP8 recommend seems nice, is there any Comfyui node for this FP8 model? then I will try it immediately. thanks a lot.

You can try this node: https://github.com/set-soft/ComfyUI_OmniGen_Wrapper is a fork of @chflame163 repo, which uses the quantization from @Manni1000 ... which is the same used by @newgenai79

This node is really experimental, this means that inputs will change, but is a good thing to play with @staoxiao model in Comfy_UI when you don't have tons of memory. It can to t2i (just text prompt) with around 6 GB and ti2i (also images) using 1024 pixels works fine on 12 GB.

I updated the readme explaining how to download the 3.88 GB version of the model and the option added by this node (not found in the original node).

An example workflow can be found in the following image:

For this workflow I used the example from this repo, so the input image of the girl is: https://github.com/VectorSpaceLab/OmniGen/blob/main/imgs/demo_cases/t2i_woman_with_book.png

Now a note for @staoxiao: The above image was generated using the model quantized on disk and quantized on VRAM. I conducted various tests using T2I inference at 512x512 with 15 steps (15 seconds of inference aprox on an RTX3060). I found that the FP8 model on disk and/or VRAM have little impact on the generated image, but huge impact on the disk space used and VRAM used. Also: loading the 3.88 GB from disk is faster than loading the 15.5 GB original model, even when the loader expands the weights and then the code compacts them again. I also found that the option use_kv_cache has much more impact in the result than the quantization, in fact I get different images, like changing the seed.

[set-soft]

Hi @staoxiao !
Here I explain why I'm not creating a PR:

1. I know almost nothing about neural networks
2. I know Python, but not PyTorch
3. I didn't write the quantization code

That said, I understand what the patch does and I can help you to integrate it. Lamentably the original patch is full of changes in blank spaces, so it looks much bigger than what it really is. Its more relevant parts are:

1. In the model.py: adds an option to apply quantification. When enabled the nn.Linear layers gets converted to int8 and the class to handle them is changed to a class that expands them in the forward computation.
2. In the pipeline.py: here the strategy is to keep in VRAM only the needed stuff, so the code moves the VAE to GPU, encodes the images and then moves the VAE to CPU, then moves the main model to GPU, does the job, and moves the main model back to CPU, then the VAE is moved again to GPU, decodes the resulting image and finally the VAE goes back to the CPU space. This takes a couple of extra seconds, which is not important when you are doing 50 steps, and solves two problems: (A) you have more memory during inference, because the VAE is not in VRAM and (B) you don't fail at the end when the decode is done, because the main model isn't in VRAM. The (B) problem is very common when you are tight in VRAM, and is very frustrating, you waited until the image is generated, but you lose it because you can't decode it (ComfyUI can catch this problem for models using a separated node for the VAE and retries using a tiled VAE approach).
3. In the app.py: it adds the option to use int8 and a class wrapper to handle the global pipe variable, so when you enable/disable the int8 option the pipe is recreated.

Applying these patches you'll reach a much broader audience. But you could also provide examples at lower resolutions. The 1024x1024 image I posted before needed around 15 minutes, but using 512x512 and 25 steps takes just a little more than 1 minute (using a 512x512 version of the input image). Of course you'll need to cherry pick more seeds ;-) When I tried the same girl and prompt, with the same seed I got:

Which is really good, but the book was also replaced, so isn't good for a show.

P.S. the above image also has the Comfy_UI workflow embedded

[set-soft]

Oh! I almost forgot, @staoxiao please fix your requirements.txt file, I guess it was generated by conda or something similar, but makes things really hard to use, you have to create a venv and all stuff must be downloaded again. This is because the file uses fixed versions like torch==2.3.1, this is not true!! your code doesn't need torch to be exactly 2.3.1. I tried it with 2.5.x and it works without problems, I think torch>=2.3.1 will do. Same for other dependencies. In the case of the Comfy_UI node I'm using:

transformers>=4.45.0
diffusers>=0.29.0
numpy<2
datasets
accelerate
timm
peft
safetensors
huggingface_hub

And it works. But the most important thing is PyTorch, lamentably the Python Packages design and management is poor and broken, so when you install a new copy of PyTorch it also installs all the CUDA libs, so this is various GBs. By relaxing the PyTorch version people can use the one it has installed, and most probably cached in their pip cache.

I created a separated venv for OmniGen, but then I integrated the Comfy_UI node to a venv where I have Comfy_UI, and I didn't need to waste disk with PyTorch 2.3.1 (which BTW is very old and doesn't even exist versions compiled for CUDA 12.x)

[herocoming777]

@set-soft I did everything you mentioned above, and still got black image. using ComfyUI_OmniGen_Wrapper and downloaded the 3.88G fp8 model files and renamed to the original one, no error report, just got black image as before.

I think it's still the same problem that the bfloat16 format or the quantization not supported by my GPU card 2080TI.
Anyway, thanks a lot for your help, I think your method still very helpful for others with lower VRAM.(NVIDIA GPU：RTX30s and RTX40s)

[set-soft]

Hi @herocoming777 !

What the console says? Are you familiarized with other tools like A1111? Do you need to disable FP16 on other tools?

I think it's still the same problem that the bfloat16 format

I don't know much about it, but I guess it should generate an error.
But is true that the code is forcing BF16.

or the quantization not supported

I doubt it, this is a very simple process, is just convert the weights into unsigned chars and then back again to floats.

You can also try this node: https://github.com/1038lab/ComfyUI-OmniGen/
@1038lab added low VRAM options yesterday, I didn't have time to test it yet.

[1038lab]

If the hardware doesn't support FP16, I've added an FP8 option in the latest update. To use it, simply update the custom node and set the model_precision to FP8. For more details on this update, please refer to the update page here:
https://github.com/1038lab/ComfyUI-OmniGen/blob/main/UPDATE.md