My attempt of fitting Kandinsky-3 into roughly ~8.5GB VRAM for most part

[SLAPaper]

https://github.com/SLAPaper/Kandinsky-3/blob/lowvram/examples/kandinsky3.ipynb

Using techniques:

1. Split the model into 3 parts: Text Encoder/ UNet/ MOVQ, gc others before loading another into memory
2. Text Encoder: load in 4bits + only load encoder part (this is ~8.5GB VRAM)
3. UNet: load in float8_e4m3 + autocast (this is ~5.1GB VRAM)
4. MOVQ: autocast (this is ~11GB, but for very short time and little computation so using some system RAM is acceptable)

Possible further improvement:

1. improve MOVQ by tiling? or also using float8_e4m3 format? or implement sequencial offload?
2. offload to CPU rather than delete and gc to save disk IO time
3. implement manual cast method to support other devices that don't support autocast

Reference documents:

1. T5 in 4bits: https://huggingface.co/docs/transformers/model_doc/t5
2. UNet in fp8: A big improvement for dtype casting system with fp8 storage type and manual cast AUTOMATIC1111/stable-diffusion-webui#14031
3. Sequential gc: https://github.com/PixArt-alpha/PixArt-alpha/blob/master/asset/docs/pixart.md

[seruva19]

That's very interesting, thank you for sharing! I will study your work for sure.