Can it support segmented processing of long audio or audio streaming?

[mqgw]

Compared to generating results for a long audio all at once, segmented processing or splitting long audio into a stream of short audio clips can reduce the time to first frame of video, decreasing waiting time.

[set-soft]

Or perhaps some kind of "tiled decode" (in reference to tiled VAE decode used for images).
The memory is consumed by:

data_out = self.decode_latent_into_image(s_r = s_r, s_r_feats = s_r_feats, r_d = sample)

This is why I mention decoding by segments

[set-soft]

OK, I asked some help to Gemini Pro and we refactored decode_latent_into_image and save_image.

With this I was able to generate a 4 minutes long video using less than 6 GiB of VRAM and less than 8 GiB of RAM (in a machine with 12 GiB of VRAM and 32 GiB of RAM).

The trick is to avoid:

1. Have the full video in VRAM, I'm creating it on RAM
2. Create the whole video to then adapt it to torchvision images, I'm doing it on-the-fly

These two things drastically reduces the amount of VRAM and RAM needed.

[set-soft]

For those interested on testing the optimizations here is my fork: https://github.com/set-soft/float

[nitinmukesh]

@set-soft

Thank you.

I made this change

def decode_latent_into_image

            img_t_cpu = img_t.cpu()
            d_hat.append(img_t_cpu)

[set-soft]

Hi @nitinmukesh !
This will take you a little farther, but you'll need insane amounts of RAM. My fork does much more than this.
I started doing this, but soon found I needed 60 GB of RAM for the 4 minutes video, and as soon as the code tried to convert the images to frames to create the video I ran out of memory.
The code in my fork avoids such a bad memory management by converting the images on-the-fly.

[nitinmukesh]

I noticed it. For shorter audio it works, but for 4 min audio it need insane amount of RAM. I will use your fork.
Thank you for sharing.

[adkkml]

For those interested on testing the optimizations here is my fork: https://github.com/set-soft/float

I have tried to instal this in my ComfyUI Portable but IIMPORT FAILED, appreciated for any help, please

[set-soft]

For those interested on testing the optimizations here is my fork: https://github.com/set-soft/float

I have tried to instal this in my ComfyUI Portable but IIMPORT FAILED, appreciated for any help, please

For a ComfyUI node you need this repo: https://github.com/set-soft/ComfyUI-FLOAT/
The https://github.com/deepbrainai-research/float/ is the original implementation of the paper, CLI Python app, not ComfyUI nodes.

[adkkml]

For those interested on testing the optimizations here is my fork: https://github.com/set-soft/float

I have tried to instal this in my ComfyUI Portable but IIMPORT FAILED, appreciated for any help, please

For a ComfyUI node you need this repo: https://github.com/set-soft/ComfyUI-FLOAT/ The https://github.com/deepbrainai-research/float/ is the original implementation of the paper, CLI Python app, not ComfyUI nodes.

Using the set-soft repo, it wouldn't let me create a long video (OOM on GTX3090 24GB VRAM). If using your FLOAT.py (with edited some line in the top part to be the same as set-soft), it allow to generate long FLOAT video. Thank You. But sometimes, when saving the video the RAM would hit 100% which makes the computer "hang" a bit. If try using your generate.py it ended with error as follows:

File "G:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-FLOAT\generate.py", line 188, in save_video_frames
os.makedirs(os.path.dirname(video_path) or '.', exist_ok=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 251, in dirname
File "", line 213, in split
TypeError: expected str, bytes or os.PathLike object, not NoneType

[set-soft]

Hi @adkkml !

1. Use the full code from https://github.com/set-soft/ComfyUI-FLOAT/ update it, is evolving. Use git pull. And avoid mixing it.
2. If the above mentioned code fails please post the full traceback in an issue for https://github.com/set-soft/ComfyUI-FLOAT/ do not ask for help here, this will confuse users. I can generate more than 5 minutes of video with 12 GB of VRAM and 32 GB of RAM (+ 32 GB of swap, virtually 64 GB).
3. If you want a command line version of the code try: https://github.com/set-soft/float

Both are forks that I modified to allow long videos using an RTX3060

P.S. The command line version needs less memory than the ComfyUI nodes.
P.S.2. Don't confuse the https://github.com/yuvraj108c/ComfyUI-FLOAT nodes, these use almost the same code found in this repo

[adkkml]

Hi @adkkml !

1. Use the full code from https://github.com/set-soft/ComfyUI-FLOAT/ update it, is evolving. Use git pull. And avoid mixing it.
2. If the above mentioned code fails please post the full traceback in an issue for https://github.com/set-soft/ComfyUI-FLOAT/ do not ask for help here, this will confuse users. I can generate more than 5 minutes of video with 12 GB of VRAM and 32 GB of RAM (+ 32 GB of swap, virtually 64 GB).
3. If you want a command line version of the code try: https://github.com/set-soft/float

Both are forks that I modified to allow long videos using an RTX3060

P.S. The command line version needs less memory than the ComfyUI nodes. P.S.2. Don't confuse the https://github.com/yuvraj108c/ComfyUI-FLOAT nodes, these use almost the same code found in this repo

Thank you for the guidance. I tried set-soft/ComfyUI-FLOAT, got OOM (3.09 minutes audio). I will post the error in the set-soft/ComfyUI-FLOAT.