If CFG Uses Parallel Computation, Does Acceleration Become Impossible? #17
Description
The general idea of CFG-Cache is to use the residual of the latent conditioned and unconditioned at time t to be equivalent to the residual of the latent conditioned and unconditioned at time t-1, thus solving for the residual of the unconditioned latent at time t-1, while separately calculating high-frequency and low-frequency information. However, in most algorithms using CFG, conditioned and unconditioned latents are processed in parallel, meaning they are concatenated together in the batch for computation.
My question is, if there is enough GPU memory and parallel computation is used, does the acceleration from CFG-Cache essentially disappear? Is this acceleration only effective when conditioned and unconditioned are processed sequentially? If parallel computation can still accelerate, could you explain why this is the case?