Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add code for skipping CFG on early steps #15607

Open
wants to merge 7 commits into
base: dev
Choose a base branch
from

Conversation

drhead
Copy link
Contributor

@drhead drhead commented Apr 23, 2024

Description

  • Implements an option to skip CFG on early sampling steps as recommended in https://arxiv.org/abs/2404.07724
  • Somewhat awkward to use as it stands. I'll probably change it to floating point for a percentage of sampling steps rather than discrete steps. done. Would recommend anyone that uses it uses it as a quicksetting.
  • Probably shouldn't be used for img2img or inpainting (unless using almost 1.0 denoising strength because the assumptions of the paper will almost certainly not hold for those cases (haven't tested this though).
  • Improves sample diversity, image quality, and in some cases converges faster. Also often makes higher CFG settings stable. Also, makes the skipped steps twice as fast since the second sample from CFG isn't being made.
  • In some cases, may reduce prompt obedience. Particularly, this will also take away your negative prompt on early steps.
  • Paper also recommends something similar to Negative Guidance Minimum Sigma, except which skips all steps in that interval instead of every other step. I'll look into adding an option to make NGMS skip all steps towards this end. done, this PR now fully implements the paper. Between the two options, I find I can often get away with skipping CFG on around 1/3 of total sampling steps.

Screenshots/videos:

25 steps, no skipped CFG steps:
grid-23147
25 steps, skipped CFG on first four steps:
grid-23146
50 steps, no CFG skipping:
grid-23148
50 steps, skipped CFG on first eight steps:
grid-23149

As you can see, the image with CFG skipping converged much faster. There are some noticeable side effects on image brightness (prompt on this image contained (dark,:1.3), so this is somewhat less dark than I would typically expect).

Checklist:

@drhead drhead changed the base branch from master to dev April 23, 2024 04:07
@drhead drhead mentioned this pull request Apr 23, 2024
4 tasks
@v0xie
Copy link
Contributor

v0xie commented May 1, 2024

Awesome PR!

Skip Early Cond should be written in metadata for reproducibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants