-
Notifications
You must be signed in to change notification settings - Fork 900
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add stop_regex
parameter to /generate
#1857
Comments
Apologies, I didn't realise this was a duplicate of #1130. I see the suggested approach is to use streaming and drop the connection. Nonetheless, is it something that you would still consider? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Feature request
Add an optional
stop_regex
parameter toGenerateParameters
. Stop generating tokens ifstop_regex
matches the generated text.Motivation
Many applications such as inline code completion require fine-grained control over generated text. For example, you may want to generate a full "block" of code in which case you may desire to stop generation once a "dedent" is matched. Another example is generating till a matching a closing brace, quote or parenthesis is generated by the model. This is sometimes hard to express using TGI's "stop sequences" feature.
Your contribution
I have a working small demo, I'm open to making a PR if needed.
A few notes:
re
package can sometimes be very slow, especially in the presence of adversarial patterns. I think the best approach to this is to disable the feature by default (similar to how grammars work) and/or use a more restrictive Regex engine.The text was updated successfully, but these errors were encountered: