Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update documentation to note support for extra parameters #69

Open
bryankruman opened this issue May 13, 2024 · 1 comment
Open

Update documentation to note support for extra parameters #69

bryankruman opened this issue May 13, 2024 · 1 comment

Comments

@bryankruman
Copy link

Greetings!

I just wanted to make a quick note that the documentation for worker-vllm and RunPod both don't seem to mention anything about vLLM supporting guided generation via Json schemas or Regex/grammar patterns, but it DOES in fact support it as vLLM itself supports it.

It's a great feature and more people should consider using it for sure. In case you're not familiar, check out the vLLM docs for details about the "extra" parameters on the OpenAI completions endpoints:

https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#extra-parameters-for-chat-api

@nerdylive123
Copy link

Yeah worth documenting this on the usage examples maybe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants