Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[serve] Add documentation page about concurrency model & best practices #48902

Open
edoakes opened this issue Nov 23, 2024 · 0 comments
Open

[serve] Add documentation page about concurrency model & best practices #48902

edoakes opened this issue Nov 23, 2024 · 0 comments
Assignees
@edoakes
Labels
docs An issue or change related to documentation P1 Issue that should be fixed within a few weeks

Comments

@edoakes
Copy link
Contributor

edoakes commented Nov 23, 2024 •
edited
Loading

Follow-up to: #48897

Should cover:

  • Default behaviors.
  • max_ongoing_requests.
  • When to use async def methods (only when there is no blocking code at all). Also mention run_in_executor.
  • Different DeploymentHandle APIs. If mixing inference and remote calls, use sync method.
@edoakes edoakes added P1 Issue that should be fixed within a few weeks docs An issue or change related to documentation labels Nov 23, 2024
@edoakes edoakes self-assigned this Nov 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@edoakes edoakes

Labels
docs An issue or change related to documentation P1 Issue that should be fixed within a few weeks
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@edoakes