Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Avoid thread starvation on many concurrent requests by making use of asyncio to lock llama_proxy context #1798

Merged
merged 7 commits into from
Dec 6, 2024
Merged

fix: Avoid thread starvation on many concurrent requests by making use of asyncio to lock llama_proxy context #1798

merged 7 commits into from
Dec 6, 2024

Commits on Nov 5, 2024

  1. fix: make use of asyncio to lock llama_proxy context

    @gjpower
    gjpower committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    222ed7c View commit details
    Browse the repository at this point in the history

  2. fix: use aclose instead of close for AsyncExitStack

    @gjpower
    gjpower committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    ab0b783 View commit details
    Browse the repository at this point in the history

  3. fix: don't call exit stack close in stream iterator as it will be cal… 

    …led by finally from on_complete anyway
    @gjpower
    gjpower committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    4da21ce View commit details
    Browse the repository at this point in the history

  4. fix: use anyio.Lock instead of asyncio.Lock

    @gjpower
    gjpower committed Nov 5, 2024
    Configuration menu
    Copy the full SHA
    9ec5460 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2024

  1. Merge branch 'main' into fix/server_llama_call_thread_starvation

    @gjpower
    gjpower authored Nov 22, 2024
    Configuration menu
    Copy the full SHA
    7bf48e3 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2024

  1. Merge branch 'main' into fix/server_llama_call_thread_starvation

    @abetlen
    abetlen authored Dec 6, 2024
    Configuration menu
    Copy the full SHA
    ad35fc1 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into fix/server_llama_call_thread_starvation

    @abetlen
    abetlen authored Dec 6, 2024
    Configuration menu
    Copy the full SHA
    9e0728b View commit details
    Browse the repository at this point in the history