Skip to content

Improve turbomind's prefix cache #3332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 38 commits into
base: main
Choose a base branch
from

Conversation

lvhan028
Copy link
Collaborator

@lvhan028 lvhan028 commented Mar 25, 2025

This Pull Request implements several improvements to the tm engine:

  • cache prompt tokens and generated tokens when prefix caching is enabled
  • remove stateful inference
  • update the chat API and CLI, get_logits and get_ppl accordingly
  • enable_prefix_caching defaults to True
  • merge pt's chat.py and tm's chat.py into one

Remaining refactoring will be processed in another PR:

  • remove get_prompt in model.py
  • deprecate the built-in chat template, and AutoTokenizer's apply_chat_template instead

test cases:

@lvhan028 lvhan028 changed the base branch from main to dev March 25, 2025 11:13
@lvhan028 lvhan028 changed the base branch from dev to main April 3, 2025 01:54
@lvhan028 lvhan028 requested review from lzhangzz and irexyc April 8, 2025 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant