Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provide Text generation usage metrics #3461

Open
johnugeorge opened this issue Feb 20, 2024 · 1 comment · May be fixed by #3547
Open

Provide Text generation usage metrics #3461

johnugeorge opened this issue Feb 20, 2024 · 1 comment · May be fixed by #3547
Assignees
@sivanantha321
Labels
kserve/llm

Comments

@johnugeorge
Copy link
Contributor

johnugeorge commented Feb 20, 2024 •
edited

With huggignface runtime, we want to bring support for usage metrics like tokens (completion tokens + prompt tokens) Ref: usage field in OpenAI response https://platform.openai.com/docs/api-reference/making-requests This can be used by the client for calculating throughput(tokens/sec) etc

Validate: In streaming mode, can we get TTFT (Time to first token)?
@sivanantha321
Copy link
Member

/assign

@sivanantha321 sivanantha321 linked a pull request Mar 25, 2024 that will close this issue
Draft
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sivanantha321 sivanantha321

Labels
kserve/llm
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Expose Initial LLM metrics sivanantha321/kserve
3 participants
@johnugeorge @yuzisun @sivanantha321