TGI metrics don't have model_name label to indicate which model the metrics belong to

[yashaswipiplani]

Feature request

The prometheus metrics exposed lack labels like model_name to filter or group by the model_name. This creates an issue when multiple models are deployed using TGI, causing all the metrics to get aggregated.

Motivation

vLLM runtime, for example provides model_name with LLM metrics:

vllm:time_per_output_token_seconds_bucket{le="0.01",model_name="/mnt/models"} 80.0
vllm:generation_tokens_total{model_name="/mnt/models"} 102.0
vllm:prompt_tokens_total{model_name="/mnt/models"} 231.0

Your contribution

We can submit a PR for the change requested.

[Narsil]

This is not up to a single TGI deployment to declare it's served_model_name, you should do that at the aggregation level (during aggregation on your prometheus probes start gathering information for your deployment labels for instance).

Aggregation across model, across pod, across deployment is something that should happen at the prom level, not at the TGI level.
Otherwise what will happen, is that users will ask for basically every flag to be added as aggregation limits which will make metrics overly bloated. Adding an aggregation here adds unecessary bandwidth to the /metrics route, which is already quite large.

So sorry but this is probably a wontfix until someone provides a good use case that is not solvable by the probe itself.