Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong metric type for minio_s3_time_ttfb_seconds_distribution? #17060

Open
MalKeshar opened this issue Apr 21, 2023 · 2 comments · Fixed by #17115
Open

Wrong metric type for minio_s3_time_ttfb_seconds_distribution? #17060

MalKeshar opened this issue Apr 21, 2023 · 2 comments · Fixed by #17115

Comments

@MalKeshar
Copy link

MalKeshar commented Apr 21, 2023

Problem: it is not possible to use histogram_quantile function on minio_s3_time_ttfb_seconds_distribution

Assumption: minio probably use wrong metric type for request duration.

  1. According prometheus documentation for distribution should be used summary and histogram metrics type
    https://prometheus.io/docs/practices/histograms/

  2. According https://github.com/minio/minio/blob/master/cmd/metrics-v2.go
    request duration is the only minio distribution metric, which use gauge.

Other distributions use Histogram.

sizeDistribution = "size_distribution"
versionDistribution = "version_distribution"
ttfbDistribution = "ttfb_seconds_distribution"

...
Name: sizeDistribution,
Help: "Distribution of object sizes in the bucket, includes label for the bucket name",
Type: histogramMetric,

Name: versionDistribution,
Help: "Distribution of object sizes in the bucket, includes label for the bucket name",
Type: histogramMetric,

Name: ttfbDistribution,
Help: "Distribution of the time to first byte across API calls",
Type: gaugeMetric,

Name: latencyMilliSec,
Help: "Replication latency in milliseconds",
Type: histogramMetric,
  1. Even data of this metric looks in tsdb as cumulative counter, not a gauge, it always goes up.

I search for answer in github issues, but didn't find it.

It was previously asked in
#14293
#12540
but there was no answer

By any chance is it possible that it's some kind of mistake? Is there any information for such decision?
Thank you.

@MalKeshar MalKeshar changed the title Wrong metric type for request duration? Wrong metric type for minio_s3_time_ttfb_seconds_distribution? Apr 21, 2023
harshavardhana added a commit to harshavardhana/minio that referenced this issue May 3, 2023
@harshavardhana harshavardhana reopened this Jul 7, 2023
@harshavardhana
Copy link
Member

This issue was fixed incorrectly, this requires an overhaul in the way we calculate. We had to move back to gaugeMetric for now.

@stale
Copy link

stale bot commented Aug 12, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 15 days if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants