27 Jun 16:09

ace7d6a

2.13.0-rc.1 Pre-release

Pre-release

This release contains 476 PRs from 67 authors, including new contributors Anthony Keydel, Armand Grillet, AvivGuiser, Daniel R. Dagfinrud, David Collier-Brown, David Grant, Dimitris Alo, Enrique Garbi, Erik Sommer, Ian Halliday, InventiveCoder, Kevin Mingtarja, Lasse Hels, Pangidoan Butar, Quentin Bisson, Rens Groothuijsen, René Gärtner, Ross Brunson, Santiago, Seiya, Spyros Panagiotopoulos, Yuri Nikolic, Zied ABID, kahirokunn, lasermoth. Thank you!

Grafana Mimir version 2.13.0-rc.0 release notes

Features and enhancements

Improved CPU performance in processing queries with regular expressions that contain many alternations (for example foo|bar|baz|...).
Improved OTLP ingestion performance by using native translation, which reduces memory usage in the distributor by up to 30% and CPU usage by up to 8%.
Configurable S3 bucket lookup type improves interoperability with S3-compatible providers, such as Tencent and Alibaba. Configure lookup type via -<prefix>.s3.bucket-lookup-type or -common.storage.s3-bucket-lookup-type.
Configuration of TLS for S3 buckets is now possible via -<prefix>.s3.http.* and -common.storage.s3.http.* configuration options.
Mimirtool can now verify the validity of a Mimir runtime configuration file with the mimirtool runtime-config verify command.
Active series are now updated along with owned series. This means the number of active series for a tenant is more accurate after scaling out ingesters. As a result, ingesters now more precisely enforce tenants' series limits. This feature is only enabled when -ingester.track-ingester-owned-series or -ingester.use-ingester-owned-series-for-limits are enabled.
Store-gateway can be configured to explicitly disable or enable tenants. This is useful in cases where compaction slows down for a tenant to temporarily exclude another tenant while compaction catches up without affecting other tenants.
Remote read (/prometheus/api/v1/read) is becoming a first-class endpoint in Mimir with support in query stats logs and query blocking in the query-frontend. More coming in future releases.

Additionally, the following previously experimental features are now considered stable:

Rules tenant federation via the source_tenants field in rule groups.
Enabling recording, and alerting rules evaluation on a per-tenant basis.
Limiting the number of tenants a federated query can query.

Important changes

In Grafana Mimir 2.13 the following behavior has changed:

The default Docker image grafana/mimir is now based on the distroless image gcr.io/distroless/static-debian12.
See Debugging distroless container images for more details on how to
work with distroless images.
Error logs in the ingester are now sampled at 10% by default. Sampled log lines contain (sampled 1/10). The cortex_discarded_samples_total metric still tracks all discarded samples.
Anonymous usage statistics now include actual CPU usage instead of available CPU cores.
Continuous-test is no longer a standalone binary and is now part of Mimir as its own target. The published Docker image has been updated to use the new packaging.

The following deprecated configuration options are removed in Grafana Mimir 2.13:

The configuration option -log.buffered, which was deprecated in 2.11 and is now enabled by default.

Experimental features

Grafana Mimir 2.13 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

Experimental support for server-side circuit breakers in ingesters on read and write requests. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options and further configured via the -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options.

The following configuration options are deprecated and will be removed in a future Grafana Mimir release:

evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead.

Bug fixes

OTLP ingestion: translate all HTTP 5xx errors into one of 502, 503, or 504, so that the otel-collector retries the failed requests.
OTLP ingestion: return properly formatted protobuf error messages when ingestion fails.
OTLP ingestion: generate target_info metric only when there are metrics in the request and there is at least one configured identifying label.
OTLP ingestion: don't discard timeseries paired with invalid exemplars. Instead, try to ingest the timeseries and discard only the invalid exemplars.
Subqueries: @ end() and @ start() now work correctly with queries split by time.
Native histograms: order exemplars before ingestion to improve success rate when ingesting multiple exemplars.
Native histograms: return HTTP 400 on invalid native histogram samples instead of HTTP 500. The metric cortex_discarded_samples_total{reason="invalid-native-histogram"} is now incremented on invalid histogram samples.

Changelog

2.13.0-rc.1

Grafana Mimir

[CHANGE] Build: grafana/mimir docker image is now based on gcr.io/distroless/static-debian12 image. Alpine-based docker image is still available as grafana/mimir-alpine, until Mimir 2.15. #8204 #8235
[CHANGE] Ingester: /ingester/flush endpoint is now only allowed to execute only while the ingester is in Running state. The 503 status code is returned if the endpoint is called while the ingester is not in Running state. #7486
[CHANGE] Distributor: Include label name in err-mimir-label-value-too-long error message: #7740
[CHANGE] Ingester: enabled 1 out 10 errors log sampling by default. All the discarded samples will still be tracked by the cortex_discarded_samples_total metric. The feature can be configured via -ingester.error-sample-rate (0 to log all errors). #7807
[CHANGE] Query-frontend: Query results caching and experimental query blocking now utilize the PromQL string-formatted query format rather than the unvalidated query as submitted to the frontend. #7742
- Query results caching should be more stable as all equivalent queries receive the same cache key, but there may be cache churn on first deploy with the updated format
- Query blocking can no longer be circumvented with an equivalent query in a different format; see Configure queries to block
[CHANGE] Query-frontend: stop using -validation.create-grace-period to clamp how far into the future a query can span. #8075
[CHANGE] Clamp GOMAXPROCS to runtime.NumCPU. #8201
[CHANGE] Anonymous usage statistics tracking: add CPU usage percentage tracking. #8282
[CHANGE] Added new metric cortex_compactor_disk_out_of_space_errors_total which counts how many times a compaction failed due to the compactor being out of disk. #8237
[CHANGE] Anonymous usage statistics tracking: report active series in addition to in-memory series. #8279
[CHANGE] Ruler: evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead (it has the same exact meaning and behaviour). #8295
[CHANGE] General: remove -log.buffered. The configuration option has been enabled by default and deprecated since Mimir 2.11. #8395
[CHANGE] Ruler: promote tenant federation from experimental to stable. #8400
[CHANGE] Ruler: promote -ruler.recording-rules-evaluation-enabled and -ruler.alerting-rules-evaluation-enabled from experimental to stable. #8400
[CHANGE] General: promote -tenant-federation.max-tenants from experimental to stable. #8400
[FEATURE] Continuous-test: now runable as a module with mimir -target=continuous-test. #7747
[FEATURE] Store-gateway: Allow specific tenants to be enabled or disabled via -store-gateway.enabled-tenants or -store-gateway.disabled-tenants CLI flags or their corresponding YAML settings. #7653
[FEATURE] New -<prefix>.s3.bucket-lookup-type flag configures lookup style type, used to access bucket in s3 compatible providers. #7684
[FEATURE] Querier: add experimental streaming PromQL engine, enabled with -querier.promql-engine=mimir. #7693 #7898 #7899 #8023 #8058 #8096 #8121 #8197 #8230 #8247 #8270 #8276 #8277 #8291 #8303 #8340 #8256 #8348
[FEATURE] New /ingester/unregister-on-shutdown HTTP endpoint allows dynamic access to ingesters' -ingester.ring.unregister-on-shutdown configuration. #7739
[FEATURE] Server: added experimental PROXY protocol support. The PROXY protocol support can be enabled via -server.proxy-protocol-enabled=true. When enabled, the support is added both to HTTP and gRPC listening ports. #7698
[FEATURE] Query-frontend, querier: new experimental /cardinality/active_native_histogram_metrics API to get active native histogram metric names with statistics about active native histogram buckets. #7982 #7986 #8008
[FEATURE] Alertmanager: Added -alertmanager.max-silences-count and -alertmanager.max-silence-size-bytes to set limits on per tenant silences. Disabled by default. #8241 #8249
[FEATURE] Ingester: add experimental support for the server-side circuit breakers when writing to and reading from ingesters. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options. Further -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options for configuring circuit-breaker are available. Added metrics cortex_ingester_circuit_breaker_results_total, `cortex_i...

Assets 118

21 Jun 18:20

dimitarvdimitrov

mimir-2.13.0-rc.0

ac21f29

2.13.0-rc.0 Pre-release

Pre-release

Grafana Mimir version 2.13.0-rc.0 release notes

Features and enhancements

Improved CPU performance in processing queries with regular expressions that contain many alternations (for example foo|bar|baz|...).
Improved OTLP ingestion performance by using native translation, which reduces memory usage in the distributor by up to 30% and CPU usage by up to 8%.
Configurable S3 bucket lookup type improves interoperability with S3-compatible providers, such as Tencent and Alibaba. Configure lookup type via -<prefix>.s3.bucket-lookup-type or -common.storage.s3-bucket-lookup-type.
Configuration of TLS for S3 buckets is now possible via -<prefix>.s3.http.* and -common.storage.s3.http.* configuration options.
Mimirtool can now verify the validity of a Mimir runtime configuration file with the mimirtool runtime-config verify command.
Active series are now updated along with owned series. This means the number of active series for a tenant is more accurate after scaling out ingesters. As a result, ingesters now more precisely enforce tenants' series limits. This feature is only enabled when -ingester.track-ingester-owned-series or -ingester.use-ingester-owned-series-for-limits are enabled.
Store-gateway can be configured to explicitly disable or enable tenants. This is useful in cases where compaction slows down for a tenant to temporarily exclude another tenant while compaction catches up without affecting other tenants.
Remote read (/prometheus/api/v1/read) is becoming a first-class endpoint in Mimir with support in query stats logs and query blocking in the query-frontend. More coming in future releases.

Additionally, the following previously experimental features are now considered stable:

Rules tenant federation via the source_tenants field in rule groups.
Enabling recording, and alerting rules evaluation on a per-tenant basis.
Limiting the number of tenants a federated query can query.

Important changes

In Grafana Mimir 2.13 the following behavior has changed:

The default Docker image grafana/mimir is now based on the distroless image gcr.io/distroless/static-debian12.
See Debugging distroless container images for more details on how to
work with distroless images.
Error logs in the ingester are now sampled at 10% by default. Sampled log lines contain (sampled 1/10). The cortex_discarded_samples_total metric still tracks all discarded samples.
Anonymous usage statistics now include actual CPU usage instead of available CPU cores.
Continuous-test is no longer a standalone binary and is now part of Mimir as its own target. The published Docker image has been updated to use the new packaging.

The following deprecated configuration options are removed in Grafana Mimir 2.13:

The configuration option -log.buffered, which was deprecated in 2.11 and is now enabled by default.

Experimental features

Grafana Mimir 2.13 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

Experimental support for server-side circuit breakers in ingesters on read and write requests. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options and further configured via the -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options.

The following configuration options are deprecated and will be removed in a future Grafana Mimir release:

evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead.

Bug fixes

OTLP ingestion: translate all HTTP 5xx errors into one of 502, 503, or 504, so that the otel-collector retries the failed requests.
OTLP ingestion: return properly formatted protobuf error messages when ingestion fails.
OTLP ingestion: generate target_info metric only when there are metrics in the request and there is at least one configured identifying label.
OTLP ingestion: don't discard timeseries paired with invalid exemplars. Instead, try to ingest the timeseries and discard only the invalid exemplars.
Subqueries: @ end() and @ start() now work correctly with queries split by time.
Native histograms: order exemplars before ingestion to improve success rate when ingesting multiple exemplars.
Native histograms: return HTTP 400 on invalid native histogram samples instead of HTTP 500. The metric cortex_discarded_samples_total{reason="invalid-native-histogram"} is now incremented on invalid histogram samples.

Changelog

2.13.0-rc.0

Grafana Mimir

[CHANGE] Build: grafana/mimir docker image is now based on gcr.io/distroless/static-debian12 image. Alpine-based docker image is still available as grafana/mimir-alpine, until Mimir 2.15. #8204 #8235
[CHANGE] Ingester: /ingester/flush endpoint is now only allowed to execute only while the ingester is in Running state. The 503 status code is returned if the endpoint is called while the ingester is not in Running state. #7486
[CHANGE] Distributor: Include label name in err-mimir-label-value-too-long error message: #7740
[CHANGE] Ingester: enabled 1 out 10 errors log sampling by default. All the discarded samples will still be tracked by the cortex_discarded_samples_total metric. The feature can be configured via -ingester.error-sample-rate (0 to log all errors). #7807
[CHANGE] Query-frontend: Query results caching and experimental query blocking now utilize the PromQL string-formatted query format rather than the unvalidated query as submitted to the frontend. #7742
- Query results caching should be more stable as all equivalent queries receive the same cache key, but there may be cache churn on first deploy with the updated format
- Query blocking can no longer be circumvented with an equivalent query in a different format; see Configure queries to block
[CHANGE] Query-frontend: stop using -validation.create-grace-period to clamp how far into the future a query can span. #8075
[CHANGE] Clamp GOMAXPROCS to runtime.NumCPU. #8201
[CHANGE] Anonymous usage statistics tracking: add CPU usage percentage tracking. #8282
[CHANGE] Added new metric cortex_compactor_disk_out_of_space_errors_total which counts how many times a compaction failed due to the compactor being out of disk. #8237
[CHANGE] Anonymous usage statistics tracking: report active series in addition to in-memory series. #8279
[CHANGE] Ruler: evaluation_delay field in the rule group configuration has been deprecated. Please use query_offset instead (it has the same exact meaning and behaviour). #8295
[CHANGE] General: remove -log.buffered. The configuration option has been enabled by default and deprecated since Mimir 2.11. #8395
[CHANGE] Ruler: promote tenant federation from experimental to stable. #8400
[CHANGE] Ruler: promote -ruler.recording-rules-evaluation-enabled and -ruler.alerting-rules-evaluation-enabled from experimental to stable. #8400
[CHANGE] General: promote -tenant-federation.max-tenants from experimental to stable. #8400
[FEATURE] Continuous-test: now runable as a module with mimir -target=continuous-test. #7747
[FEATURE] Store-gateway: Allow specific tenants to be enabled or disabled via -store-gateway.enabled-tenants or -store-gateway.disabled-tenants CLI flags or their corresponding YAML settings. #7653
[FEATURE] New -<prefix>.s3.bucket-lookup-type flag configures lookup style type, used to access bucket in s3 compatible providers. #7684
[FEATURE] Querier: add experimental streaming PromQL engine, enabled with -querier.promql-engine=mimir. #7693 #7898 #7899 #8023 #8058 #8096 #8121 #8197 #8230 #8247 #8270 #8276 #8277 #8291 #8303 #8340 #8256 #8348
[FEATURE] New /ingester/unregister-on-shutdown HTTP endpoint allows dynamic access to ingesters' -ingester.ring.unregister-on-shutdown configuration. #7739
[FEATURE] Server: added experimental PROXY protocol support. The PROXY protocol support can be enabled via -server.proxy-protocol-enabled=true. When enabled, the support is added both to HTTP and gRPC listening ports. #7698
[FEATURE] Query-frontend, querier: new experimental /cardinality/active_native_histogram_metrics API to get active native histogram metric names with statistics about active native histogram buckets. #7982 #7986 #8008
[FEATURE] Alertmanager: Added -alertmanager.max-silences-count and -alertmanager.max-silence-size-bytes to set limits on per tenant silences. Disabled by default. #8241 #8249
[FEATURE] Ingester: add experimental support for the server-side circuit breakers when writing to and reading from ingesters. This can be enabled using -ingester.push-circuit-breaker.enabled and -ingester.read-circuit-breaker.enabled options. Further -ingester.push-circuit-breaker.* and -ingester.read-circuit-breaker.* options for configuring circuit-breaker are available. Added metrics cortex_ingester_circuit_breaker_results_total, `cortex...

Assets 118

03 Apr 22:18

pr00se

mimir-2.12.0

c7aab9e

2.12.0

This release contains 531 PRs from 60 authors, including new contributors Benoit Schipper, Derek Cadzow, Edwin, Itay Kalfon, Ivan Farré Vicente, Jan O. Rundshagen, Jorge Turrado Ferrero, Lukas Monkevicius, Mickaël Canévet, Rafael Sathler, Rajakavitha Kodhandapani, Tim Kotowski, Vladimir Varankin, Zach, Zach Day, Zirko, blut, github-actions[bot], ncharaf, zhehao-grafana. Thank you!

Grafana Mimir version 2.12.0 release notes

Grafana Labs is excited to announce version 2.12 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

Added support to only count series that are considered active through the Cardinality API endpoint /api/v1/cardinality/label_names by passing the count_method parameter.
If set to active it counts only series that are considered active according to the -ingester.active-series-metrics-idle-timeout flag setting rather than counting all in-memory series.
The "Store-gateway: bucket tenant blocks" admin page contains a new column "No Compact".
If block no compaction marker is set, it specifies the reason and the date the marker is added.
The estimated number of compaction jobs based on the current bucket-index is now computed by the compactor.
The result is tracked by the new cortex_bucket_index_compaction_jobs metric.
If this computation fails, the cortex_bucket_index_compaction_jobs_errors_total metric is updated instead.
The estimated number of compaction jobs is also shown in Top tenants, Tenants, and Compactor dashboards.
Added mimir-distroless container image built upon a distroless image (gcr.io/distroless/static-debian12).
This improvement minimizes attack surfaces and potential CVEs by trimming down the dependencies within the image.
After comprehensive testing, the Mimir maintainers plan to shift from the current image to the distroless version.

Additionally, the following previously experimental features are now considered stable:

The number of pre-allocated workers used to forward push requests to the ingesters, configurable via the -distributor.reusable-ingester-push-workers CLI flag on distributors.
It now defaults to 2000.
Note that this is a performance optimization, and not a limiting feature.
If not enough workers available, new goroutines will be spawned.
The number of gRPC server workers used to serve the requests, configurable via the -server.grpc.num-workers CLI flag.
It now defaults to 100.
Note that this is the number of pre-allocated long-lived workers, and not a limiting feature.
If not enough workers are available, new goroutines will be spawned.
The maximum number of concurrent index header loads across all tenants, configurable via the -blocks-storage.bucket-store.index-header.lazy-loading-concurrency CLI flag on store-gateways.
It defaults to 4.
The maximum time to wait for the query-frontend to become ready before rejecting requests, configurable via the -query-frontend.not-running-timeout CLI flag on query-frontends.
It now defaults to 2s.
The CLI flag that allows queriers to reduce pressure on ingesters by initially querying only the minimum set of ingesters required to reach quorum, -querier.minimize-ingester-requests.
It is now enabled by default.
Spread-minimizing token-related CLI flags: -ingester.ring.token-generation-strategy, -ingester.ring.spread-minimizing-zones and -ingester.ring.spread-minimizing-join-ring-in-order.
You can read more about this feature in our blog post.

Important changes

In Grafana Mimir 2.12 the following behavior has changed:

Store-gateway now persists a sparse version of the index-header to disk on construction and loads sparse index-headers from disk instead of the whole index-header.
This improves the speed at which index headers are lazy-loaded from disk by up to 90%. The added disk usage is in the order of 1-2%.
Alertmanager deprecated the v1 API. All v1 API endpoints now respond with a JSON deprecation notice and a status code of 410.
All endpoints have a v2 equivalent.
The list of endpoints is:
- <alertmanager-web.external-url>/api/v1/alerts
- <alertmanager-web.external-url>/api/v1/receivers
- <alertmanager-web.external-url>/api/v1/silence/{id}
- <alertmanager-web.external-url>/api/v1/silences
- <alertmanager-web.external-url>/api/v1/status
Exemplar's label traceID has been changed to trace_id to be consistent with the OpenTelemetry standard.
Errors returned by ingesters now contain only gRPC status codes.
Previously they contained both gRPC and HTTP status codes.

{{< admonition type="warning" >}}
To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.
{{< /admonition >}}
Responses with gRPC status codes are now reported as status_code labels in the cortex_request_duration_seconds and cortex_ingester_client_request_duration_seconds metrics.
Responses with HTTP 4xx status codes are now treated as errors and used in status_code label of request duration metric.

The default value of the following CLI flags have been changed:

-blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
-blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
-blocks-storage.bucket-store.tenant-sync-concurrency from 10 to 1.
-query-frontend.max-cache-freshness from 1m to 10m.
-distributor.write-requests-buffer-pooling-enabled from false to true.
-locks-storage.bucket-store.block-sync-concurrency from 20 to 4.
-memberlist.stream-timeout from 10s to 2s.
-server.report-grpc-codes-in-instrumentation-label-enabled from false to true.

The following deprecated configuration options are removed in Grafana Mimir 2.12:

The YAML setting frontend.cache_unaligned_requests.
Experimental CLI flag -querier.prefer-streaming-chunks-from-ingesters.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.14:

The CLI flag -ingester.limit-inflight-requests-using-grpc-method-limiter.
It now defaults to true.
The CLI flag -ingester.return-only-grpc-errors.
It now defaults to true.

{{< admonition type="warning" >}}
To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.
{{< /admonition >}}
The CLI flag -ingester.client.report-grpc-codes-in-instrumentation-label-enabled.
It now defaults to true.
The CLI flag -distributor.limit-inflight-requests-using-grpc-method-limiter.
It now defaults to true.
The CLI flag -distributor.enable-otlp-metadata-storage.
It now defaults to true.
The CLI flag -querier.max-query-into-future.

The following metrics are removed or deprecated:

cortex_bucket_store_blocks_loaded_by_duration has been removed.
cortex_distributor_sample_delay_seconds has been deprecated and will be removed in Mimir 2.14.

Experimental features

Grafana Mimir 2.12 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

The maximum number of tenant IDs that may be for a federated query can be configured via the -tenant-federation.max-tenants CLI flag on query-frontends.
By default, it's 0, meaning that the limit is disabled.
Sharding of active series queries can be enabled via the -query-frontend.shard-active-series-queries CLI flag on query-frontends.
Timely head compaction can be enabled via the -blocks-storage.tsdb.timely-head-compaction-enabled on ingesters.
If enabled, the head compaction happens when the min block range can no longer be appended, without requiring 1.5x the chunk range worth of data in the head.
Streaming of responses from querier to query-frontend can be enabled via the -querier.response-streaming-enabled CLI flag on queriers.
This is currently supported only for responses from the /api/v1/cardinality/active_series endpoint.
The maximum response size for active series queries, in bytes, can be set via the -querier.active-series-results-max-size-bytes CLI flag on queriers.
Metric relabeling on a per-tenant basis can be forcefully disabled via the -distributor.metric-relabeling-enabled CLI flag on rulers.
Metrics relabeling is enabled by default.
Query Queue Load Balancing by Query Component. Tenant query queues in the query-scheduler can now be split into subqueues by which query component is expected to be utilized to complete the query: ingesters, store-gateways, both, or uncategorized.
Dequeuing queries for a given tenant will rotate through the query component subqueues via simple round-robin.
In the event that the one of the query components (ingesters or store-gateways) experience a slowdown, queries only utilizing the other query component can continue to be serviced.
This feature is r...

Contributors

narqo, replay, and 56 other contributors

Assets 118

20 Mar 12:21

duricanikolic

mimir-2.12.0-rc.1

6c30057

2.12.0-rc.1 Pre-release

Pre-release

This release contains 525 PRs from 60 authors, including new contributors Benoit Schipper, Derek Cadzow, Edwin, Itay Kalfon, Ivan Farré Vicente, Jan O. Rundshagen, Jorge Turrado Ferrero, Lukas Monkevicius, Mickaël Canévet, Rafael Sathler, Rajakavitha Kodhandapani, Tim Kotowski, Vladimir Varankin, Zach, Zach Day, Zirko, blut, github-actions[bot], ncharaf, zhehao-grafana. Thank you!

Grafana Mimir version 2.12.0-rc.1 release notes

Grafana Labs is excited to announce version 2.12 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

Added support to only count series that are considered active through the Cardinality API endpoint /api/v1/cardinality/label_names by passing the count_method parameter.
If set to active it counts only series that are considered active according to the -ingester.active-series-metrics-idle-timeout flag setting rather than counting all in-memory series.
The "Store-gateway: bucket tenant blocks" admin page contains a new column "No Compact".
If block no compaction marker is set, it specifies the reason and the date the marker is added.
The estimated number of compaction jobs based on the current bucket-index is now computed by the compactor.
The result is tracked by the new cortex_bucket_index_compaction_jobs metric.
If this computation fails, the cortex_bucket_index_compaction_jobs_errors_total metric is updated instead.
The estimated number of compaction jobs is also shown in Top tenants, Tenants, and Compactor dashboards.
Added mimir-distroless container image built upon a distroless image (gcr.io/distroless/static-debian12).
This improvement minimizes attack surfaces and potential CVEs by trimming down the dependencies within the image.
After comprehensive testing, the Mimir maintainers plan to shift from the current image to the distroless version.

Additionally, the following previously experimental features are now considered stable:

The number of pre-allocated workers used to forward push requests to the ingesters, configurable via the -distributor.reusable-ingester-push-workers CLI flag on distributors.
It now defaults to 2000.
Note that this is a performance optimization, and not a limiting feature.
If not enough workers available, new goroutines will be spawned.
The number of gRPC server workers used to serve the requests, configurable via the -server.grpc.num-workers CLI flag.
It now defaults to 100.
Note that this is the number of pre-allocated long-lived workers, and not a limiting feature.
If not enough workers are available, new goroutines will be spawned.
The maximum number of concurrent index header loads across all tenants, configurable via the -blocks-storage.bucket-store.index-header.lazy-loading-concurrency CLI flag on store-gateways.
It defaults to 4.
The maximum time to wait for the query-frontend to become ready before rejecting requests, configurable via the -query-frontend.not-running-timeout CLI flag on query-frontends.
It now defaults to 2s.
The CLI flag that allows queriers to reduce pressure on ingesters by initially querying only the minimum set of ingesters required to reach quorum, -querier.minimize-ingester-requests.
It is now enabled by default.
Spread-minimizing token-related CLI flags: -ingester.ring.token-generation-strategy, -ingester.ring.spread-minimizing-zones and -ingester.ring.spread-minimizing-join-ring-in-order.
You can read more about this feature in our blog post.

Important changes

In Grafana Mimir 2.12 the following behavior has changed:

Store-gateway now persists a sparse version of the index-header to disk on construction and loads sparse index-headers from disk instead of the whole index-header.
This improves the speed at which index headers are lazy-loaded from disk by up to 90%. The added disk usage is in the order of 1-2%.
Alertmanager deprecated the v1 API. All v1 API endpoints now respond with a JSON deprecation notice and a status code of 410.
All endpoints have a v2 equivalent.
The list of endpoints is:
- <alertmanager-web.external-url>/api/v1/alerts
- <alertmanager-web.external-url>/api/v1/receivers
- <alertmanager-web.external-url>/api/v1/silence/{id}
- <alertmanager-web.external-url>/api/v1/silences
- <alertmanager-web.external-url>/api/v1/status
Exemplar's label traceID has been changed to trace_id to be consistent with the OpenTelemetry standard.
Errors returned by ingesters now contain only gRPC status codes.
Previously they contained both gRPC and HTTP status codes.
To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.
Responses with gRPC status codes are now reported as status_code labels in the cortex_request_duration_seconds and cortex_ingester_client_request_duration_seconds metrics.
Responses with HTTP 4xx status codes are now treated as errors and used in status_code label of request duration metric.

The default value of the following CLI flags have been changed:

-blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
-blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
-blocks-storage.bucket-store.tenant-sync-concurrency from 10 to 1.
-query-frontend.max-cache-freshness from 1m to 10m.
-distributor.write-requests-buffer-pooling-enabled from false to true.
-locks-storage.bucket-store.block-sync-concurrency from 20 to 4.
-memberlist.stream-timeout from 10s to 2s.
-server.report-grpc-codes-in-instrumentation-label-enabled from false to true.

The following deprecated configuration options are removed in Grafana Mimir 2.12:

The YAML setting frontend.cache_unaligned_requests.
Experimental CLI flag -querier.prefer-streaming-chunks-from-ingesters.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.14:

The CLI flag -ingester.limit-inflight-requests-using-grpc-method-limiter.
It now defaults to true.
The CLI flag -ingester.return-only-grpc-errors.
It now defaults to true.
To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.
The CLI flag -ingester.client.report-grpc-codes-in-instrumentation-label-enabled.
It now defaults to true.
The CLI flag -distributor.limit-inflight-requests-using-grpc-method-limiter.
It now defaults to true.
The CLI flag -distributor.enable-otlp-metadata-storage.
It now defaults to true.
The CLI flag -querier.max-query-into-future.

The following metrics are removed or deprecated:

cortex_bucket_store_blocks_loaded_by_duration has been removed.
cortex_distributor_sample_delay_seconds has been deprecated and will be removed in Mimir 2.14.

Experimental features

Grafana Mimir 2.12 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

The maximum number of tenant IDs that may be for a federated query can be configured via the -tenant-federation.max-tenants CLI flag on query-frontends.
By default, it's 0, meaning that the limit is disabled.
Sharding of active series queries can be enabled via the -query-frontend.shard-active-series-queries CLI flag on query-frontends.
Timely head compaction can be enabled via the -blocks-storage.tsdb.timely-head-compaction-enabled on ingesters.
If enabled, the head compaction happens when the min block range can no longer be appended, without requiring 1.5x the chunk range worth of data in the head.
Streaming of responses from querier to query-frontend can be enabled via the -querier.response-streaming-enabled CLI flag on queriers.
This is currently supported only for responses from the /api/v1/cardinality/active_series endpoint.
The maximum response size for active series queries, in bytes, can be set via the -querier.active-series-results-max-size-bytes CLI flag on queriers.
Metric relabeling on a per-tenant basis can be forcefully disabled via the -distributor.metric-relabeling-enabled CLI flag on rulers.
Metrics relabeling is enabled by default.
Query Queue Load Balancing by Query Component. Tenant query queues in the query-scheduler can now be split into subqueues by which query component is expected to be utilized to complete the query: ingesters, store-gateways, both, or uncategorized.
Dequeuing queries for a given tenant will rotate through the query component subqueues via simple round-robin.
In the event that the one of the query components (ingesters or store-gateways) experience a slowdown, queries only utilizing the the other query component can continue to be serviced.
This feature is recommended to be enabled.
The following CLI flags must be set to true in order to be in effect:
- `-query...

Assets 118

12 Mar 19:10

duricanikolic

mimir-2.12.0-rc.0

d0ac52d

2.12.0-rc.0 Pre-release

Pre-release

Grafana Mimir version 2.12.0-rc.0 release notes

Grafana Labs is excited to announce version 2.12 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

Added support to only count series that are considered active through the Cardinality API endpoint /api/v1/cardinality/label_names by passing the count_method parameter.
If set to active it counts only series that are considered active according to the -ingester.active-series-metrics-idle-timeout flag setting rather than counting all in-memory series.
The "Store-gateway: bucket tenant blocks" admin page contains a new column "No Compact".
If block no compaction marker is set, it specifies the reason and the date the marker is added.
The estimated number of compaction jobs based on the current bucket-index is now computed by the compactor.
The result is tracked by the new cortex_bucket_index_compaction_jobs metric.
If this computation fails, the cortex_bucket_index_compaction_jobs_errors_total metric is updated instead.
The estimated number of compaction jobs is also shown in Top tenants, Tenants, and Compactor dashboards.
Added mimir-distroless container image built upon a distroless image (gcr.io/distroless/static-debian12).
This improvement minimizes attack surfaces and potential CVEs by trimming down the dependencies within the image.
After comprehensive testing, the Mimir maintainers plan to shift from the current image to the distroless version.

Additionally, the following previously experimental features are now considered stable:

The number of pre-allocated workers used to forward push requests to the ingesters, configurable via the -distributor.reusable-ingester-push-workers CLI flag on distributors.
It now defaults to 2000.
Note that this is a performance optimization, and not a limiting feature.
If not enough workers available, new goroutines will be spawned.
The number of gRPC server workers used to serve the requests, configurable via the -server.grpc.num-workers CLI flag.
It now defaults to 100.
Note that this is the number of pre-allocated long-lived workers, and not a limiting feature.
If not enough workers are available, new goroutines will be spawned.
The maximum number of concurrent index header loads across all tenants, configurable via the -blocks-storage.bucket-store.index-header.lazy-loading-concurrency CLI flag on store-gateways.
It defaults to 4.
The maximum time to wait for the query-frontend to become ready before rejecting requests, configurable via the -query-frontend.not-running-timeout CLI flags on query-frontends.
It now defaults to 2s.
Spread-minimizing token-related CLI flags: -ingester.ring.token-generation-strategy, -ingester.ring.spread-minimizing-zones and -ingester.ring.spread-minimizing-join-ring-in-order.
You can read more about this feature in our blog post.

Important changes

In Grafana Mimir 2.12 the following behavior has changed:

Store-gateway now persists a sparse version of the index-header to disk on construction and loads sparse index-headers from disk instead of the whole index-header.
This improves the speed at which index headers are lazy-loaded from disk by up to 90%. The added disk usage is in the order of 1-2%.
Alertmanager deprecated the v1 API. All v1 API endpoints now respond with a JSON deprecation notice and a status code of 410.
All endpoints have a v2 equivalent.
The list of endpoints is:
- <alertmanager-web.external-url>/api/v1/alerts
- <alertmanager-web.external-url>/api/v1/receivers
- <alertmanager-web.external-url>/api/v1/silence/{id}
- <alertmanager-web.external-url>/api/v1/silences
- <alertmanager-web.external-url>/api/v1/status
Exemplar's label traceID has been changed to trace_id to be consistent with the OpenTelemetry standard.
Errors returned by ingesters now contain only gRPC status codes.
Previously they contained both gRPC and HTTP status codes.
To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.
Responses with gRPC status codes are now reported as status_code labels in the cortex_request_duration_seconds and cortex_ingester_client_request_duration_seconds metrics.
Responses with HTTP 4xx status codes are now treated as errors and used in status_code label of request duration metric.

The default value of the following CLI flags have been changed:

-blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
-blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
-blocks-storage.bucket-store.tenant-sync-concurrency from 10 to 1.
-query-frontend.max-cache-freshness from 1m to 10m.
-distributor.write-requests-buffer-pooling-enabled from false to true.
-locks-storage.bucket-store.block-sync-concurrency from 20 to 4.
-memberlist.stream-timeout from 10s to 2s.
-server.report-grpc-codes-in-instrumentation-label-enabled from false to true.

The following deprecated configuration options are removed in Grafana Mimir 2.12:

The YAML setting frontend.cache_unaligned_requests.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.14:

The CLI flag -ingester.limit-inflight-requests-using-grpc-method-limiter.
It now defaults to true.
The CLI flag -ingester.return-only-grpc-errors.
It now defaults to true.
To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.
The CLI flag -ingester.client.report-grpc-codes-in-instrumentation-label-enabled.
It now defaults to true.
The CLI flag -distributor.limit-inflight-requests-using-grpc-method-limiter.
It now defaults to true.
The CLI flag -distributor.enable-otlp-metadata-storage.
It now defaults to true.
The CLI flag -querier.max-query-into-future.

The following metrics are removed or deprecated:

cortex_bucket_store_blocks_loaded_by_duration has been removed.
cortex_distributor_sample_delay_seconds has been deprecated and will be removed in Mimir 2.14.

Experimental features

Grafana Mimir 2.12 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

The maximum number of tenant IDs that may be for a federated query can be configured via the -tenant-federation.max-tenants CLI flag on query-frontends.
By default, it's 0, meaning that the limit is disabled.
Sharding of active series queries can be enabled via the -query-frontend.shard-active-series-queries CLI flag on query-frontends.
Timely head compaction can be enabled via the -blocks-storage.tsdb.timely-head-compaction-enabled on ingesters.
If enabled, the head compaction happens when the min block range can no longer be appended, without requiring 1.5x the chunk range worth of data in the head.
Streaming of responses from querier to query-frontend can be enabled via the -querier.response-streaming-enabled CLI flag on queriers.
This is currently supported only for responses from the /api/v1/cardinality/active_series endpoint.
The maximum response size for active series queries, in bytes, can be set via the -querier.active-series-results-max-size-bytes CLI flag on queriers.
Metric relabeling on a per-tenant basis can be forcefully disabled via the -distributor.metric-relabeling-enabled CLI flag on rulers.
Metrics relabeling is enabled by default.
Query Queue Load Balancing by Query Component. Tenant query queues in the query-scheduler can now be split into subqueues by which query component is expected to be utilized to complete the query: ingesters, store-gateways, both, or uncategorized.
Dequeuing queries for a given tenant will rotate through the query component subqueues via simple round-robin.
In the event that the one of the query components (ingesters or store-gateways) experience a slowdown, queries only utilizing the the other query component can continue to be serviced.
This feature is recommended to be enabled.
The following CLI flags must be set to true in order to be in effect:
- -query-frontend.additional-query-queue-dimensions-enabled on the query-frontend.
- -query-scheduler.additional-query-queue-dimensions-enabled on the query-scheduler.
Owned series tracking in ingesters can be enabled via the -ingester.track-ingester-owned-series CLI flag.
When enabled, inge...

Assets 118

26 Dec 17:26

leizor

mimir-2.11.0

c8939ea

2.11.0

This release contains 532 PRs from 55 authors, including new contributors Benjamin, Dominik Kepinski, Jonathan Donzallaz, Juraj Michálek, Kai.Ke, Ludovic Terrier, Luke, Maciej Lech, Matthew Penner, Michael Potter, Mihai Țimbota-Belin, Rasmus Werner Salling, Ying WANG, chencs, fayzal-g, kalle (jag), sarthaktyagi-505, whoami. Thank you!

Grafana Mimir version 2.11.0 release notes

Grafana Labs is excited to announce version 2.11 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

Sampled logging of errors in the ingester. A high-traffic Mimir cluster can occasionally become bogged down logging high volumes of repeated errors. You can now reduce the amount of errors outputted to logs by setting a sample rate via the -ingester.error-sample-rate CLI flag.
Add total request size instance limit for ingesters. This limit protects the ingesters against requests that together may cause an OOM. Enable this feature by setting the -ingester.instance-limits.max-inflight-push-requests-bytes CLI flag in combination with the -ingester.limit-inflight-requests-using-grpc-method-limiter CLI flag.
Reduce the resolution of incoming native histograms samples if the incoming sample has too many buckets compared to -validation.max-native-histogram-buckets. This is enabled by default but can be turned off by setting the -validation.reduce-native-histogram-over-max-buckets CLI flag to false.
Improved query-scheduler performance under load. This is particularly apparent for clusters with large numbers of queriers.
Ingester to querier chunks streaming reduces the memory utilization of queriers and reduces the likelihood of OOMs.
Ingester query request minimization reduces the number of query requests to ingesters, improving performance and resource utilization for both ingesters and queriers.

Experimental features

Grafana Mimir 2.11 includes new features that are considered experimental and disabled by default. Please use them with caution and report any issue you encounter:

Block specified queries on a per-tenant basis. This is configured via the blocked_queries limit. See the docs for more information.
Store metadata when ingesting metrics via OTLP. This makes metric description and type available when ingesting metrics via OTLP. You can enable this feature by setting the CLI flag -distributor.enable-otlp-metadata-storage to true.
Reject gRPC push requests that the ingester/distributor is unable to accept before reading them into memory. You can enable this feature by using the -ingester.limit-inflight-requests-using-grpc-method-limiter and/or the -distributor.limit-inflight-requests-using-grpc-method-limiter CLI flags for the ingester and/or the distributor, respectively.
Customize the memcached client write and read buffer size. The buffer allocated for each memcached connection can be configured via the following CLI flags:
- For the blocks storage:
  - -blocks-storage.bucket-store.chunks-cache.memcached.read-buffer-size-bytes
  - -blocks-storage.bucket-store.chunks-cache.memcached.write-buffer-size-bytes
  - -blocks-storage.bucket-store.index-cache.memcached.read-buffer-size-bytes
  - -blocks-storage.bucket-store.index-cache.memcached.write-buffer-size-bytes
  - -blocks-storage.bucket-store.metadata-cache.memcached.read-buffer-size-bytes
  - -blocks-storage.bucket-store.metadata-cache.memcached.write-buffer-size-bytes
- For the query frontend:
  - -query-frontend.results-cache.memcached.read-buffer-size-bytes
  - -query-frontend.results-cache.memcached.write-buffer-size-bytes
- For the ruler storage:
  - -ruler-storage.cache.memcached.read-buffer-size-bytes
  - -ruler-storage.cache.memcached.write-buffer-size-bytes
Configure the number of long-living workers used to process gRPC requests. This can decrease CPU usage by reducing the number of stack allocations. Configure this feature by using the -server.grpc.num-workers CLI flag.
Enforce a limit in bytes on the PostingsForMatchers cache used by ingesters. This limit can be configured via the -blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes and -blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes CLI flags.
Pre-allocate the pool of workers in the distributor that are used to send push requests to ingesters. This can decrease CPU usage by reducing the number of stack allocations. You can enable this feature by using the -distributor.reusable-ingester-push-worker flag.
Include a Retry-After header in recoverable error responses from the distributor. This can protect your Mimir cluster from clients including Prometheus that default to retrying very quickly. Enable this feature by setting the -distributor.retry-after-header.enabled CLI flag.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.11 the following behavior has changed:

The utilization-based read path limiter now operates on Go heap size instead of RSS from the Linux proc file system.

The following configuration options had been previously deprecated and are removed in Grafana Mimir 2.11:

The CLI flag -querier.iterators.
The CLI flag -query.batch-iterators.
The CLI flag -blocks-storage.bucket-store.bucket-index.enabled.
The CLI flag -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes.
The CLI flag -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes.
The CLI flag -blocks-storage.bucket-store.max-chunk-pool-bytes.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.13:

The CLI flag -log.buffered; this is now the default behavior.

The following metrics are removed:

cortex_query_frontend_workers_enqueued_requests_total; use cortex_query_frontend_enqueue_duration_seconds_count instead.

The following configuration option defaults were changed:

The CLI flag -blocks-storage.bucket-store.index-header.sparse-persistence-enabled now defaults to true.
The default value for the CLI flag -blocks-storage.bucket-store.index-header.lazy-loading-concurrency was changed from 0 to 4.
The default value for the CLI flag -blocks-storage.tsdb.series-hash-cache-max-size-bytes was changed from 1GB to 350MB.
The default value for the CLI flag -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage was changed from 10 to 15.

Bug fixes

Ingester: Respect context cancelation during query execution. PR 6085
Distributor: Return 529 when ingestion rate limit is hit and the distributor.service_overload_status_code_on_rate_limit_enabled flag is active. PR 6549
Query-scheduler: Prevent accumulation of stale querier connections. PR 6100
Packaging: Fix preremove script preventing upgrades on RHEL based OS. PR 6067

Changelog

2.11.0

Grafana Mimir

[CHANGE] The following deprecated configurations have been removed: #6673 #6779 #6808 #6814
- -querier.iterators
- -querier.batch-iterators
- -blocks-storage.bucket-store.max-chunk-pool-bytes
- -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes
- -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes
- -blocks-storage.bucket-store.bucket-index.enabled
[CHANGE] Querier: Split worker GRPC config into separate client configs for the frontend and scheduler to allow TLS to be configured correctly when specifying the tls_server_name. The GRPC config specified under -querier.frontend-client.* will no longer apply to the scheduler client, and will need to be set explicitly under -querier.scheduler-client.*. #6445 #6573
[CHANGE] Store-gateway: enable sparse index headers by default. Sparse index headers reduce the time to load an index header up to 90%. #6005
[CHANGE] Store-gateway: lazy-loading concurrency limit default value is now 4. #6004
[CHANGE] General: enabled -log.buffered by default. The -log.buffered has been deprecated and will be removed in Mimir 2.13. #6131
[CHANGE] Ingester: changed default -blocks-storage.tsdb.series-hash-cache-max-size-bytes setting from 1GB to 350MB. The new default cache size is enough to store the hashes for all series in a ingester, assuming up to 2M in-memory series per ingester and using the default 13h retention period for local TSDB blocks in the ingesters. #6130
[CHANGE] Query-frontend: removed cortex_query_frontend_workers_enqueued_requests_total. Use cortex_query_frontend_enqueue_duration_seconds_count instead. #6121
[CHANGE] Ingester / querier: enable ingester to querier chunks streaming by default and mark it as stable. #6174
[CHANGE] Ingester / querier: enable ingester query request minimisation by default and mark it as stable. #6174
[CHANGE] Ingester: changed the default value for the experimental configuration parameter -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage from 10 to 15. #6186
[CHANGE] Ingester: /ingester/push HTTP endpoint has been removed. This endpoint was added for testing and troubleshooting, but was never documented or used for anything. #6299
[CHANGE] Experimental...

Assets 114

20 Dec 20:46

leizor

mimir-2.11.0-rc.0

229cba4

2.11.0-rc.0 Pre-release

Pre-release

This release contains 531 PRs from 55 authors, including new contributors Benjamin, Dominik Kepinski, Jonathan Donzallaz, Juraj Michálek, Kai.Ke, Ludovic Terrier, Luke, Maciej Lech, Matthew Penner, Michael Potter, Mihai Țimbota-Belin, Rasmus Werner Salling, Ying WANG, chencs, fayzal-g, kalle (jag), renovate[bot], sarthaktyagi-505, whoami. Thank you!

Grafana Mimir version 2.11.0-rc.0 release notes

Grafana Labs is excited to announce version 2.11 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

Sampled logging of errors in the ingester. A high-traffic Mimir cluster can occasionally become bogged down logging high volumes of repeated errors. You can now reduce the amount of errors outputted to logs by setting a sample rate via the -ingester.error-sample-rate CLI flag.
Add total request size instance limit for ingesters. This limit protects the ingesters against requests that together may cause an OOM. Enable this feature by setting the -ingester.instance-limits.max-inflight-push-requests-bytes CLI flag in combination with the -ingester.limit-inflight-requests-using-grpc-method-limiter CLI flag.
Reduce the resolution of incoming native histograms samples if the incoming sample has too many buckets compared to -validation.max-native-histogram-buckets. This is enabled by default but can be turned off by setting the -validation.reduce-native-histogram-over-max-buckets CLI flag to false.
Improved query-scheduler performance under load. This is particularly apparent for clusters with large numbers of queriers.
Ingester to querier chunks streaming reduces the memory utilization of queriers and reduces the likelihood of OOMs.
Ingester query request minimization reduces the number of query requests to ingesters, improving performance and resource utilization for both ingesters and queriers.

Experimental features

Grafana Mimir 2.11 includes new features that are considered experimental and disabled by default. Please use them with caution and report any issue you encounter:

Block specified queries on a per-tenant basis. This is configured via the blocked_queries limit. See the docs for more information.
Store metadata when ingesting metrics via OTLP. This makes metric description and type available when ingesting metrics via OTLP. You can enable this feature by setting the CLI flag -distributor.enable-otlp-metadata-storage to true.
Reject gRPC push requests that the ingester/distributor is unable to accept before reading them into memory. You can enable this feature by using the -ingester.limit-inflight-requests-using-grpc-method-limiter and/or the -distributor.limit-inflight-requests-using-grpc-method-limiter CLI flags for the ingester and/or the distributor, respectively.
Customize the memcached client write and read buffer size. The buffer allocated for each memcached connection can be configured via the following CLI flags:
- For the blocks storage:
  - -blocks-storage.bucket-store.chunks-cache.memcached.read-buffer-size-bytes
  - -blocks-storage.bucket-store.chunks-cache.memcached.write-buffer-size-bytes
  - -blocks-storage.bucket-store.index-cache.memcached.read-buffer-size-bytes
  - -blocks-storage.bucket-store.index-cache.memcached.write-buffer-size-bytes
  - -blocks-storage.bucket-store.metadata-cache.memcached.read-buffer-size-bytes
  - -blocks-storage.bucket-store.metadata-cache.memcached.write-buffer-size-bytes
- For the query frontend:
  - -query-frontend.results-cache.memcached.read-buffer-size-bytes
  - -query-frontend.results-cache.memcached.write-buffer-size-bytes
- For the ruler storage:
  - -ruler-storage.cache.memcached.read-buffer-size-bytes
  - -ruler-storage.cache.memcached.write-buffer-size-bytes
Configure the number of long-living workers used to process gRPC requests. This can decrease CPU usage by reducing the number of stack allocations. Configure this feature by using the -server.grpc.num-workers CLI flag.
Enforce a limit in bytes on the PostingsForMatchers cache used by ingesters. This limit can be configured via the -blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes and -blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes CLI flags.
Pre-allocate the pool of workers in the distributor that are used to send push requests to ingesters. This can decrease CPU usage by reducing the number of stack allocations. You can enable this feature by using the -distributor.reusable-ingester-push-worker flag.
Include a Retry-After header in recoverable error responses from the distributor. This can protect your Mimir cluster from clients including Prometheus that default to retrying very quickly. Enable this feature by setting the -distributor.retry-after-header.enabled CLI flag.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.11 the following behavior has changed:

The utilization-based read path limiter now operates on Go heap size instead of RSS from the Linux proc file system.

The following configuration options had been previously deprecated and are removed in Grafana Mimir 2.11:

The CLI flag -querier.iterators.
The CLI flag -query.batch-iterators.
The CLI flag -blocks-storage.bucket-store.bucket-index.enabled.
The CLI flag -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes.
The CLI flag -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes.
The CLI flag -blocks-storage.bucket-store.max-chunk-pool-bytes.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.13:

The CLI flag -log.buffered; this is now the default behavior.

The following metrics are removed:

cortex_query_frontend_workers_enqueued_requests_total; use cortex_query_frontend_enqueue_duration_seconds_count instead.

The following configuration option defaults were changed:

The CLI flag -blocks-storage.bucket-store.index-header.sparse-persistence-enabled now defaults to true.
The default value for the CLI flag -blocks-storage.bucket-store.index-header.lazy-loading-concurrency was changed from 0 to 4.
The default value for the CLI flag -blocks-storage.tsdb.series-hash-cache-max-size-bytes was changed from 1GB to 350MB.
The default value for the CLI flag -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage was changed from 10 to 15.

Bug fixes

Ingester: Respect context cancelation during query execution. PR 6085
Distributor: Return 529 when ingestion rate limit is hit and the distributor.service_overload_status_code_on_rate_limit_enabled flag is active. PR 6549
Query-scheduler: Prevent accumulation of stale querier connections. PR 6100
Packaging: Fix preremove script preventing upgrades on RHEL based OS. PR 6067

Changelog

2.11.0-rc.0

Grafana Mimir

[CHANGE] The following deprecated configurations have been removed: #6673 #6779 #6808 #6814
- -querier.iterators
- -querier.batch-iterators
- -blocks-storage.bucket-store.max-chunk-pool-bytes
- -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes
- -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes
- -blocks-storage.bucket-store.bucket-index.enabled
[CHANGE] Querier: Split worker GRPC config into separate client configs for the frontend and scheduler to allow TLS to be configured correctly when specifying the tls_server_name. The GRPC config specified under -querier.frontend-client.* will no longer apply to the scheduler client, and will need to be set explicitly under -querier.scheduler-client.*. #6445 #6573
[CHANGE] Store-gateway: enable sparse index headers by default. Sparse index headers reduce the time to load an index header up to 90%. #6005
[CHANGE] Store-gateway: lazy-loading concurrency limit default value is now 4. #6004
[CHANGE] General: enabled -log.buffered by default. The -log.buffered has been deprecated and will be removed in Mimir 2.13. #6131
[CHANGE] Ingester: changed default -blocks-storage.tsdb.series-hash-cache-max-size-bytes setting from 1GB to 350MB. The new default cache size is enough to store the hashes for all series in a ingester, assuming up to 2M in-memory series per ingester and using the default 13h retention period for local TSDB blocks in the ingesters. #6130
[CHANGE] Query-frontend: removed cortex_query_frontend_workers_enqueued_requests_total. Use cortex_query_frontend_enqueue_duration_seconds_count instead. #6121
[CHANGE] Ingester / querier: enable ingester to querier chunks streaming by default and mark it as stable. #6174
[CHANGE] Ingester / querier: enable ingester query request minimisation by default and mark it as stable. #6174
[CHANGE] Ingester: changed the default value for the experimental configuration parameter -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage from 10 to 15. #6186
[CHANGE] Ingester: /ingester/push HTTP endpoint has been removed. This endpoint was added for testing and troubleshooting, but was never documented or used for anything. #6299...

Assets 118

12 Dec 15:34

dimitarvdimitrov

mimir-2.9.4

520f1d9

2.9.4

Changelog

2.9.4

Grafana Mimir

[ENHANCEMENT] Update Docker base images from alpine:3.18.3 to alpine:3.18.5. #6895

All changes in this release: mimir-2.9.3...mimir-2.9.4

Assets 106

12 Dec 16:34

dimitarvdimitrov

mimir-2.10.5

fbecc92

2.10.5

Changelog

2.10.5

Grafana Mimir

[ENHANCEMENT] Update Docker base images from alpine:3.18.3 to alpine:3.18.5. #6897
[BUGFIX] Fixed possible series matcher corruption leading to wrong series being included in query results. #6886

Documentation

[ENHANCEMENT] Document the concept of native histograms and how to send them to Mimir, migration path. #6757
[ENHANCEMENT] Document native histograms query and visualization. #6757

All changes in this release: mimir-2.10.4...mimir-2.10.5

Assets 106

13 Nov 16:47

fayzal-g

mimir-2.9.3

97e892a

2.9.3

This release contains 1 PR from 1 author. Thank you!

Changelog

2.9.3

[BUGFIX] Update go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp to 0.44 which includes a fix for CVE-2023-45142. #6637

All changes in this release: mimir-2.9.2...mimir-2.9.3

Assets 106

Releases: grafana/mimir

2.13.0-rc.1

Grafana Mimir version 2.13.0-rc.0 release notes

Features and enhancements

Important changes

Experimental features

Bug fixes

Changelog

2.13.0-rc.1

Grafana Mimir

2.13.0-rc.0

Grafana Mimir version 2.13.0-rc.0 release notes

Features and enhancements

Important changes

Experimental features

Bug fixes

Changelog

2.13.0-rc.0

Grafana Mimir

2.12.0

Grafana Mimir version 2.12.0 release notes

Features and enhancements

Important changes

Experimental features

Contributors

2.12.0-rc.1

Grafana Mimir version 2.12.0-rc.1 release notes

Features and enhancements

Important changes

Experimental features

2.12.0-rc.0

Grafana Mimir version 2.12.0-rc.0 release notes

Features and enhancements

Important changes

Experimental features

2.11.0

Grafana Mimir version 2.11.0 release notes

Features and enhancements

Experimental features

Helm chart improvements

Important changes

Bug fixes

Changelog

2.11.0

Grafana Mimir

2.11.0-rc.0

Grafana Mimir version 2.11.0-rc.0 release notes

Features and enhancements

Experimental features

Helm chart improvements

Important changes

Bug fixes

Changelog

2.11.0-rc.0

Grafana Mimir

2.9.4

Changelog

2.9.4

Grafana Mimir

2.10.5

Changelog

2.10.5

Grafana Mimir

Documentation

2.9.3

Changelog

2.9.3