Compactor: Upload Sparse Index Headers to Object Storage

[dmwilson-grafana]

What this PR does

• Adds an option -compactor.upload-sparse-index-headers which allows the compactor to upload sparse index headers to object storage. When this option is set to true, the compactor will build and upload sparse-index-headers after all other files in the block have been uploaded to object storage. When this option is set, it's a "best effort" process, the compactor won't exit an otherwise successful compaction job early if there's an error affecting the sparse-index-header upload.

• To make this change, this PR moves the contents of pkg/storegateway/indexheader to pkg/storage/indexheader. Both StoreGateway and Compactor import pkg/storage/indexheader.

• As implemented, sparse headers uploaded from the compactor use DefaultPostingOffsetInMemorySampling (1/32), not blocks-storage.bucket-store.posting-offsets-in-mem-sampling.

  • This PR adds the sampling rate to the sparse protos and sparse-index-header file, this allows stream readers to downsample from higher to lower sampling rates (e.g. 1/16 -> 1/32).

• Adding an extra file to upload for each block will slow the comapctor a bit. In testing, enabling upload of sparse-index-headers added <100ms/block.

Which issue(s) this PR fixes or relates to

• Compactor: store sparse index headers in object store #8166 - New store-gateway replicas don't have any sparse index headers right after starting up and must load the index header and create the sparse header from the data on disk before being able to serve requests for a given block. This leads to severe increases in latency after scaling out store-gateways.

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[56quarters]

Before this is marked as ready for review, pkg/storage or somewhere under there is the directory that the index header code should live in instead of pkg/util

[github-actions]

💻 Deploy preview available: https://deploy-preview-mimir-10684-zb444pucvq-vp.a.run.app/docs/mimir/latest/

[dimitarvdimitrov]

thanks for working on this. The logic looks sound, but i think we can move around where it sits

[dimitarvdimitrov]

LGTM, thanks for working on this. I left small suggestions and pushed a change to the changelog because i think it got lost in rebasing

[dimitarvdimitrov]

i realized we aren't monitoring the process of constructing and parsing the sparse headers. So we wouldn't know if the compactors suddenly start failing to construct or upload these sparse headers. We also wouldn't know if the store-gateways suddenly find them corrupted. If this happens the query path will silently degrade.

We should add some metrics to track failures to construct and uploading and to download and parse and alerts on those in case they keep failing for a long time. Do you want to give this a try in a follow-up PR?

[dimitarvdimitrov]

thanks for working through the comments

LGTM

[tacole02]

Thanks, @dmwilson-grafana !