Do we need backups for the volumes?

[ecurbelo]

I am preparing the deployment of Mimir in a Kubernertes clusters and now i am checking if we need backups for the EBS volumes used by several Mimir components. They are quite a few because of the number of replicas that we have. So my question is if it's really useful having backups or the components can "recover themselves" if the EBS volume is corrupted and they start again with an empty one.

Right now the deployed components I have with volumes are:

Compactors

In the doc says that the volumes are used for

• Files downloaded from object storage used as input to compaction.
• Block files produced by the compactor to be uploaded to object storage.

So i assume that having a backup every several hours for this is not that useful?

Ingesters

Maybe it is interesting here because of the Write-ahead log or the Write-behind log? I have the 3 default replicas. I do not know exactly how it works in depth and if a backup done 2-3 times per day makes sense here.

Store-gateway

Also read in the documentation that:

• Store-gateways periodically re-download the bucket index to obtain an updated view of the long-term storage and discover new blocks uploaded by ingesters and compactors, or deleted by compactors.

So, i assume that this component is not worth backing up.

---

So, do you make backups of these volumes?

[56quarters]

Hello!

In general, we don't back up any of the EBS volumes Mimir uses and rely on the durability of EBS itself.

Compactors

These EBS volumes don't persist across compactor rollouts. There's nothing of value on them, they're just used as scratch space for compaction jobs. These are only EBS volumes to avoid impacting the availability of the underlying node (i.e. not filling up its disk).

Store-gateway

These EBS volumes don't persist across store-gateway rollouts. Store-gateways will download anything they need from object storage on startup. It's faster to not have to do that but this is an optimization, there's no value backing these up.

Ingester

We don't back these up and instead rely on the durability of EBS itself. The data stored on the volumes here is replicated to 3 ingesters and uploaded to S3 after ~2-3 hours. You could make an argument that backing them up is useful but I don't personally think it is.

[ecurbelo]

Thank you so much for the quick and detailed response! 😃