How to accelerate the purge of non-current objects beyond the lifecycle or accelerate MinIO Scanner

[Davisanity]

I have a MinIO cluster, but its scanner has been running very slowly. As a result, there are a large number of non-current objects in the cluster that exceed the non-current lifecycle but have not been purged, these non-current objects will take up my space so that I want to purge them. To address this, I made adjustments using the mc client. Originally, the settings were delay: 10, max_wait: 15, cycle: 1. I adjusted them to delay: 6, max_wait: 10, cycle: 1. However, when I checked the Last full scan time using mc admin scanner status, it showed 150d0h15m. I don't feel any improvement after changing the settings. Is there any way to make the MinIO scanner work faster or to purge non-current objects beyond the lifecycle more promptly? And what will happen with MinIO scanner if I execute mc admin service restart ?

The version of MinIO is RELEASE.2022-12-12T19-27-27Z, I used 48 nodes, each node have 10 drives, each drive has 7TiB, EC:6.
The usage of the cluster is 416TiB, and there's 1011543188(1B) objects 948779906 versions in the cluster

Belows are the result of mc admin scanner status :
With setting delay: 10, max_wait: 15, cycle: 1 :
Last full scan time: 111d12h18m Estimated 0.27/month
Last minute statistics
Objects Scanned 392 objects; Avg: 12.703ms; Rate: 564480/day
Versions Scanner: 395 versions; Avg; 27.151us; Rate: 568800/day

With setting delay: 6, max_wait: 10, cycle: 1:
Last full scan time: 150d0h15m Estimated 0.20/month
Last minute statistics
Objects Scanned 19384 objects; Avg: 23.16 ms; Rate: 27912960/day
Versions Scanner: 23257 versions; Avg; 2.803 us; Rate: 33490080/day

[Jazz9]

mc admin config set ALIAS scanner speed=default

try these - fastest, fast, default, slow, slowest

no restart required.

[harshavardhana]

The version of MinIO is RELEASE.2022-12-12T19-27-27Z, I used 48 nodes, each node have 10 drives, each drive has 7TiB, EC:6. The usage of the cluster is 416TiB, and there's 1011543188(1B) objects 948779906 versions in the cluster

Please upgrade to latest - if you looking for professional-level help - your cluster needs to subscribe itself at https://min.io/pricing

[Davisanity]

Thanks for the answer, but may I know in which version has optimization or enhancement with scanner?

[harshavardhana]

Latest the one we released today.

[Davisanity]

Ok, I've read it, but may I know in which specific version do MinIO team enhance for MinIO scanner, because I see the release note in the latest version: RELEASE.2023-07-18T17-49-40Z , there's no description of enhancement for MinIO scanner. Sorry for bothering you

[ravindk89]

@Davisanity FOSS-licensed deployments should track latest stable as a general practice. It's not as if we only release optimizations once a quarter, they can happen at any time and in any release. Optimizations may also not be that clear - a fix in one subsystem might improve performance or behavior in another.

Our best practices here are to test upgrades in a lower environment such as DEV, QA, or UAT, then apply to production. All other support is best-effort only, and doing detailed debugging and remediation of performance issues - scanner, S3 API, etc. - are handled via SUBNET and our commercial offering, as they by nature require direct engineering engagement, log review, etc.

One note - I/O is a zero-sum game. Your workload/normal I/O competes with Scanner I/O. If you increase the aggressiveness of the scanner overall, you decrease available workload I/O. Your setting changes took the scanner rate from 564,480 objects/day to 27,912,960 / day - the scanner total time may take time to update, but that is a ~50x increase in scanner speed already.

We appreciate your understanding here.

[Davisanity]

@ravindk89 Thanks for your detail and patient answer!!

[xpader]

Scanner is totally a bad design, I think the full scanner of bucket should be just a of admin command to use for migration or similar to fix something, not a regular running mechanism.

Minio should build a metadata database, use some kvdb or tsdb, and use just a simple query to fetch the file list for lifecycle etc.

Scan hundreds of thousands or millions of files is horrible.

[klauspost]

Nobody is forcing you to use MinIO. Hope you find what you are looking for since it doesn't seem like you need a scalable solution. Bye.

[xpader]

I'm not here to criticize minio. I really hope minio can consider whether there may be a better solution to make minio better..