high block output when idling

[helmut72]

NOTE

I have installed Minio in a Docker container, created a bucket, uploaded a file and let Minio running without any access. After idling Minio for some hours, block output in Docker (docker stats minio, column block i/o) is growing. Even my SQL servers don't write that much when in use (private server, few users).

Expected Behavior

When Minio idles it shouldn't write that much to disk.

Current Behavior

Minio write huge data (compared to my other 40+ containers) to disk when idling.

Steps to Reproduce (for bugs)

Install Minio: https://docs.min.io/minio/baremetal/installation/deploy-minio-single-node-single-drive.html#deploy-containerized-single-node-single-drive-minio

1. Create a bucket
2. wait for hours, days
3. check block i/o in docker stats minio

Context

Wrting somethin to disk isn't helpful to avoid wearing of a SSD.

Your Environment

• all Minio versions published this year
• Server setup and configuration: Ubuntu 22.04, latest Docker. But also with Ubuntu 20.04 and Docker early this year. Maybe also some older versions, don't use Minio that long

[klauspost]

Minio runs a scanner that continuously checks your data.

Block I/O doesn't tell you if it is reads or writes. Reads are expected (also when idle, and it doesn't wear your SSD) and state is consisted during scans, IIRC every minute.

Executing mc admin config get ALIAS scanner will show your scanner config. You can make delay and max_wait bigger if you want to slow the scanner down. For both bigger values gives slower scanning, with delay being the main controller.

Use mc admin scanner info ALIAS to see scanner speed.

[helmut72]

Block I/O doesn't tell you if it is reads or writes.

Now I got a spare PC and use smartctl tool on the host, which reads the stats of the raw SSD. Started only MinIO with few buckets and files (35 files, 60MB in size), no S3 or admin access since yesterday = MinIO just idles (the whole PC too).

This is the result:

Mon 12:09 am

smartctl data written: 51335,251 GB

docker stats:
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
b99385add0eb   minio     0.79%     279.3MiB / 15.31GiB   1.78%     665kB / 71.3MB   2.87MB / 338MB   10

Mon 06:49 pm

smartctl data written: 51336,632 GB

docker stats:
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
b99385add0eb   minio     0.93%     281MiB / 15.31GiB   1.79%     666kB / 71.3MB   11.1MB / 1.35GB   10

Tue 07:55 am

smartctl data written: 51339,356 GB

docker stats:
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O         PIDS
b99385add0eb   minio     0.00%     282MiB / 15.31GiB   1.80%     668kB / 71.3MB   27.2MB / 3.35GB   10

IMHO block I/O (especially it's the O in I/O = output = writes) correlates with the numbers from smartctl and block I/O in docker stats command is a valid value.

[mishak87]

I have same issue with running Minio SDSN on ZFS pool as storage for various backups it keeps writing to disk every minute or two despite nothing begin read or written.

The simple use-case for Minio to run S3 api for a disk folder for infrequent writes is degraded because of it.
Imagine running this on RaspberryPi where it will silently degrade the SD card.

[mishak87]

I suspect this is the culprit will try to make PR when I find where it is in code.
2049B written every 15s...

[pid 933526] mkdirat(AT_FDCWD, "/export/.minio.sys/tmp", 0777 <unfinished ...>
[pid 933491] futex(0xc00008a548, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 933526] <... mkdirat resumed>)     = -1 EEXIST (File exists)
[pid 933526] openat(AT_FDCWD, "/export/.minio.sys/tmp/9a65388f-b5e8-4198-b3b6-1ce3bcf07f5b", O_WRONLY|O_CREAT|O_TRUNC|O_DSYNC|O_CLOEXEC, 0666) = 9
[pid 933526] fcntl(9, F_GETFL)          = 0x9001 (flags O_WRONLY|O_DSYNC|O_LARGEFILE)
[pid 933526] fcntl(9, F_SETFL, O_WRONLY|O_NONBLOCK|O_DSYNC|O_LARGEFILE) = 0
[pid 933526] epoll_ctl(4, EPOLL_CTL_ADD, 9, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=462947784, u64=9214040237319652808}}) = -1 EPERM (Operation not permitted)
[pid 933526] fcntl(9, F_GETFL)          = 0x9801 (flags O_WRONLY|O_NONBLOCK|O_DSYNC|O_LARGEFILE)
[pid 933526] fcntl(9, F_SETFL, O_WRONLY|O_DSYNC|O_LARGEFILE) = 0
[pid 933526] write(9, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2049) = 2049
[pid 933526] close(9)                   = 0
[pid 933526] openat(AT_FDCWD, "/export/.minio.sys/tmp/9a65388f-b5e8-4198-b3b6-1ce3bcf07f5b", O_RDONLY|O_NOATIME|O_CLOEXEC) = 9
[pid 933526] fcntl(9, F_GETFL)          = 0x48000 (flags O_RDONLY|O_LARGEFILE|O_NOATIME)
[pid 933526] fcntl(9, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_NOATIME) = 0
[pid 933526] epoll_ctl(4, EPOLL_CTL_ADD, 9, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=462947785, u64=9214040237319652809}}) = -1 EPERM (Operation not permitted)
[pid 933526] fcntl(9, F_GETFL)          = 0x48800 (flags O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_NOATIME)
[pid 933526] fcntl(9, F_SETFL, O_RDONLY|O_LARGEFILE|O_NOATIME) = 0
[pid 933526] fstat(9, {st_mode=S_IFREG|0644, st_size=2049, ...}) = 0
[pid 933526] read(9, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2049) = 2049
[pid 933526] close(9)

[mishak87]

It should be possible to disable it by setting env _MINIO_DRIVE_ACTIVE_MONITORING=off

minio/cmd/globals.go

Line 429 in 1aa8896

globalDriveMonitoring = env.Get("_MINIO_DRIVE_ACTIVE_MONITORING", config.EnableOn) == config.EnableOn

Will test tomorrow.

[harshavardhana]

Locking conversation here, monitoring a drive is a production requirement and turning it off is your own risk.

So please do what makes it okay for you. But we don't recommend it.