Question on Linux disk reads on multipath systems

[mnellemann]

Hi,

On a Linux system backed by multipath FC I would expect the see the io on the individual devices (sda, sdb, sdc, sdb), but numbers looks very low. There are also the /dev/dm- and /dev/mapper/mpath devices, but those do not add up either.

Looking for any information on best way to go about this.

iotop command running while producing some writes

grafana metrics (through OSHI) from the same time:

[root@pvs-fra2-lnx02 ~]# iostat -d; sleep 10; iostat -d
Linux 4.18.0-408.el8.ppc64le (pvs-fra2-lnx02) 	11/18/2022 	_ppc64le_	(40 CPU)

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.02         0.04         0.00       8303          0
sdb               0.38         1.83       264.67     396987   57488546
sdc               0.02         0.04         0.00       8303          0
sdd               0.38         1.88       264.63     408923   57480335
sdg               0.02         0.04         0.00       8303          0
sde               0.02         0.04         0.00       8303          0
sdf               0.38         1.82       264.22     396394   57391517
sdh               0.38         1.86       264.42     404982   57433951
dm-0              1.45         7.25      1057.94    1574011  229794350
dm-1              0.00         0.01         0.00       2304          0
dm-2              0.77         7.22      1057.94    1569016  229794350

Linux 4.18.0-408.el8.ppc64le (pvs-fra2-lnx02) 	11/18/2022 	_ppc64le_	(40 CPU)

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               0.02         0.04         0.00       8303          0
sdb               0.38         1.83       265.52     396987   57675215
sdc               0.02         0.04         0.00       8303          0
sdd               0.39         1.88       265.51     408923   57673088
sdg               0.02         0.04         0.00       8303          0
sde               0.02         0.04         0.00       8303          0
sdf               0.39         1.82       265.10     396410   57585723
sdh               0.38         1.86       265.31     405054   57630612
dm-0              1.46         7.25      1061.44    1574099  230564639
dm-1              0.00         0.01         0.00       2304          0
dm-2              0.77         7.22      1061.44    1569104  230564639

Multipath configuration:

mpatha (360050768108104bfb800000000001efa) dm-0 IBM,2145
size=120G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='service-time 0' prio=0 status=enabled
| |- 0:0:0:0 sda 8:0   active undef running
| |- 1:0:0:0 sdc 8:32  active undef running
| |- 2:0:0:0 sde 8:64  active undef running
| `- 3:0:0:0 sdg 8:96  active undef running
`-+- policy='service-time 0' prio=0 status=active
  |- 0:0:1:0 sdb 8:16  active undef running
  |- 1:0:1:0 sdd 8:48  active undef running
  |- 2:0:1:0 sdf 8:80  active undef running
  `- 3:0:1:0 sdh 8:112 active undef running

Devices present:

[root@pvs-fra2-lnx02 bin]# ls -l /dev/dm-*
brw-rw----. 1 root disk 253, 0 Nov 15 20:05 /dev/dm-0
brw-rw----. 1 root disk 253, 1 Nov 15 20:05 /dev/dm-1
brw-rw----. 1 root disk 253, 2 Nov 15 20:05 /dev/dm-2
[root@pvs-fra2-lnx02 bin]# ls -l /dev/mapper/mpatha*
lrwxrwxrwx. 1 root root 7 Nov 15 20:05 /dev/mapper/mpatha -> ../dm-0
lrwxrwxrwx. 1 root root 7 Nov 15 20:05 /dev/mapper/mpatha1 -> ../dm-1
lrwxrwxrwx. 1 root root 7 Nov 15 20:05 /dev/mapper/mpatha2 -> ../dm-2

[dbwiddis]

OSHI reports disk statistics via udev "stat" key (same data as /sys/block/<dev>/stat).

At present I have zero experience or knowledge of how that field interacts with multipath or where else to get metrics.