Partition of nvme as metadata device of hdd

[zhucan]

At present, only can use a whole piece nvme or hdd device as metadata device of hdd to create osd， the nvme disk evenly partitioned by rook; but we want to use different partition of multi nvme device as metadata device of hdd;

1. First partition all Nvme disks
2. Let the metadata device of the hdd disk specify the partition

My means. We can specify metadata of hdd, not evenly partitioned by rook.

[travisn]

To use partitions, you could follow the cluster-on-pvc example. As long as you define the local PVs with the partitions, this should work.

[travisn]

@zhucan If not using PVCs, it is only possible to use a single metadata device for all the OSDs on a node.

[zhucan]

is it possible to support it? @travisn

[travisn]

@zhucan What exactly do you want to support? You want to specify a metadata partition for a single OSD instead of the whole disk? That would first require a change to ceph-volume, you would need to open a ceph tracker for that.

[zhucan]

@travisn yes, we want to specify a metadata partition for a single OSD instead of the whole disk. I think "ceph-volume" has support it.

[travisn]

@zhucan If you can find the ceph-volume command for using the existing metadata partition, it would be helpful to consider supporting this in rook. I know others are interested in the feature too.

[zhucan]

and other question is: the nvme name is different from different node.

[travisn]

The metadataDevice name can be different if defining the nodes in the Cluster CR.

[microyahoo]

Rook currently only supports one metadata device per node, and then it is divided into multiple partitions, one for each osd. If one of the osd disks is broken, then I replace the disk with good one and add it back to the cluster, or remove it from the cluster and then add one good disk back again, what state will the cluster be in at this time? Can this osd automatically use the previous metadata device partition? Thanks. @travisn

[microyahoo]

Are there any plans to improve? @travisn @satoru-takeuchi

[satoru-takeuchi]

At least I have no plan.

[zhucan]

1. Partition the disk first
2. In yaml, specify partitions for each data disk
3. create osd

I think it is more in line with actual usage scenarios and more convenient for operation and maintenance. @travisn @satoru-takeuchi

[travisn]

Certainly it is more desirable, but it is just not implemented today for non-pvc clusters.

[travisn]

#13240 opened to discuss a possible fix for this