Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rbd: get volumegroup in secondary cluster #5118

Open
wants to merge 1 commit into
base: devel
Choose a base branch
from

Conversation

iPraveenParihar
Copy link
Contributor

Describe what this PR does

Currently, GetVolumeGroup() fetches the RBD group from the pool using the clusterID & poolID encoded in the VolumeGroupHandle. However, this approach may fail in a secondary mirrored cluster, where the clusterID & poolID could differ.

This commit ensures that GetVolumeGroup leverages the clusterIDMapping and RBDPoolIDMapping to locate the RBD group in the appropriate pool if it is not found in the pool corresponding to the poolID encoded in the VolumeGroupHandle.

Future concerns

List items that are not part of the PR and do not impact it's
functionality, but are work items that can be taken up subsequently.

Checklist:

  • Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
  • Reviewed the developer guide on Submitting a Pull Request
  • Pending release notes updated with breaking and/or notable changes for the next major release.
  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • Integration tests have been added, if necessary.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

  • /retest ci/centos/<job-name>: retest the <job-name> after unrelated
    failure (please report the failure too!)

@mergify mergify bot added the component/rbd Issues related to RBD label Feb 3, 2025
@iPraveenParihar iPraveenParihar force-pushed the rbd/get-volumegroup-in-secondary-cluster branch from f511417 to a045d55 Compare February 3, 2025 10:36
Currently, `GetVolumeGroup()` fetches the RBD group from the
pool using the clusterID & poolID encoded in the VolumeGroupHandle.
However, this approach may fail in a secondary mirrored cluster,
where the clusterID & poolID could differ.

This commit ensures that `GetVolumeGroup` leverages the
clusterIDMapping and RBDPoolIDMapping to locate the RBD group in the
appropriate  pool if it is not found in the pool corresponding
to the poolID encoded in the VolumeGroupHandle.

Signed-off-by: Praveen M <[email protected]>
@iPraveenParihar iPraveenParihar force-pushed the rbd/get-volumegroup-in-secondary-cluster branch from a045d55 to 13817b7 Compare February 3, 2025 10:45
@iPraveenParihar
Copy link
Contributor Author

iPraveenParihar commented Feb 3, 2025

Tried relocation of VolumeGroupReplication resources -

  • Before
I0203 11:30:01.281170       1 utils.go:266] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC call: /replication.Controller/EnableVolumeReplication
I0203 11:30:01.281583       1 utils.go:267] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC request: {"parameters":{"schedulingInterval":"1m"},"replication_source":{"Type":{"Volumegroup":{"volume_group_id":"0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e"}}},"secrets":"***stripped***"}
I0203 11:30:01.285860       1 rbd_util.go:1274] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:30:01.285910       1 rbd_util.go:1287] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new poolID mapping 5 for existing pooID 2
E0203 11:30:01.289116       1 omap.go:80] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e omap not found (pool="replicapool", namespace="", name="csi.volume.a3b2910a-ae92-456e-951d-f11e5dd3437e"): rados: ret=-2, No such file or directory
W0203 11:30:01.289189       1 voljournal.go:737] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e unable to read omap keys: pool or key missing: key not found: rados: ret=-2, No such file or directory
E0203 11:30:01.295327       1 rbd_journal.go:715] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e failed to get image id replicapool/csi-vol-a3b2910a-ae92-456e-951d-f11e5dd3437e: Failed as image not found (internal RBD image not found: rbd: ret=-2, No such file or directory)
E0203 11:30:01.295462       1 utils.go:271] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC error: rpc error: code = NotFound desc = pool  not found for 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e: pool not found: pool ID(2) not found in Ceph cluster
I0203 11:30:01.326722       1 utils.go:266] ID: 29 GRPC call: /volumegroup.Controller/ModifyVolumeGroupMembership
I0203 11:30:01.327269       1 utils.go:267] ID: 29 GRPC request: {"secrets":"***stripped***","volume_group_id":"0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e","volume_ids":["0001-0009-rook-ceph-0000000000000002-df47962d-00b6-4078-a26d-6cecb8de0d65"]}
E0203 11:30:01.334576       1 utils.go:271] ID: 29 GRPC error: rpc error: code = NotFound desc = could not find volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e": failed to get volume group with id "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e": failed to initialize volume group with id "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e": failed to get pool for volume group id "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e": pool not found: pool ID(2) not found in Ceph cluster
  • After changes
I0203 11:35:28.678616       1 utils.go:266] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC call: /replication.Controller/EnableVolumeReplication
I0203 11:35:28.679538       1 utils.go:267] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC request: {"parameters":{"schedulingInterval":"1m"},"replication_source":{"Type":{"Volumegroup":{"volume_group_id":"0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e"}}},"secrets":"***stripped***"}
I0203 11:35:28.724266       1 util.go:125] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:28.724338       1 util.go:136] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new poolID mapping 5 for existing poolID 2
I0203 11:35:28.725095       1 util.go:201] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e object for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e" has been initialized
I0203 11:35:28.730288       1 omap.go:221] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e got omap values: (pool="replicapool", namespace="", name="csi.volume.group.a3b2910a-ae92-456e-951d-f11e5dd3437e"): map[0001-0009-rook-ceph-0000000000000002-df47962d-00b6-4078-a26d-6cecb8de0d65: csi.creationtime:2025-01-31T06:38:56.50611142Z csi.groupname:csi-vol-group-a3b2910a-ae92-456e-951d-f11e5dd3437e csi.volname:vgrcontent-d85282f5-e003-4c8a-9a64-0ba8cb4b8469]
I0203 11:35:28.732359       1 rbd_util.go:1257] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:28.732416       1 rbd_util.go:1270] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new poolID mapping 5 for existing pooID 2
I0203 11:35:28.734837       1 omap.go:89] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e got omap values: (pool="replicapool", namespace="", name="csi.volume.df47962d-00b6-4078-a26d-6cecb8de0d65"): map[csi.imageid:3768b4a33bf4 csi.imagename:csi-vol-df47962d-00b6-4078-a26d-6cecb8de0d65 csi.volname:pvc-180a6d69-58eb-4e96-972f-18142b68907e csi.volume.owner:rook-ceph]
I0203 11:35:28.851215       1 volume_group.go:116] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GetVolumeGroup(0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e) returns {commonVolumeGroup:{id:0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e requestName:vgrcontent-d85282f5-e003-4c8a-9a64-0ba8cb4b8469 name:csi-vol-group-a3b2910a-ae92-456e-951d-f11e5dd3437e creationTime:0xc00084cc78 clusterID:rook-ceph objectUUID:a3b2910a-ae92-456e-951d-f11e5dd3437e credentials:0xc00072a960 conn:<nil> ioctx:<nil> monitors:192.168.122.209:6789 pool:replicapool namespace: csiDriver:default journal:0xc000a14520} volumes:[0xc000038488] volumesToFree:[0xc000038488]}
I0203 11:35:28.854186       1 rbd_util.go:1257] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:28.854230       1 rbd_util.go:1270] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new poolID mapping 5 for existing pooID 2
I0203 11:35:28.858994       1 omap.go:89] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e got omap values: (pool="replicapool", namespace="", name="csi.volume.df47962d-00b6-4078-a26d-6cecb8de0d65"): map[csi.imageid:3768b4a33bf4 csi.imagename:csi-vol-df47962d-00b6-4078-a26d-6cecb8de0d65 csi.volname:pvc-180a6d69-58eb-4e96-972f-18142b68907e csi.volume.owner:rook-ceph]
W0203 11:35:28.963636       1 replication.go:160] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e mirroringMode is not set in parameters, setting to mirroringMode to default (snapshot)
I0203 11:35:28.963692       1 replication.go:139] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e "flattenMode" is not set in parameters, setting to default (never)
I0203 11:35:28.963995       1 util.go:339] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e connection established for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e"
I0203 11:35:28.964209       1 util.go:390] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e iocontext created for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e" in pool "replicapool"
I0203 11:35:28.967555       1 utils.go:273] ID: 25 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC response: {}
...
...
I0203 11:35:28.981948       1 utils.go:266] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC call: /replication.Controller/PromoteVolume
I0203 11:35:28.982809       1 utils.go:267] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC request: {"parameters":{"schedulingInterval":"1m"},"replication_source":{"Type":{"Volumegroup":{"volume_group_id":"0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e"}}},"secrets":"***stripped***"}
I0203 11:35:28.984847       1 util.go:125] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:28.984876       1 util.go:136] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new poolID mapping 5 for existing poolID 2
I0203 11:35:28.985503       1 util.go:201] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e object for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e" has been initialized
I0203 11:35:28.989595       1 omap.go:221] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e got omap values: (pool="replicapool", namespace="", name="csi.volume.group.a3b2910a-ae92-456e-951d-f11e5dd3437e"): map[0001-0009-rook-ceph-0000000000000002-df47962d-00b6-4078-a26d-6cecb8de0d65: csi.creationtime:2025-01-31T06:38:56.50611142Z csi.groupname:csi-vol-group-a3b2910a-ae92-456e-951d-f11e5dd3437e csi.volname:vgrcontent-d85282f5-e003-4c8a-9a64-0ba8cb4b8469]
I0203 11:35:28.994245       1 rbd_util.go:1257] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:28.994285       1 rbd_util.go:1270] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new poolID mapping 5 for existing pooID 2
I0203 11:35:29.001126       1 omap.go:89] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e got omap values: (pool="replicapool", namespace="", name="csi.volume.df47962d-00b6-4078-a26d-6cecb8de0d65"): map[csi.imageid:3768b4a33bf4 csi.imagename:csi-vol-df47962d-00b6-4078-a26d-6cecb8de0d65 csi.volname:pvc-180a6d69-58eb-4e96-972f-18142b68907e csi.volume.owner:rook-ceph]
I0203 11:35:29.085889       1 volume_group.go:116] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GetVolumeGroup(0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e) returns {commonVolumeGroup:{id:0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e requestName:vgrcontent-d85282f5-e003-4c8a-9a64-0ba8cb4b8469 name:csi-vol-group-a3b2910a-ae92-456e-951d-f11e5dd3437e creationTime:0xc00084d218 clusterID:rook-ceph objectUUID:a3b2910a-ae92-456e-951d-f11e5dd3437e credentials:0xc000815960 conn:<nil> ioctx:<nil> monitors:192.168.122.209:6789 pool:replicapool namespace: csiDriver:default journal:0xc000a159e0} volumes:[0xc000039b08] volumesToFree:[0xc000039b08]}
I0203 11:35:29.088270       1 rbd_util.go:1257] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:29.088395       1 rbd_util.go:1270] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new poolID mapping 5 for existing pooID 2
I0203 11:35:29.094509       1 omap.go:89] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e got omap values: (pool="replicapool", namespace="", name="csi.volume.df47962d-00b6-4078-a26d-6cecb8de0d65"): map[csi.imageid:3768b4a33bf4 csi.imagename:csi-vol-df47962d-00b6-4078-a26d-6cecb8de0d65 csi.volname:pvc-180a6d69-58eb-4e96-972f-18142b68907e csi.volume.owner:rook-ceph]
I0203 11:35:29.189939       1 util.go:339] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e connection established for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e"
I0203 11:35:29.189997       1 util.go:390] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e iocontext created for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e" in pool "replicapool"
I0203 11:35:29.214615       1 replication.go:476] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e Added scheduling at interval 1m, start time  for Id 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e
I0203 11:35:29.215239       1 utils.go:273] ID: 26 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC response: {}
...
...
I0203 11:35:29.062071       1 utils.go:266] ID: 27 GRPC call: /volumegroup.Controller/ModifyVolumeGroupMembership
I0203 11:35:29.064212       1 utils.go:267] ID: 27 GRPC request: {"secrets":"***stripped***","volume_group_id":"0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e","volume_ids":["0001-0009-rook-ceph-0000000000000002-df47962d-00b6-4078-a26d-6cecb8de0d65"]}
I0203 11:35:29.066962       1 util.go:125] ID: 27 found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:29.067214       1 util.go:136] ID: 27 found new poolID mapping 5 for existing poolID 2
I0203 11:35:29.067677       1 util.go:201] ID: 27 object for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e" has been initialized
I0203 11:35:29.070963       1 omap.go:221] ID: 27 got omap values: (pool="replicapool", namespace="", name="csi.volume.group.a3b2910a-ae92-456e-951d-f11e5dd3437e"): map[0001-0009-rook-ceph-0000000000000002-df47962d-00b6-4078-a26d-6cecb8de0d65: csi.creationtime:2025-01-31T06:38:56.50611142Z csi.groupname:csi-vol-group-a3b2910a-ae92-456e-951d-f11e5dd3437e csi.volname:vgrcontent-d85282f5-e003-4c8a-9a64-0ba8cb4b8469]
I0203 11:35:29.073851       1 rbd_util.go:1257] ID: 27 found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:29.073985       1 rbd_util.go:1270] ID: 27 found new poolID mapping 5 for existing pooID 2
I0203 11:35:29.076215       1 omap.go:89] ID: 27 got omap values: (pool="replicapool", namespace="", name="csi.volume.df47962d-00b6-4078-a26d-6cecb8de0d65"): map[csi.imageid:3768b4a33bf4 csi.imagename:csi-vol-df47962d-00b6-4078-a26d-6cecb8de0d65 csi.volname:pvc-180a6d69-58eb-4e96-972f-18142b68907e csi.volume.owner:rook-ceph]
I0203 11:35:29.140844       1 volume_group.go:116] ID: 27 GetVolumeGroup(0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e) returns {commonVolumeGroup:{id:0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e requestName:vgrcontent-d85282f5-e003-4c8a-9a64-0ba8cb4b8469 name:csi-vol-group-a3b2910a-ae92-456e-951d-f11e5dd3437e creationTime:0xc000437aa0 clusterID:rook-ceph objectUUID:a3b2910a-ae92-456e-951d-f11e5dd3437e credentials:0xc0004094e0 conn:<nil> ioctx:<nil> monitors:192.168.122.209:6789 pool:replicapool namespace: csiDriver:default journal:0xc00077ab70} volumes:[0xc000a76b48] volumesToFree:[0xc000a76b48]}
I0203 11:35:29.143816       1 util.go:125] ID: 27 found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:29.143946       1 util.go:136] ID: 27 found new poolID mapping 5 for existing poolID 2
I0203 11:35:29.144454       1 util.go:201] ID: 27 object for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e" has been initialized
I0203 11:35:29.150484       1 omap.go:221] ID: 27 got omap values: (pool="replicapool", namespace="", name="csi.volume.group.a3b2910a-ae92-456e-951d-f11e5dd3437e"): map[0001-0009-rook-ceph-0000000000000002-df47962d-00b6-4078-a26d-6cecb8de0d65: csi.creationtime:2025-01-31T06:38:56.50611142Z csi.groupname:csi-vol-group-a3b2910a-ae92-456e-951d-f11e5dd3437e csi.volname:vgrcontent-d85282f5-e003-4c8a-9a64-0ba8cb4b8469]
I0203 11:35:29.152605       1 rbd_util.go:1257] ID: 27 found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:29.153595       1 rbd_util.go:1270] ID: 27 found new poolID mapping 5 for existing pooID 2
I0203 11:35:29.157216       1 omap.go:89] ID: 27 got omap values: (pool="replicapool", namespace="", name="csi.volume.df47962d-00b6-4078-a26d-6cecb8de0d65"): map[csi.imageid:3768b4a33bf4 csi.imagename:csi-vol-df47962d-00b6-4078-a26d-6cecb8de0d65 csi.volname:pvc-180a6d69-58eb-4e96-972f-18142b68907e csi.volume.owner:rook-ceph]
I0203 11:35:29.235656       1 volume_group.go:116] ID: 27 GetVolumeGroup(0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e) returns {commonVolumeGroup:{id:0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e requestName:vgrcontent-d85282f5-e003-4c8a-9a64-0ba8cb4b8469 name:csi-vol-group-a3b2910a-ae92-456e-951d-f11e5dd3437e creationTime:0xc00084dbc0 clusterID:rook-ceph objectUUID:a3b2910a-ae92-456e-951d-f11e5dd3437e credentials:0xc0004094e0 conn:<nil> ioctx:<nil> monitors:192.168.122.209:6789 pool:replicapool namespace: csiDriver:default journal:0xc00065e8f0} volumes:[0xc0007e2d88] volumesToFree:[0xc0007e2d88]}
I0203 11:35:29.237277       1 rbd_util.go:1257] ID: 27 found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:29.237394       1 rbd_util.go:1270] ID: 27 found new poolID mapping 5 for existing pooID 2
I0203 11:35:29.239790       1 omap.go:89] ID: 27 got omap values: (pool="replicapool", namespace="", name="csi.volume.df47962d-00b6-4078-a26d-6cecb8de0d65"): map[csi.imageid:3768b4a33bf4 csi.imagename:csi-vol-df47962d-00b6-4078-a26d-6cecb8de0d65 csi.volname:pvc-180a6d69-58eb-4e96-972f-18142b68907e csi.volume.owner:rook-ceph]
I0203 11:35:29.298298       1 util.go:339] ID: 27 connection established for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e"
I0203 11:35:29.298582       1 util.go:390] ID: 27 iocontext created for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e" in pool "replicapool"
I0203 11:35:29.305227       1 replication.go:139] ID: 27 "flattenMode" is not set in parameters, setting to default (never)
I0203 11:35:29.305728       1 utils.go:273] ID: 27 GRPC response: {"volume_group":{"volume_group_id":"0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e","volumes":[{"capacity_bytes":1073741824,"volume_context":{"imageName":"csi-vol-df47962d-00b6-4078-a26d-6cecb8de0d65","journalPool":"replicapool","pool":"replicapool"},"volume_id":"0001-0009-rook-ceph-0000000000000002-df47962d-00b6-4078-a26d-6cecb8de0d65"}]}}
...
...
I0203 11:35:29.227208       1 utils.go:266] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC call: /replication.Controller/GetVolumeReplicationInfo
I0203 11:35:29.227391       1 utils.go:267] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC request: {"replication_source":{"Type":{"Volumegroup":{"volume_group_id":"0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e"}}},"secrets":"***stripped***"}
I0203 11:35:29.229642       1 util.go:125] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:29.229680       1 util.go:136] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new poolID mapping 5 for existing poolID 2
I0203 11:35:29.230496       1 util.go:201] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e object for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e" has been initialized
I0203 11:35:29.232514       1 omap.go:221] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e got omap values: (pool="replicapool", namespace="", name="csi.volume.group.a3b2910a-ae92-456e-951d-f11e5dd3437e"): map[0001-0009-rook-ceph-0000000000000002-df47962d-00b6-4078-a26d-6cecb8de0d65: csi.creationtime:2025-01-31T06:38:56.50611142Z csi.groupname:csi-vol-group-a3b2910a-ae92-456e-951d-f11e5dd3437e csi.volname:vgrcontent-d85282f5-e003-4c8a-9a64-0ba8cb4b8469]
I0203 11:35:29.237704       1 rbd_util.go:1257] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:29.237811       1 rbd_util.go:1270] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new poolID mapping 5 for existing pooID 2
I0203 11:35:29.244772       1 omap.go:89] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e got omap values: (pool="replicapool", namespace="", name="csi.volume.df47962d-00b6-4078-a26d-6cecb8de0d65"): map[csi.imageid:3768b4a33bf4 csi.imagename:csi-vol-df47962d-00b6-4078-a26d-6cecb8de0d65 csi.volname:pvc-180a6d69-58eb-4e96-972f-18142b68907e csi.volume.owner:rook-ceph]
I0203 11:35:29.327090       1 volume_group.go:116] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GetVolumeGroup(0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e) returns {commonVolumeGroup:{id:0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e requestName:vgrcontent-d85282f5-e003-4c8a-9a64-0ba8cb4b8469 name:csi-vol-group-a3b2910a-ae92-456e-951d-f11e5dd3437e creationTime:0xc000603260 clusterID:rook-ceph objectUUID:a3b2910a-ae92-456e-951d-f11e5dd3437e credentials:0xc00072b220 conn:<nil> ioctx:<nil> monitors:192.168.122.209:6789 pool:replicapool namespace: csiDriver:default journal:0xc000988e90} volumes:[0xc000810fc8] volumesToFree:[0xc000810fc8]}
I0203 11:35:29.329456       1 rbd_util.go:1257] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new clusterID mapping rook-ceph for existing clusterID rook-ceph
I0203 11:35:29.329506       1 rbd_util.go:1270] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e found new poolID mapping 5 for existing pooID 2
I0203 11:35:29.332065       1 omap.go:89] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e got omap values: (pool="replicapool", namespace="", name="csi.volume.df47962d-00b6-4078-a26d-6cecb8de0d65"): map[csi.imageid:3768b4a33bf4 csi.imagename:csi-vol-df47962d-00b6-4078-a26d-6cecb8de0d65 csi.volname:pvc-180a6d69-58eb-4e96-972f-18142b68907e csi.volume.owner:rook-ceph]
I0203 11:35:29.396844       1 util.go:339] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e connection established for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e"
I0203 11:35:29.396901       1 util.go:390] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e iocontext created for volume group "0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e" in pool "replicapool"
I0203 11:35:29.400927       1 replication.go:879] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e mirrorStatus: {GlobalMirrorGroupStatus:0xc000517e00}
I0203 11:35:29.401079       1 group_mirror.go:291] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e Site status of MirrorUUID: , state: stopped, description: remote group is non-primary, lastUpdate: 1738582505, up: true
I0203 11:35:29.401116       1 group_mirror.go:291] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e Site status of MirrorUUID: 55717782-910d-42a0-bcc7-8e28614d1c21, state: stopped, description: remote group is non-primary, lastUpdate: 1738582508, up: true
I0203 11:35:29.401195       1 group_mirror.go:353] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e description: {"local_snapshot_timestamp":0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0}
I0203 11:35:29.401217       1 group_mirror.go:354] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e description: {0 0 0xc00051e2d0}
I0203 11:35:29.401259       1 replication.go:928] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e Madhu description: remote group is non-primary, {"local_snapshot_timestamp":0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0}
I0203 11:35:29.401300       1 replication.go:945] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e Madhu the description after unmarshalling is {0 0 0xc00051e3d0}
E0203 11:35:29.401328       1 replication.go:894] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e failed to parse last sync info from "remote group is non-primary, {\"local_snapshot_timestamp\":0,\"last_snapshot_bytes\":0,\"last_snapshot_sync_seconds\":0}": empty local snapshot timestamp: last sync time not found
E0203 11:35:29.401648       1 utils.go:271] ID: 28 Req-ID: 0001-0009-rook-ceph-0000000000000002-a3b2910a-ae92-456e-951d-f11e5dd3437e GRPC error: rpc error: code = NotFound desc = failed to get last sync info: empty local snapshot timestamp: last sync time not found

cc @Madhu-1 @Nikhil-Ladha

@iPraveenParihar iPraveenParihar marked this pull request as ready for review February 3, 2025 11:38
}

vg, err := generateVolumeGroup(id, csiDriver, creds, csiID)
if err != nil && !errors.Is(err, util.ErrPoolNotFound) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to make use of the following function instead of singular pool not found check ?

func shouldRetryVolumeGeneration(err error) bool {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/rbd Issues related to RBD
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants