[RFE]: Mounting multiple GlusterFS volumes fails if referencing one or more missing volumes on the server.

[sky-philipalmeida]

Is your feature request related to a problem? Please describe.
We recently bumped into an issue that took us some time to identify and it maybe affect production deploys in the future.
We had 92 kadalustorages objects to deploy each one with it's own dedicate gluster volume.
One of the objects was referencing a missing Gluster volume on the server.

Kadalu provisioner mounted the missing Gluster volume and by doing so it crashed and was not able to mount unmounted volumes.
The behaviour was not deterministic depending on ordering of the missing volume and there was no clear error message that would guide us on the correct path quick.

Describe the solution you'd like
Prevent Kadalu from trying to updating provisioner if volume does not exist (or any other error).
Prior to mount the volume test for volume presence and have a clear error message stating the volume was not mounted because it does not exist.

Replicate issue
Create multiple kadalustorages objects referencing diferent volumes.
Make sure at least one or more volumes do not exist on the server.
After creating the objects go into the provisioner shell (kadalu-csi-provisioner-0) and do a df -h . This command should hang.
In folder /mnt doing a ls | xargs -L 1 -d '\n' ls $0 would also hang but gives a chance to identify which mount is failing. It will only list volumes mounted correclly then it will hang on volumes not accessible.

[leelavg]

We had 92 kadalustorages objects to deploy each one with it's own dedicate gluster volume.

• Is this correct? Each kds CR is configured in External mode w/ 1 PV : 1 Gluster Volume.

Kadalu provisioner mounted the missing Gluster volume

• ack, need to check it, iirc we see if the gluster volume is reachable before mounting it at some places maybe we need to check that everywhere or we aren't handling errors properly

[sky-philipalmeida]

Yes its correct, as soon I create the missing volume they all mount.