OBS - Error mounting volume in OpenShift

[phalamaj-sendati]

Hello,

We are trying to use the obs-csi-driver on an OpenShift Cluster ( version 4.16.27 ) in Huawei Cloud and we have an error when mounting the PV in the nodes.

We were able to deploy the solution using the instructions in the repo ( https://github.com/huaweicloud/huaweicloud-csi-driver/blob/master/docs/obs/obs.md ) , with just two minor changes:

--- csi-obs-node.yaml	2025-01-15 22:23:20.875559848 +0800
+++ csi-obs-node-ocp.yaml	2025-01-23 01:01:41.248251660 +0800
@@ -20,6 +20,7 @@
       tolerations:
         - operator: "Exists"
       hostPID: true
+      hostNetwork: true
       containers:
         - name: liveness-probe
           volumeMounts:
@@ -29,7 +30,7 @@
           args:
             - --csi-address=/csi/csi.sock
             - --probe-timeout=3s
-            - --health-port=28889
+            - --health-port=28890
             - --v=5
         - name: node-driver-registrar
           image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0
@@ -63,7 +64,7 @@
               exec:
                 command: [ "/bin/sh", "-c", "/obs-csi/stop-server.sh" ]
           ports:
-            - containerPort: 28889
+            - containerPort: 28890
               name: healthz
               protocol: TCP
           livenessProbe:

• The change in the port is to avoid conflict with csi driver of SFS-turbo
• And the change "hostNetwork: true" is to make the pods able to see the metadata of the ECS instances at http://169.254.169.254.

After all of this we have the service class and the dinamic provisioning working good, but the problem raise when OpenShift tries to link the PV with the node to start the pod.

at the deployment of the pod we find the following events.

15m         Warning   FailedMount              pod/frontend-deployment-7467fd4fbc-skl6d    MountVolume.SetUp failed for volume "pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75" : rpc error: code = Internal desc = Failed to mount pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 at /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount: rpc error: code = Internal desc = Failed to execute the command, body: {"Data":"failed to mount CMD: obsfs obsfs pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount -o url=obs.la-south-2.myhuaweicloud.com -o passwd_file=/var/lib/csi/5385b829-fcf8-4d64-bda8-d80a896c28f2 -o big_writes -o max_write=131072 -o use_ino, output: obsfs: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory\n, error: exit status 127"}
15m         Warning   FailedMount              pod/frontend-deployment-7467fd4fbc-skl6d    MountVolume.SetUp failed for volume "pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75" : rpc error: code = Internal desc = Failed to mount pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 at /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount: rpc error: code = Internal desc = Failed to execute the command, body: {"Data":"failed to mount CMD: obsfs obsfs pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount -o url=obs.la-south-2.myhuaweicloud.com -o passwd_file=/var/lib/csi/63a07564-c231-4327-bcb6-c88c2be67b72 -o big_writes -o max_write=131072 -o use_ino, output: obsfs: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory\n, error: exit status 127"}
15m         Warning   FailedMount              pod/frontend-deployment-7467fd4fbc-skl6d    MountVolume.SetUp failed for volume "pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75" : rpc error: code = Internal desc = Failed to mount pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 at /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount: rpc error: code = Internal desc = Failed to execute the command, body: {"Data":"failed to mount CMD: obsfs obsfs pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount -o url=obs.la-south-2.myhuaweicloud.com -o passwd_file=/var/lib/csi/56233375-2a1e-4676-bf39-dd005e520929 -o big_writes -o max_write=131072 -o use_ino, output: obsfs: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory\n, error: exit status 127"}
13m         Warning   FailedMount              pod/frontend-deployment-7467fd4fbc-skl6d    MountVolume.SetUp failed for volume "pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75" : rpc error: code = Internal desc = Failed to mount pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 at /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount: rpc error: code = Internal desc = Failed to execute the command, body: {"Data":"failed to mount CMD: obsfs obsfs pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount -o url=obs.la-south-2.myhuaweicloud.com -o passwd_file=/var/lib/csi/0f8f5742-ead1-4040-86d6-c14de7dce86b -o big_writes -o max_write=131072 -o use_ino, output: obsfs: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory\n, error: exit status 127"}
104s        Warning   FailedMount              pod/frontend-deployment-7467fd4fbc-skl6d    (combined from similar events): MountVolume.SetUp failed for volume "pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75" : rpc error: code = Internal desc = Failed to mount pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 at /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount: rpc error: code = Internal desc = Failed to execute the command, body: {"Data":"failed to mount CMD: obsfs obsfs pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75 /var/lib/kubelet/pods/8d0ed9e7-8c8e-42d1-be1c-f6dcf82c7971/volumes/kubernetes.io~csi/pvc-e9bbcb03-6e76-4674-96a5-dcaf79e3cb75/mount -o url=obs.la-south-2.myhuaweicloud.com -o passwd_file=/var/lib/csi/dd094d70-0e14-49c9-aa63-9fe8e6cdef1e -o big_writes -o max_write=131072 -o use_ino, output: obsfs: error while loading shared libraries: libcrypto.so.10: cannot open shared object file: No such file or directory\n, error: exit status 127"}

The worker node has the following OS:

[core@host-192-168-0-197 ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux CoreOS release 4.16
[core@host-192-168-0-197 ~]$ sudo find / -iname libcrypto.so.10
find: File system loop detected; ‘/sysroot/ostree/deploy/rhcos/deploy/3f79e452161ec82aadee7a6212efb89f9e520de82eeb9b6c24b7c8a6efadc3c3.1’ is part of the same file system loop as ‘/’.
/sysroot/ostree/deploy/rhcos/var/lib/containers/storage/overlay/89169d87dbe2b72ba42bfbb3579c957322baca28e03a1e558076542a1c1b2b4a/diff/usr/lib64/libcrypto.so.10
/sysroot/ostree/deploy/rhcos/var/lib/containers/storage/overlay/174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02/diff/usr/lib64/libcrypto.so.10
/sysroot/ostree/deploy/rhcos/var/lib/containers/storage/overlay/be1f7344519a2b9b41aa7154581b1b5503c2168abb3bc1edc99b656dca2426fe/diff/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/97356b4db3a0a55ab2528e77773b338614bfc7ca5515f467f18135bc5da28fb1/merged/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/89169d87dbe2b72ba42bfbb3579c957322baca28e03a1e558076542a1c1b2b4a/diff/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02/diff/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/91f312abe1f58eee97401eca3d34be22abf081f6f6fb29a8a77607ce42d262df/merged/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/be1f7344519a2b9b41aa7154581b1b5503c2168abb3bc1edc99b656dca2426fe/diff/usr/lib64/libcrypto.so.10
/var/lib/containers/storage/overlay/8fd1a9c0df259a554ab0fa91c2b3200f845c280b35b5b1959febd00f9c24f35f/merged/usr/lib64/libcrypto.so.10
[core@host-192-168-0-197 ~]$ 

Attach you can find the definitions of the pod, the PV and the PVC in the Manifest.tgz ( tar gzipped ) file.

Best