Snapshotting CephFS not working

[LarsKumbier]

Hey everyone,
I'm currently working on backup and restore for a Rook Ceph installation. In order to have a minimal timeframe of changes within and between the 3 storage nodes, I'd like to snapshot the PVs.
I've worked through the CephFS snapshotting documentation and installed the CRDs for VolumeSnapshotClasses, VolumeSnapshotContents and VolumeSnapshots from the Kubernetes CSI External Snapshotter Repo and added the Snapshotter RBAC and Controller.
Next, I've added the SnapshotClass, customized for my namespace:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-cephfsplugin-snapclass
driver: rook-ceph-storage-standard.cephfs.csi.ceph.com # csi-provisioner-name
parameters:
  clusterID: rook-ceph-storage-standard # namespace:cluster
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph-storage-standard # namespace:cluster
deletionPolicy: Delete

So far so good. But if I create a snapshot of some demo-PVC, it does not work. Example:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: lars-backup-claim
spec:
  storageClassName: storage-standard
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: lars-backup-snapshot
spec:
  volumeSnapshotClassName: csi-cephfsplugin-snapclass
  source:
    persistentVolumeClaimName: lars-backup-claim

The snapshot stays on NotReady, with the event store waiting for the CSI driver:

# kubectl get volumesnapshot lars-backup-snapshot
NAME                   READYTOUSE   SOURCEPVC           SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS                SNAPSHOTCONTENT                                    CREATIONTIME   AGE
lars-backup-snapshot   false        lars-backup-claim                                         csi-cephfsplugin-snapclass   snapcontent-b052fa6e-af84-40aa-ae9a-9126064f7b92                  17h

# kubectl describe volumesnapshot lars-backup-snapshot
Name:         lars-backup-snapshot
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2024-03-20T16:30:18Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generation:        1
  Resource Version:  3536440
  UID:               b052fa6e-af84-40aa-ae9a-9126064f7b92
Spec:
  Source:
    Persistent Volume Claim Name:  lars-backup-claim
  Volume Snapshot Class Name:      csi-cephfsplugin-snapclass
Status:
  Bound Volume Snapshot Content Name:  snapcontent-b052fa6e-af84-40aa-ae9a-9126064f7b92
  Ready To Use:                        false
Events:                                <none>

I've attached the cephfs-provisioner log, where I can see a lot being done, but no warnings or errors.

I've double-checked that the provisioner-secret is correctly found (by putting in a typo and seeing an appropriate error message in the provisioner log). Beside that, I'm unsure how to proceed debugging this issue.

Any help appreciated.

[Madhu-1]

@LarsKumbier i dont see any errors in the logs.

• what version of cephcsi and Rook you are using?
• Can you also provide volumesnapshotcontent yaml output
• Provide snapshotter controller logs
• provide csi-cephfsplugin and csi-snapshotter container logs from both cephfs provisioner pods

Make sure you have set the log levels to 5 for both

rook/deploy/examples/operator.yaml

Lines 61 to 67 in b3b6775

	# Set logging level for cephCSI containers maintained by the cephCSI.
	# Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.
	# CSI_LOG_LEVEL: "0"
	# Set logging level for Kubernetes-csi sidecar containers.
	# Supported values from 0 to 5. 0 for general useful logs (the default), 5 for trace level verbosity.
	# CSI_SIDECAR_LOG_LEVEL: "0"

[LarsKumbier]

Hey Madhu,

thanks for answering.

• Rook is in version v1.12.8, Ceph is in version v18.2.0
• volumesnapshotcontents.yaml
• the logs from my previous post actually were from the current snapshot leader - apologies.

The csi-snapshotter logs was a good tip, it does not find the volumesnapshotcontents CRD:

W0320 04:01:29.645056       1 reflector.go:424] github.com/kubernetes-csi/external-snapshotter/client/v6/informers/externalversions/factory.go:117: failed to list *v1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io)
E0320 04:01:29.645083       1 reflector.go:140] github.com/kubernetes-csi/external-snapshotter/client/v6/informers/externalversions/factory.go:117: Failed to watch *v1.VolumeSnapshotContent: failed to list *v1.VolumeSnapshotContent: the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io)

But the CRD can be retrieved in the namespace:

# kubectl -n rook-ceph get crd volumesnapshotcontents.snapshot.storage.k8s.io
NAME                                             CREATED AT
volumesnapshotcontents.snapshot.storage.k8s.io   2024-03-20T12:39:40Z

I'll redeploy the operator with the hightened log level and get back to you.