Fix Velero backup timeout on preparing data upload

[nickatnceas]

I'm seeing a new backup error on k8s-dev, here's one example from velero backup describe full-test-1 --details:

  Operation for persistentvolumeclaims vegbank/vegbankdb-cnpg-2:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-201b6530-55e5-459d-ae29-a885efee0151.05a689f8-dae0-46d8fc0fe
    Items to Update:
                           datauploads.velero.io velero/full-test-1-pbmbw
    Phase:                 Failed
    Operation Error:       timeout on preparing data upload
    Progress description:  Failed
    Created:               2025-10-17 10:58:09 -0700 PDT

    vegbank/vegbankdb-cnpg-2:
      Data Movement:
        Operation ID: du-201b6530-55e5-459d-ae29-a885efee0151.05a689f8-dae0-46d8fc0fe
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 0
        Result: failed

[nickatnceas]

The timeout appears to be 30 minutes.

This might be the issue and fix: https://velero.io/docs/v1.16/data-movement-backup-pvc-configuration/

CephFS does make a full copy of data when making a RW clone. If we change the config to make RO clones instead it should be nearly instant.

[mbjones]

What is the difference between a RW and RO clone? Can the RO clone be used to do a restore that becomes RW?

[nickatnceas]

Based on the comment at vmware-tanzu/velero#7747 (comment) the clone is only needed to be RO.

The main difference is that CephFS makes a full copy of data when making a RW clone, which could take us hours/days on large volumes.

[nickatnceas]

I made the following changes:

backuppvc.json

{
    "backupPVC": {
        "csi-cephfs-sc": {
            "readOnly": true
        },
        "csi-cephfs-sc-ephemeral": {
            "readOnly": true
        }
    }
}

$ kubectl create cm backuppvc -n velero --from-file backupPVC.json

Added - --node-agent-configmap=backuppvc

$ kubectl edit ds node-agent -n velero

spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      name: node-agent
  template:
    metadata:
      creationTimestamp: null
      labels:
        component: velero
        name: node-agent
        role: node-agent
    spec:
      containers:
      - args:
        - node-agent
        - server
        - --features=EnableCSI
        - --node-agent-configmap=backuppvc

After some trial and error the backup timeout error are now gone:

NAME                                 STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
full-test-4                          PartiallyFailed   1        0          2025-10-17 15:39:10 -0700 PDT   29d       default            <none>
full-backup-20251017030055           Failed            0        0          2025-10-16 20:00:55 -0700 PDT   89d       default            <none>
full-backup-20251016030053           PartiallyFailed   15       0          2025-10-15 20:00:53 -0700 PDT   88d       default            <none>

[nickatnceas]

Multiple timeout errors exist on k8s-prod, I'm going to make the same changes there.

[nickatnceas]

Made the changes on k8s-prod and ran a test backup.