Error on backend creation with tridentctl using qnap-iscsi storage driver

[4rkDev]

Error on backend creation with tridentctl using qnap-iscsi storage driver

Issue Overview

When attempting to create a backend using the tridentctl command, I encounter an error that suggests a login failure with the qnap-iscsi storage driver. The command used and the error received are as follows:

Command

./bin/linux-amd64/tridentctl create backend -f Samples/backend.json -n trident

Error Message

Error: could not create backend: problem initializing storage driver 'qnap-iscsi': rpc error: code = Unknown desc = login failed; please check if storage is online (400 Bad Request) command terminated with exit code 1

Backend Configuration File (backend.json)

{
  "version": 1,
  "storageDriverName": "qnap-iscsi",
  "backendName": "proxmox",
  "storageAddress": "192.168.150.1",
  "username": "admin",
  "password": "Mypassword",
  "debugTraceFlags": { "method": true },
  "storage": [
    {
      "labels": { "storage": "proxmox" },
      "serviceLevel": "Any"
    },
    {
      "labels": { "performance": "basic" },
      "features": {
        "tiering": "Disable"
      },
      "serviceLevel": "Non-Tiering"
    }
  ]
}

Steps to Reproduce

1. Ensure the storage system is online and accessible at the address 192.168.150.1.
2. Verify the credentials are correct and have the necessary permissions.
3. Execute the tridentctl command as shown above.

Expected Behavior

The backend should be created successfully without any errors.

Actual Behavior

The command fails indicating a login failure with the storage system.

[pkerwien]

@4rkDev Are you using a virtual switch on your QNAP NAS? I'm seeing issues with that and cannot get version 1.2.0 to work properly when using a virtual switch. When uninstalling Virtualization Station and removing the virtual switch, my problems disappear. (Haven't filed an issue ticket yet).

[4rkDev]

@pkerwien thank you for the information. I had the virtual switch enabled on my QNAP NAS. Following your suggestion, I've disabled it, but unfortunately, I am still facing the same error. I appreciate any other ideas or suggestions you might have.

[pkerwien]

@4rkDev I will make more experiements later today. I'm using the trident CRDs and no longer run tridentctl, so I don't know if that could be related. It shouldn't. They are both just creating the necessary configuration in k8s. But I haven't tried tridentctl. I will also see what happens if I have a working system without a virtual switch, add it and see what happens when it is removed.

Which QNAP firmware are you using?

[4rkDev]

I've also tried directly with the yaml and it doesn't work for me either; I get the same error. I have been able to verify that the errors that the trident controller was giving since I deleted the virtual switch have disappeared. I am getting this error, which is the same one I got from the tridentctl, here something more verbose:

time="2024-05-03T09:55:05Z" level=debug msg="Initializing storage driver." crdControllerEvent=update driver=qnap-iscsi logLayer=core requestID=5e54f327-0d52-4af7-97eb-2293fdcd0e43 requestSource=CRD workflow="cr=reconcile"
time="2024-05-03T09:55:05Z" level=error msg="Could not initialize storage driver." crdControllerEvent=update error="rpc error: code = Unknown desc = login failed; please check if storage is online" logLayer=core requestID=5e54f327-0d52-4af7-97eb-2293fdcd0e43 requestSource=CRD
time="2024-05-03T09:55:05Z" level=debug msg="Failed storage backend." backendName=proxmox backendUUID= crdControllerEvent=update driver=qnap-iscsi logLayer=core requestID=5e54f327-0d52-4af7-97eb-2293fdcd0e43 requestSource=CRD workflow="cr=reconcile"
time="2024-05-03T09:55:05Z" level=debug msg="NewStorageBackendForConfig failed." backend="&{0xc000cade60 proxmox 97e7b001-86ad-4d72-ba3c-52332218867f false failed map[] map[] b9291b32-d491-48af-90c4-6343452ee045 false}" backendUUID=97e7b001-86ad-4d72-ba3c-52332218867f configR
time="2024-05-03T09:55:05Z" level=warning msg="Cannot terminate an uninitialized backend." backend=proxmox backendUUID=97e7b001-86ad-4d72-ba3c-52332218867f crdControllerEvent=update driver=qnap-iscsi logLayer=core requestID=5e54f327-0d52-4af7-97eb-2293fdcd0e43 requestSource=C
time="2024-05-03T09:55:05Z" level=info msg="deferred syncing TridentBackendConfig 'trident/backend-qts', requeuing; problem initializing storage driver 'qnap-iscsi': rpc error: code = Unknown desc = login failed; please check if storage is online" crdControllerEvent=update lo
time="2024-05-03T09:55:08Z" level=debug msg="Storage prefix is absent, will use default prefix." crdControllerEvent=update logLayer=core requestID=5e54f327-0d52-4af7-97eb-2293fdcd0e43 requestSource=CRD workflow="cr=reconcile"
time="2024-05-03T09:55:08Z" level=debug msg="Credentials field not empty." crdControllerEvent=update logLayer=core requestID=5e54f327-0d52-4af7-97eb-2293fdcd0e43 requestSource=CRD workflow="cr=reconcile"
time="2024-05-03T09:55:08Z" level=debug msg="Parsed commonConfig: Version:1 StorageDriverName:\"qnap-iscsi\" BackendName:\"proxmox\" Debug:false DebugTraceFlags:map[string]bool{\"method\":false} DisableDelete:false StoragePrefixRaw:json.RawMessage(nil) StoragePrefix:(*string)
time="2024-05-03T09:55:08Z" level=debug msg="Retrieved backend secret." crdControllerEvent=update logLayer=core requestID=5e54f327-0d52-4af7-97eb-2293fdcd0e43 requestSource=CRD workflow="cr=reconcile"

I am using the firmware version QuTS hero h5.1.5.2680.

[pkerwien]

@4rkDev Unfortunately QuTS hero doesn't work and is not supported. But it would be really great if QNAP could add QuTS hero support as well.

@JimmyTanMPM Any plans to support QuTS hero in future releases?

[LeonaChen2727]

Support for QuTS hero is currently under development.

[louhisuo]

+1 for QuTS hero support.
Any estimate when QNAP CSI Plugin is compatible with QuTS hero which I am using?

[LeonaChen2727]

We are pleased to inform you that QNAP CSI Plugin v1.3.0 has been released today.
This new version now supports QuTS hero.

We appreciate your continued support.

[4rkDev]

Thank you very much for the update! I'm glad to hear that version 1.3.0 of the QNAP CSI Plugin now supports QuTS hero. I really appreciate the hard work and effort from the team to include this feature. I will test it next week and provide feedback on how it goes.

[pkerwien]

I have made some quick initial testing using QuTS hero h5.2.0.2860 and RKE2 v1.28.12. Created an nginx pod with a PVC. Everything works so far without issues. I will continue later and test snapshots and other features.

[kgh02017]

I am facing the same problem. I am using ContainerStation, but is it possible for them to coexist?
I am not using a virtual switch, but I do have a ethernet bridge.

[davidcheng0716]

hi, @kgh02017

I need more information to identify the issue,
(1) What's your firmware version ?
(2) the storage can be accessed from the cluster ?

It will be helpful if you can provide your network adapters and backend config.

[kgh02017]

(1) QuTS hero h5.2.0.2860
(2) Yes

• backend config (JSON)
  {
  "version": 1,
  "storageDriverName": "qnap-iscsi",
  "backendName": "qnap-ts473a",
  "storageAddress": "xx.xx.xx.xx",
  "username": "xxx",
  "password": "xxxxxxxxxxxxx",
  "debugTraceFlags": {"method":true},
  "storage": [
  {
  "labels": {"storage": "qnap-ts473a"},
  "serviceLevel": "Any"
  }
  ]
  }

• Network Adapter
  

[davidcheng0716]

@kgh02017

It should be work, can you do the following steps to make sure the ip is accessible?

(In NAS)
(1) create a iscsi lun and target

(In Cluster)
(2) iscsiadm -m discovery -t st -p "storage IP"
(3) iscsiadm -m node -T "target IQN" -p "storage IP" --login
(4) iscsiadm -m session

btw, the error message is the same as "login failed", right? please double check the ip, user and password, thx.

[kgh02017]

(In NAS)
(1) create a iscsi lun and target

Should I create a LUN in advance ?

(In Cluster)
(2) iscsiadm -m discovery -t st -p "storage IP"

$ sudo iscsiadm -m discovery -t st -p xx,xx,xx,xx
xx.xx.xx.xx:3260,1 iqn.2004-04.com.qnap:ts-473a:iscsi.target-ocp

(3) iscsiadm -m node -T "target IQN" -p "storage IP" --login

$ sudo iscsiadm -m node -T iqn.2004-04.com.qnap:ts-473a:iscsi.target-ocp -p xx.xx.xx.xx --login
Logging in to [iface: default, target: iqn.2004-04.com.qnap:ts-473a:iscsi.target-ocp, portal: xx.xx.xx.xx,3260]
iscsiadm: Could not login to [iface: default, target: iqn.2004-04.com.qnap:ts-473a:iscsi.target-ocp, portal: xx.xx.xx.xx,3260].
iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
iscsiadm: Could not log into all portals

Where to pass uid/password ?
Should I put uid/password to iscsid.conf in advance ?

[kgh02017]

btw, the error message is the same as "login failed", right? please double check the ip, user and password, thx.

$ ./tridentctl create backend -f backend-qnap-ts473a.json -n trident
Error: could not create backend: problem initializing storage driver 'qnap-iscsi': rpc error: code = Unknown desc = login failed; please check if storage is online (400 Bad Request)
command terminated with exit code 1
Error: exit status 1

[4rkDev]

Issue: Login Failed During Trident Backend Initialization

I am experiencing an issue while initializing the Trident backend using the qnap-iscsi driver for QNAP storage. Despite verifying the user credentials and confirming that the storage network is correctly configured, I continue to receive the following error:

Error Logs:

trident-main time="2024-09-04T08:54:32Z" level=debug msg="<<<< Initialize" Method=Initialize Type=QTSDriver crdControllerEvent=add logLayer=core requestID=780123a7-2630-4b3e-9138-79ed35143fbe requestSource=CRD workflow="cr=reconcile"
trident-main time="2024-09-04T08:54:32Z" level=error msg="Could not initialize storage driver." crdControllerEvent=add error="rpc error: code = Unknown desc = login failed; please check if storage is online" logLayer=core requestID=780123a7-2630-4b3e-9138-79ed35143fbe requestSource=CRD workflow="cr=reconcile"
trident-main time="2024-09-04T08:54:32Z" level=debug msg="Failed storage backend." backendName=hero backendUUID= crdControllerEvent=add driver=qnap-iscsi logLayer=core requestID=780123a7-2630-4b3e-9138-79ed35143fbe requestSource=CRD workflow="cr=reconcile"
trident-main time="2024-09-04T08:54:32Z" level=warning msg="Cannot terminate an uninitialized backend." backend=hero backendUUID=37f0d553-9569-4117-8c77-15de1482e5d0 crdControllerEvent=add driver=qnap-iscsi logLayer=core requestID=780123a7-2630-4b3e-9138-79ed35143fbe requestSource=CRD workflow="cr=reconcile"
trident-main time="2024-09-04T08:54:32Z" level=info msg="deferred syncing TridentBackendConfig 'trident/backend-hero', requeuing; reconcile deferred; problem initializing storage driver 'qnap-iscsi': rpc error: code = Unknown desc = login failed; please check if storage is online"

Backend Configuration:

The backend configuration is applied as follows (the username and password are obfuscated for security):

apiVersion: v1
kind: Secret
metadata:
  name: backend-hero
  namespace: trident
type: Opaque
stringData:
  username: <obfuscated>
  password: <obfuscated>
  storageAddress: 192.168.150.1
---
apiVersion: trident.qnap.io/v1
kind: TridentBackendConfig
metadata:
  name: backend-hero
  namespace: trident
spec:
  version: 1
  storageDriverName: qnap-iscsi
  backendName: hero
  credentials:
    name: backend-hero
  debugTraceFlags:
    method: true
  storage:
    - serviceLevel: Any
      labels:
        performance: any

Details:

• The storage system does respond to pings, confirming that it is online and reachable on the network.
• Both the username and password have been verified and work when used directly for login.
• The network adapter shows a 25 Gbps connection with an MTU of 9000, as seen in the screenshot below:

![Network Adapter [Configuration]]

Steps Taken:

1. I have ensured that the network settings for the storage backend are properly configured and reachable from the Trident controller.
2. I have confirmed the validity of the credentials used in the Secret for backend-hero.
3. I attempted to re-initialize the backend, but the same error persists.

It seems that Trident is unable to establish a successful connection despite the storage being online and accessible via network tools. Any assistance or insights on resolving this issue would be greatly appreciated.

[patriot81]

Try to deactivate:
"Force Secure Connection (HTTPS): After enabling this option, you can only connect and log into the NAS using HTTPS."

[kgh02017]

@patriot81

I tried to do, but no luck.