[Question] Fence_kubevirt STONITH failure when storage LAN fails.

[HideoYamauchi]

Hi All,

Please tell me how fence_kubevirt works.

I have built a Pacemaker cluster on OCP-V.
The ACT/STB virtual machines are placed on different Compute nodes, and the virtual machines are stored in storage (NFS) on a storage LAN different from the management LAN.

Step 1) Disconnect the NFS-LAN of the Compute node where the ACT virtual machine is running.

Step 2) STB detects the ACT failure, and fence_kubevirt's STONITH is processed.
On the Web console, the target virtual machine transitions from Stopping to Stopped, and appears to have stopped correctly.

Step 3) However, fence_kubevirt does not succeed because the virtual machine instance remains (Failed in oc get vmi).

[root@rh95-02 ~]# crm_mon -rfA1
Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: rh95-02 (version 2.1.8-3.el9-3980678f0) - partition with quorum
  * Last updated: Thu May 29 12:03:03 2025 on rh95-02
  * Last change:  Thu May 29 12:01:11 2025 by hacluster via hacluster on rh95-02
  * 2 nodes configured
  * 10 resource instances configured

Node List:
  * Node rh95-01: UNCLEAN (offline)
  * Online: [ rh95-02 ]

Full List of Resources:
  * Clone Set: pgsql-clone [pgsql] (promotable):
    * pgsql     (ocf:linuxhajp:pgsql):   Unpromoted rh95-01 (UNCLEAN)
    * Unpromoted: [ rh95-02 ]
  * Resource Group: primary-group:
    * ipaddr-primary    (ocf:heartbeat:IPaddr2):         Started rh95-01 (UNCLEAN)
    * ipaddr-replication        (ocf:heartbeat:IPaddr2):         Started rh95-01 (UNCLEAN)
  * Clone Set: ping-clone [ping]:
    * Started: [ rh95-02 ]
    * Stopped: [ rh95-01 ]
  * Clone Set: storage-mon-clone [storage-mon]:
    * Started: [ rh95-02 ]
    * Stopped: [ rh95-01 ]
  * fence1-kubevirt     (stonith:fence_kubevirt):        Started rh95-02
  * fence2-kubevirt     (stonith:fence_kubevirt):        Stopped

Node Attributes:
  * Node: rh95-02:
    * master-pgsql                      : 100
    * pgsql-data-status                 : STREAMING|SYNC
    * pgsql-master-baseline             : 0000000015000060
    * pgsql-status                      : HS:alone
    * ping-status                       : 1

Migration Summary:

Failed Fencing Actions:
  * reboot of rh95-01 by rh95-02 for pacemaker-controld.69801@rh95-02 last failed at 2025-05-29 12:03:03.038906 +09:00
  * reboot of rh95-01 by rh95-02 for pacemaker-controld.69801@rh95-02 last failed at 2025-05-29 12:03:01.356906 +09:00
  * reboot of rh95-01 by rh95-02 for pacemaker-controld.69801@rh95-02 last failed at 2025-05-29 12:02:59.685906 +09:00
(snip)
Pending Fencing Actions:
  * reboot of rh95-01 for pacemaker-controld.69801@rh95-02 last pending
[root@rh95-02 ~]#

[nttossc@dn-ocp-bastion ~]$ oc get vmi
NAME            AGE   PHASE     IP            NODENAME       READY
ocp-rh95-01     26h   Failed    10.130.2.24   dn-ocp-cmp01   False
ocp-rh95-02     26h   Running   10.130.0.86   dn-ocp-cmp02   True
ocp-rh95-ping   8d    Running   10.131.0.42   dn-ocp-cmp03   True
[nttossc@dn-ocp-bastion ~]$

I understand that the above STONITH failure is as specified (failed because the virtual machine instance remains), but is that correct?
Also, I believe that this control (deletion of virtual machine instance) is managed by the OCP side and cannot be handled by fence_kubevirt.

(snip)
def get_power_status(conn, options):
    logging.debug("Starting get status operation")
    try:
        apiversion = options.get("--apiversion")
        namespace = _get_namespace(options)
        name = options.get("--plug")
        vmi_api = conn.resources.get(api_version=apiversion,
                                              kind='VirtualMachineInstance')
        vmi = vmi_api.get(name=name, namespace=namespace)
        return translate_status(vmi.status.phase)
    except ApiException as e:
        if e.status == 404:
            try:
                vm_api = conn.resources.get(api_version=apiversion, kind='VirtualMachine')
                vm = vm_api.get(name=name, namespace=namespace)
            except ApiException as e:
                logging.error("VM %s doesn't exist", name)
                fail(EC_FETCH_VM_UUID)
            return "off"
        logging.error("Failed to get power status, with API Exception: %s", e)
        fail(EC_STATUS)
    except Exception as e:
        logging.error("Failed to get power status, with Exception: %s", e)
        fail(EC_STATUS)

def translate_status(instance_status):
    if instance_status == "Running":
        return "on"
    return "unknown"
(snip)

Best Regards,
Hideo Yamauchi.

[oalbrigt]

Can you try updating the if() to if instance_status == "Running" or instance_status == "Failed": to see if that works?

[HideoYamauchi]

Hi @oalbrigt

Understood.
I will check and get back to you tomorrow.

Many thanks,
Hideo Yamauchi.

[oalbrigt]

You might have to add a logger.info() in get_power_status() if it's not working out of the box. I just guessed that "Failed" is the result it returns.

[HideoYamauchi]

You might have to add a logger.info() in get_power_status() if it's not working out of the box. I just guessed that "Failed" is the result returns.

Okay!

[HideoYamauchi]

Hi @oalbrigt

Made the following changes.

def translate_status(instance_status):
    #if instance_status == "Running":
    #print(instance_status)
    if instance_status == "Running" or instance_status == "Failed":
        return "on"
    return "unknown"

If you look closely, you will see that this time, after STONITH is executed, it appears as Stopped on the Web console, but oc get vmi shows the virtual machine targeted for STONITH as Failed, so STONITH does not complete.
Even if you make this change, Failed is considered to be in the on state, so STONITH will not complete.

---

I have also included the following fix, but it appears that gracePeriod=0 is not effective for a virtual machine with a failed storage LAN.

• fence_kubevirt: set grace_period_seconds=0 to immediately power off the node #616

It takes time for a virtual machine with a failed storage LAN to stop (for the display in the Web console to change from Stoppin to Stopped).

def virtctl_vm_action(conn, action, namespace, name, apiversion):
    path = '/apis/subresources.{api_version}/namespaces/{namespace}/virtualmachines/{name}/{action}'
    path = path.format(api_version=apiversion, namespace=namespace, name=name, action=action)
    #return conn.request('put', path, header_params={'accept': '*/*'})
    return conn.request('put', path, header_params={'accept': '*/*'}, body={'gracePeriod': 0} if action == 'stop' else None)

Best Regards,
Hideo Yamauchi.

[oalbrigt]

Is there still a virt-launcher Pod for the failed VMI?

[HideoYamauchi]

Hi @oalbrigt

Is there still a virt-launcher Pod for the failed VMI?

The lancher POD of ocp-rh95-01x, which is the target of STONITH, appears to remain in Terminating mode.

STONITH was executed on ocp-rh95-01x, and the Web console still displayed the Stopping status as shown below.

[nttossc@dn-ocp-bastion ~]$ oc get vmi
NAME                                 AGE    PHASE     IP             NODENAME       READY
ocp-rh95-01x                         8h     Running   10.129.4.10    dn-ocp-cmp02   False
ocp-rh95-02x                         5d8h   Running   10.130.2.153   dn-ocp-cmp01   True
ocp-rh95-ping                        6d9h   Running   10.130.2.157   dn-ocp-cmp01   True
rh95-test-cpu16-mem12                2d     Running   10.130.2.156   dn-ocp-cmp01   True
rhel9-ivory-antelope-71-vcp8-mem16   8h     Running   10.131.0.127   dn-ocp-cmp03   True
[nttossc@dn-ocp-bastion ~]$ oc get pods
NAME                                                     READY   STATUS        RESTARTS   AGE
virt-launcher-ocp-rh95-01x-wcn7m                         1/1     Terminating   0          8h
virt-launcher-ocp-rh95-02x-2n5lz                         1/1     Running       0          5d8h
virt-launcher-ocp-rh95-ping-chwx6                        0/1     Completed     0          6d9h
virt-launcher-ocp-rh95-ping-mjqh4                        1/1     Running       0          34h
virt-launcher-rh95-test-cpu16-mem12-zrr68                1/1     Running       0          2d
virt-launcher-rhel9-ivory-antelope-71-vcp8-mem16-r4lxv   1/1     Running       0          8h
[nttossc@dn-ocp-bastion ~]$

For ocp-rh95-01x, STONITH was failing and the Web console displayed the status as Stopped, as shown below.

[nttossc@dn-ocp-bastion ~]$ oc get vmi
NAME                                 AGE    PHASE     IP             NODENAME       READY
ocp-rh95-01x                         8h     Failed    10.129.4.10    dn-ocp-cmp02   False
ocp-rh95-02x                         5d8h   Running   10.130.2.153   dn-ocp-cmp01   True
ocp-rh95-ping                        6d9h   Running   10.130.2.157   dn-ocp-cmp01   True
rh95-test-cpu16-mem12                2d     Running   10.130.2.156   dn-ocp-cmp01   True
rhel9-ivory-antelope-71-vcp8-mem16   8h     Running   10.131.0.127   dn-ocp-cmp03   True
[nttossc@dn-ocp-bastion ~]$ oc get pods
NAME                                                     READY   STATUS        RESTARTS   AGE
virt-launcher-ocp-rh95-01x-wcn7m                         1/1     Terminating   0          8h
virt-launcher-ocp-rh95-02x-2n5lz                         1/1     Running       0          5d8h
virt-launcher-ocp-rh95-ping-chwx6                        0/1     Completed     0          6d9h
virt-launcher-ocp-rh95-ping-mjqh4                        1/1     Running       0          34h
virt-launcher-rh95-test-cpu16-mem12-zrr68                1/1     Running       0          2d
virt-launcher-rhel9-ivory-antelope-71-vcp8-mem16-r4lxv   1/1     Running       0          8h
[nttossc@dn-ocp-bastion ~]$

Best Regards,
Hideo Yamauchi.

[oalbrigt]

We're not able to get the status of storage LAN via the API currently, so I'm in discussions with some Openshift devs to consider adding it.

[HideoYamauchi]

Hi @oalbrigt

Thanks for your comment.
I understand.

I'll wait for future OCP-V updates to detect storage failures.
For now, I'll keep this issue open.

Best Regards,
Hideo Yamauchi.