submariner-k8s-broker-admin-dockercfg is continuously getting populated multiple times

[BhavaniYalamanchili]

Hi Support Team,

We are observing multiple secrets getting populated in submariner-k8s-broker on site 1. The submariner version is 0.18.0.

History of this is, earlier we have seen that submariner operator logs was like below

ERR ..oller/controller.go:324 Reconciler error error="error building an authorized RestConfig for the broker: cannot access the API server "[https://api.hv1-ocp-kn.kbs.drv:6443\](https://api.hv1-ocp-kn.kbs.drv:6443%5C)": Get "[https://api.hv1-ocp-kn.kbs.drv:6443/api/v1/namespaces/submariner-k8s-broker/secrets/any\](https://api.hv1-ocp-kn.kbs.drv:6443/api/v1/namespaces/submariner-k8s-broker/secrets/any%5C)": tls: failed to verify certificate: x509: certificate signed by unknown authority" Submariner={"name":"submariner","namespace":"submariner-operator"} controller=submariner-controller controllerGroup=submariner.io controllerKind=Submariner name=submariner namespace=submariner-operator

we have tried deleting the broker secret on both sites, and waited some time, they didn't come up, tried to reinstall the submariner, but reinstall failed

message: 'Get "https://api.hv1-ocp-kn.kbs.drv:6443/api/v1/nodes?labelSelector=node-role.kubernetes.io%2Fmaster":
tls: failed to verify certificate: x509: certificate signed by unknown authority'

we checked the kn custom-ca.crt and recreated it, and tried reinstalling the submariner, with --check-broker-certificate=false flag, the submariner was working fine, but found a misbehaviour that multiple secrets on the site 1 are populated continuously.

FROM SITE 1

SUBCTL SHOW ALL

# KN Side
sh-5.1$ /root/.local/bin/subctl show all --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
 ✓ Detecting broker(s)
NAMESPACE               NAME                COMPONENTS                        GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS
submariner-k8s-broker   submariner-broker   service-discovery, connectivity   no          242.0.0.0/8      65536

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP     NAT   CABLE DRIVER   SUBNETS                           STATUS      RTT avg.
control-1-ru3.hv1-ocp-la.kbs.d   site2     10.23.49.13   no    libreswan      192.168.128.0/18, 172.18.0.0/16   connected   540.806µs

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP   PUBLIC IP     CABLE DRIVER   TYPE
site1     10.23.48.12   10.23.48.12   libreswan      local
site1     10.23.48.13   10.23.48.13   libreswan      local
site1     10.23.48.14   10.23.48.14   libreswan      local
site2     10.23.49.13   10.23.49.13   libreswan      remote

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY
control-1-ru2.hv1-ocp-kn.kbs.d   passive     There are no connections
control-1-ru3.hv1-ocp-kn.kbs.d   passive     There are no connections
control-1-ru4.hv1-ocp-kn.kbs.d   active      All connections (1) are established

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [192.168.64.0/18]
        Cluster CIDRs:   [172.17.0.0/16]

 ✓ Showing versions
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH
submariner-gateway              quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64
submariner-routeagent           quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64
submariner-metrics-proxy        quay.io/submariner   0.18.0       release-0.18-011349c6f17e   amd64
submariner-operator             quay.io/submariner   0.18.0       release-0.18-68fefdd74105   amd64
submariner-lighthouse-agent     quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64
submariner-lighthouse-coredns   quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64


# LA from KN side

sh-5.1$ /root/.local/bin/subctl show all --kubeconfig /connection/kube-config/16c2029d3c/kubeconfig
Cluster "default-cluster"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP     NAT   CABLE DRIVER   SUBNETS                          STATUS      RTT avg.
control-1-ru4.hv1-ocp-kn.kbs.d   site1     10.23.48.14   no    libreswan      192.168.64.0/18, 172.17.0.0/16   connected   622.124µs

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP   PUBLIC IP     CABLE DRIVER   TYPE
site2     10.23.49.12   10.23.49.12   libreswan      local
site2     10.23.49.13   10.23.49.13   libreswan      local
site1     10.23.48.14   10.23.48.14   libreswan      remote
site2     10.23.49.14   10.23.49.14   libreswan      local

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY
control-1-ru2.hv1-ocp-la.kbs.d   passive     There are no connections
control-1-ru3.hv1-ocp-la.kbs.d   active      All connections (1) are established
control-1-ru4.hv1-ocp-la.kbs.d   passive     There are no connections

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [192.168.128.0/18]
        Cluster CIDRs:   [172.18.0.0/16]

 ✓ Showing versions
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH
submariner-gateway              quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64
submariner-routeagent           quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64
submariner-metrics-proxy        quay.io/submariner   0.18.0       release-0.18-011349c6f17e   amd64
submariner-operator             quay.io/submariner   0.18.0       release-0.18-68fefdd74105   amd64
submariner-lighthouse-agent     quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64
submariner-lighthouse-coredns   quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

SUBCTL DIAGNOSE ALL

# KN Side
sh-5.1$ /root/.local/bin/subctl diagnose all --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.27.13+fd36fb9" is supported

 ✗ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✗ Error getting the Broker's REST config: error getting auth rest config: cannot access the API server "https://api.hv1-ocp-kn.kbs.drv:6443": Get "https://api.hv1-ocp-kn.kbs.drv:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: certificate signed by unknown authority

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✓ Checking OVN version
 ✓ The ovn-nb database version 7.1.0 is supported
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ Cluster is running with "OVNKubernetes" CNI which internally implements kube-proxy functionality
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.18.0


# LA from KN side

sh-5.1$ /root/.local/bin/subctl diagnose all --kubeconfig /connection/kube-config/16c2029d3c/kubeconfig
Cluster "default-cluster"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.27.13+fd36fb9" is supported

 ✗ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✗ Error getting the Broker's REST config: error getting auth rest config: cannot access the API server "https://api.hv1-ocp-kn.kbs.drv:6443": Get "https://api.hv1-ocp-kn.kbs.drv:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: certificate signed by unknown authority

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✓ Checking OVN version
 ✓ The ovn-nb database version 7.1.0 is supported
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ Cluster is running with "OVNKubernetes" CNI which internally implements kube-proxy functionality
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.18.0

SUBCTL GATHER

# KN Side
sh-5.1$ /root/.local/bin/subctl gather --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
Gathering information from cluster "local-config"
 ✓ Gathering operator logs
 ✓ Found 1 pods matching label selector "name=submariner-operator"
 ✓ Gathering operator resources
 ✓ Found 1 submariners in namespace "submariner-operator"
 ✓ Found 1 servicediscoveries in namespace "submariner-operator"
 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
 ✓ Gathering connectivity logs
 ✓ Found 3 pods matching label selector "app=submariner-gateway"
 ✓ Found 7 pods matching label selector "app=submariner-routeagent"
 ✓ Found 3 pods matching label selector "app=submariner-metrics-proxy"
 ✓ Found 0 pods matching label selector "app=submariner-globalnet"
 ✓ Found 0 pods matching label selector "app=submariner-addon"
 ✓ Gathering connectivity resources
 ✓ Gathering CNI data from 7 pods matching label selector "app=submariner-routeagent"
 ✓ Gathering CNI data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering cable driver data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-9dmbv"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-ct5xq"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-j5lmv"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-jcj7w"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-kmlh9"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-mhqrc"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-zgb6x"
 ✓ Found 1 gatewayroutes in namespace ""
 ✓ Found 1 nongatewayroutes in namespace ""
 ✓ Found 2 endpoints in namespace "submariner-operator"
 ✓ Found 2 clusters in namespace "submariner-operator"
 ✓ Found 3 gateways in namespace "submariner-operator"
 ✓ Found 0 clusterglobalegressips in namespace ""
 ✓ Found 0 globalegressips in namespace ""
 ✓ Found 0 globalingressips in namespace ""
 ✓ Gathering service-discovery logs
 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
 ✓ Found 7 pods matching label selector "dns.operator.openshift.io/daemonset-dns=default"
 ✓ Gathering service-discovery resources
 ✓ Found 0 serviceexports in namespace ""
 ✓ Found 0 serviceimports in namespace ""
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
 ✓ Found 1 configmaps by field selector "metadata.name=dns-default" in namespace "openshift-dns"
 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
 ✓ Gathering broker logs
 ✗ Gathering broker resources
 ✗ Error getting the broker's rest config: error getting auth rest config: cannot access the API server "https://api.hv1-ocp-kn.kbs.drv:6443": Get "https://api.hv1-ocp-kn.kbs.drv:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: certificate signed by unknown authority
Files are stored under directory "submariner-20250718135438/local-config"

Encountered following Kubernetes warnings while running:
Warning: Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.


# LA from KN side
sh-5.1$ /root/.local/bin/subctl gather --kubeconfig /connection/kube-config/16c2029d3c/kubeconfig
Cluster "default-cluster"
Gathering information from cluster "default-cluster"
 ✓ Gathering broker logs
 ✗ Gathering broker resources
 ✗ Error getting the broker's rest config: error getting auth rest config: cannot access the API server "https://api.hv1-ocp-kn.kbs.drv:6443": Get "https://api.hv1-ocp-kn.kbs.drv:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: certificate signed by unknown authority
 ✓ Gathering operator logs
 ✓ Found 1 pods matching label selector "name=submariner-operator"
 ✓ Gathering operator resources
 ✓ Found 1 submariners in namespace "submariner-operator"
 ✓ Found 1 servicediscoveries in namespace "submariner-operator"
 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
 ✓ Gathering connectivity logs
 ✓ Found 3 pods matching label selector "app=submariner-gateway"
 ✓ Found 7 pods matching label selector "app=submariner-routeagent"
 ✓ Found 3 pods matching label selector "app=submariner-metrics-proxy"
 ✓ Found 0 pods matching label selector "app=submariner-globalnet"
 ✓ Found 0 pods matching label selector "app=submariner-addon"
 ✓ Gathering connectivity resources
 ✓ Gathering CNI data from 7 pods matching label selector "app=submariner-routeagent"
 ✓ Gathering CNI data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering cable driver data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-75l67"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-8lt9l"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-bg4s8"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-drdwn"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-g9fmr"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-h9tbm"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-p69hr"
 ✓ Found 1 gatewayroutes in namespace ""
 ✓ Found 1 nongatewayroutes in namespace ""
 ✓ Found 2 endpoints in namespace "submariner-operator"
 ✓ Found 2 clusters in namespace "submariner-operator"
 ✓ Found 3 gateways in namespace "submariner-operator"
 ✓ Found 0 clusterglobalegressips in namespace ""
 ✓ Found 0 globalegressips in namespace ""
 ✓ Found 0 globalingressips in namespace ""
 ✓ Gathering service-discovery logs
 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
 ✓ Found 7 pods matching label selector "dns.operator.openshift.io/daemonset-dns=default"
 ✓ Gathering service-discovery resources
 ✓ Found 0 serviceexports in namespace ""
 ✓ Found 0 serviceimports in namespace ""
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
 ✓ Found 1 configmaps by field selector "metadata.name=dns-default" in namespace "openshift-dns"
 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
Files are stored under directory "submariner-20250718135611/default-cluster"

FROM SITE 2

SUBCTL SHOW ALL

# LA side
sh-5.1$ /root/.local/bin/subctl show all --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP     NAT   CABLE DRIVER   SUBNETS                          STATUS      RTT avg.
control-1-ru4.hv1-ocp-kn.kbs.d   site1     10.23.48.14   no    libreswan      192.168.64.0/18, 172.17.0.0/16   connected   591.112µs

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP   PUBLIC IP     CABLE DRIVER   TYPE
site2     10.23.49.12   10.23.49.12   libreswan      local
site2     10.23.49.13   10.23.49.13   libreswan      local
site1     10.23.48.14   10.23.48.14   libreswan      remote
site2     10.23.49.14   10.23.49.14   libreswan      local

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY
control-1-ru2.hv1-ocp-la.kbs.d   passive     There are no connections
control-1-ru3.hv1-ocp-la.kbs.d   active      All connections (1) are established
control-1-ru4.hv1-ocp-la.kbs.d   passive     There are no connections

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [192.168.128.0/18]
        Cluster CIDRs:   [172.18.0.0/16]

 ✓ Showing versions
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH
submariner-gateway              quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64
submariner-routeagent           quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64
submariner-metrics-proxy        quay.io/submariner   0.18.0       release-0.18-011349c6f17e   amd64
submariner-operator             quay.io/submariner   0.18.0       release-0.18-68fefdd74105   amd64
submariner-lighthouse-agent     quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64
submariner-lighthouse-coredns   quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

# KN from LA side

sh-5.1$ /root/.local/bin/subctl show all --kubeconfig /connection/kube-config/4856ad23e1/kubeconfig
Cluster "default-cluster"
 ✓ Detecting broker(s)
NAMESPACE               NAME                COMPONENTS                        GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS
submariner-k8s-broker   submariner-broker   service-discovery, connectivity   no          242.0.0.0/8      65536

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP     NAT   CABLE DRIVER   SUBNETS                           STATUS      RTT avg.
control-1-ru3.hv1-ocp-la.kbs.d   site2     10.23.49.13   no    libreswan      192.168.128.0/18, 172.18.0.0/16   connected   545.032µs

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP   PUBLIC IP     CABLE DRIVER   TYPE
site1     10.23.48.12   10.23.48.12   libreswan      local
site1     10.23.48.13   10.23.48.13   libreswan      local
site1     10.23.48.14   10.23.48.14   libreswan      local
site2     10.23.49.13   10.23.49.13   libreswan      remote

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY
control-1-ru2.hv1-ocp-kn.kbs.d   passive     There are no connections
control-1-ru3.hv1-ocp-kn.kbs.d   passive     There are no connections
control-1-ru4.hv1-ocp-kn.kbs.d   active      All connections (1) are established

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [192.168.64.0/18]
        Cluster CIDRs:   [172.17.0.0/16]

 ✓ Showing versions
COMPONENT                       REPOSITORY           CONFIGURED   RUNNING                     ARCH
submariner-gateway              quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64
submariner-routeagent           quay.io/submariner   0.18.0       release-0.18-e3f3e56b57fe   amd64
submariner-metrics-proxy        quay.io/submariner   0.18.0       release-0.18-011349c6f17e   amd64
submariner-operator             quay.io/submariner   0.18.0       release-0.18-68fefdd74105   amd64
submariner-lighthouse-agent     quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64
submariner-lighthouse-coredns   quay.io/submariner   0.18.0       release-0.18-02b6a5b37266   amd64

SUBCTL DIAGNOSE ALL

# LA side
sh-5.1$ /root/.local/bin/subctl diagnose all --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.27.13+fd36fb9" is supported

 ✗ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✗ Error getting the Broker's REST config: error getting auth rest config: cannot access the API server "https://api.hv1-ocp-kn.kbs.drv:6443": Get "https://api.hv1-ocp-kn.kbs.drv:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: certificate signed by unknown authority

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✓ Checking OVN version
 ✓ The ovn-nb database version 7.1.0 is supported
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ Cluster is running with "OVNKubernetes" CNI which internally implements kube-proxy functionality
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.18.0


# KN from LA side
sh-5.1$ /root/.local/bin/subctl diagnose all --kubeconfig /connection/kube-config/4856ad23e1/kubeconfig
Cluster "default-cluster"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.27.13+fd36fb9" is supported

 ✗ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✗ Error getting the Broker's REST config: error getting auth rest config: cannot access the API server "https://api.hv1-ocp-kn.kbs.drv:6443": Get "https://api.hv1-ocp-kn.kbs.drv:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: certificate signed by unknown authority

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✓ Checking OVN version
 ✓ The ovn-nb database version 7.1.0 is supported
 ✓ Checking gateway connections
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ Cluster is running with "OVNKubernetes" CNI which internally implements kube-proxy functionality
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic

 ✓ Checking that services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.18.0

SUBCTL GATHER

# LA Side
/root/.local/bin/subctl gather --kubeconfig /tmp/local-kubeconfig
Cluster "local-config"
Gathering information from cluster "local-config"
 ✓ Gathering broker logs
 ✗ Gathering broker resources
 ✗ Error getting the broker's rest config: error getting auth rest config: cannot access the API server "https://api.hv1-ocp-kn.kbs.drv:6443": Get "https://api.hv1-ocp-kn.kbs.drv:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: certificate signed by unknown authority
 ✓ Gathering operator logs
 ✓ Found 1 pods matching label selector "name=submariner-operator"
 ✓ Gathering operator resources
 ✓ Found 1 submariners in namespace "submariner-operator"
 ✓ Found 1 servicediscoveries in namespace "submariner-operator"
 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
 ✓ Gathering connectivity logs
 ✓ Found 3 pods matching label selector "app=submariner-gateway"
 ✓ Found 7 pods matching label selector "app=submariner-routeagent"
 ✓ Found 3 pods matching label selector "app=submariner-metrics-proxy"
 ✓ Found 0 pods matching label selector "app=submariner-globalnet"
 ✓ Found 0 pods matching label selector "app=submariner-addon"
 ✓ Gathering connectivity resources
 ✓ Gathering CNI data from 7 pods matching label selector "app=submariner-routeagent"
 ✓ Gathering CNI data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering cable driver data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-75l67"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-8lt9l"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-bg4s8"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-drdwn"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-g9fmr"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-h9tbm"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-p69hr"
 ✓ Found 1 gatewayroutes in namespace ""
 ✓ Found 1 nongatewayroutes in namespace ""
 ✓ Found 2 endpoints in namespace "submariner-operator"
 ✓ Found 2 clusters in namespace "submariner-operator"
 ✓ Found 3 gateways in namespace "submariner-operator"
 ✓ Found 0 clusterglobalegressips in namespace ""
 ✓ Found 0 globalegressips in namespace ""
 ✓ Found 0 globalingressips in namespace ""
 ✓ Gathering service-discovery logs
 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
 ✓ Found 7 pods matching label selector "dns.operator.openshift.io/daemonset-dns=default"
 ✓ Gathering service-discovery resources
 ✓ Found 0 serviceexports in namespace ""
 ✓ Found 0 serviceimports in namespace ""
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
 ✓ Found 1 configmaps by field selector "metadata.name=dns-default" in namespace "openshift-dns"
 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
Files are stored under directory "submariner-20250718133842/local-config"

Encountered following Kubernetes warnings while running:
Warning: Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.



# KN from LA side
sh-5.1$ /root/.local/bin/subctl gather --kubeconfig /connection/kube-config/4856ad23e1/kubeconfig
Cluster "default-cluster"
Gathering information from cluster "default-cluster"
 ✓ Gathering connectivity logs
 ✓ Found 3 pods matching label selector "app=submariner-gateway"
 ✓ Found 7 pods matching label selector "app=submariner-routeagent"
 ✓ Found 3 pods matching label selector "app=submariner-metrics-proxy"
 ✓ Found 0 pods matching label selector "app=submariner-globalnet"
 ✓ Found 0 pods matching label selector "app=submariner-addon"
 ✓ Gathering connectivity resources
 ✓ Gathering CNI data from 7 pods matching label selector "app=submariner-routeagent"
 ✓ Gathering CNI data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering cable driver data from 3 pods matching label selector "app=submariner-gateway"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-9dmbv"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-ct5xq"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-j5lmv"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-jcj7w"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-kmlh9"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-mhqrc"
 ✓ Gathering OVN data from OVN kube pod "ovnkube-node-zgb6x"
 ✓ Found 1 gatewayroutes in namespace ""
 ✓ Found 1 nongatewayroutes in namespace ""
 ✓ Found 2 endpoints in namespace "submariner-operator"
 ✓ Found 2 clusters in namespace "submariner-operator"
 ✓ Found 3 gateways in namespace "submariner-operator"
 ✓ Found 0 clusterglobalegressips in namespace ""
 ✓ Found 0 globalegressips in namespace ""
 ✓ Found 0 globalingressips in namespace ""
 ✓ Gathering service-discovery logs
 ✓ Found 3 pods matching label selector "component=submariner-lighthouse"
 ✓ Found 7 pods matching label selector "dns.operator.openshift.io/daemonset-dns=default"
 ✓ Gathering service-discovery resources
 ✓ Found 0 serviceexports in namespace ""
 ✓ Found 0 serviceimports in namespace ""
 ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace ""
 ✓ Found 1 configmaps by label selector "component=submariner-lighthouse" in namespace "submariner-operator"
 ✓ Found 1 configmaps by field selector "metadata.name=dns-default" in namespace "openshift-dns"
 ✓ Found 0 services by label selector "submariner.io/exportedServiceRef" in namespace ""
 ✓ Gathering broker logs
 ✗ Gathering broker resources
 ✗ Error getting the broker's rest config: error getting auth rest config: cannot access the API server "https://api.hv1-ocp-kn.kbs.drv:6443": Get "https://api.hv1-ocp-kn.kbs.drv:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: certificate signed by unknown authority
 ✓ Gathering operator logs
 ✓ Found 1 pods matching label selector "name=submariner-operator"
 ✓ Gathering operator resources
 ✓ Found 1 submariners in namespace "submariner-operator"
 ✓ Found 1 servicediscoveries in namespace "submariner-operator"
 ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-metrics-proxy" in namespace "submariner-operator"
 ✓ Found 1 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator"
 ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator"
 ✓ Found 1 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator"
Files are stored under directory "submariner-20250718134120/default-cluster"

We are bit concerned about sharing logs, would need time to get the fully collected logs. by then please let us know if you need any specific logs.

[tpantelis]

I did a search for the error message:

The error "kubernetes tls: failed to verify certificate: x509: certificate signed by unknown authority" indicates that the client attempting to connect to a Kubernetes API server cannot validate the server's TLS certificate because the Certificate Authority (CA) that signed it is not trusted by the client.

Common causes and solutions:

Self-Signed Certificates or Untrusted CA:

• Cause: The Kubernetes cluster uses self-signed certificates or certificates issued by a private CA that is not inherently trusted by the client machine.
• Solution:
  • Add the CA certificate to the client's trusted root store: Export the root CA certificate from your Kubernetes cluster and import it into the trusted root certificate store of the machine where you are running kubectl or other Kubernetes clients. The process for adding certificates varies by operating system.
  • Configure clients to trust the specific CA: Some tools allow specifying a custom CA certificate to trust. For example, you might configure a curl command with --cacert /path/to/ca.crt.

Expired Certificates:

• Cause: The TLS certificates used by the Kubernetes API server or the client's certificates have expired.
• Solution: Check the expiration dates of the relevant certificates and renew them if necessary. Kubernetes components often have automated certificate rotation, but manual intervention might be required in some cases.

Time Skew:

• Cause: Significant time differences between the client machine and the Kubernetes API server can lead to certificate validation failures.
• Solution: Ensure that the system clocks of both the client and the Kubernetes nodes are synchronized, ideally using NTP.

[skitt]

@BhavaniYalamanchili could you quantify

found a misbehaviour that multiple secrets on the site 1 are populated continuously.

How many secrets, and what do you mean by “continuously”?

[BhavaniYalamanchili]

@skitt

We can only guess at the frequency:

submariner-k8s-broker-admin-token-5w8fh [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 4h12m
submariner-k8s-broker-admin-dockercfg-jtwpp [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 4h12m
submariner-k8s-broker-admin-token-88nn5 [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 3h55m
submariner-k8s-broker-admin-dockercfg-rxfhk [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 3h55m
submariner-k8s-broker-admin-token-nz9nx [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 3h38m
submariner-k8s-broker-admin-dockercfg-62dm7 [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 3h38m
submariner-k8s-broker-admin-dockercfg-hpt5c [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 3h21m
submariner-k8s-broker-admin-token-2bhth [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 3h21m
submariner-k8s-broker-admin-token-5mlvm [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 3h4m
submariner-k8s-broker-admin-dockercfg-69csz [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 3h4m
submariner-k8s-broker-admin-token-nz7fm [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 167m
submariner-k8s-broker-admin-dockercfg-wmjqq [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 167m
submariner-k8s-broker-admin-token-gz2vr [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 150m
submariner-k8s-broker-admin-dockercfg-dv9xv [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 150m
submariner-k8s-broker-admin-token-vgwpv [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 133m
submariner-k8s-broker-admin-dockercfg-mghj2 [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 133m
submariner-k8s-broker-admin-dockercfg-m4677 [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 116m
submariner-k8s-broker-admin-token-8s5w2 [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 116m
submariner-k8s-broker-admin-token-t8c77 [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 99m
submariner-k8s-broker-admin-dockercfg-45b9z [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 99m
submariner-k8s-broker-admin-token-4p28l [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 82m
submariner-k8s-broker-admin-dockercfg-bz4n8 [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 82m
submariner-k8s-broker-admin-token-2jpbs [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 65m
submariner-k8s-broker-admin-dockercfg-nfgs4 [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 65m
submariner-k8s-broker-admin-token-p8m6l [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 48m
submariner-k8s-broker-admin-dockercfg-9m9rt [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 48m
submariner-k8s-broker-admin-token-k5jzp [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 31m
submariner-k8s-broker-admin-dockercfg-7nk6q [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 31m
submariner-k8s-broker-admin-dockercfg-dpr5j [kubernetes.io/dockercfg](https://kubernetes.io/dockercfg) 1 14m
submariner-k8s-broker-admin-token-xhcm2 [kubernetes.io/service-account-token](https://kubernetes.io/service-account-token) 4 14m

Looks like every 17 minutes?
2,977 secrets have been created in the last 17 days.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further
activity occurs. Thank you for your contributions.