cilium version doesn't detect cilium agents in EKS Anywhere

[lizrice]

Using EKS-Anywhere, cilium version doesn't find the running agents:

❯ cilium version 
cilium-cli: v0.15.0 compiled with go1.20.4 on darwin/arm64
cilium image (default): v1.13.4
cilium image (stable): v1.14.4
cilium image (running): unknown. Unable to obtain cilium version, no cilium pods found in namespace "kube-system"

However, cilium status does correctly find the agents:

 ❯ cilium status
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    disabled (using embedded mode)
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium             Desired: 2, Ready: 2/2, Available: 2/2
Deployment             cilium-operator    Desired: 2, Ready: 2/2, Available: 2/2
Containers:            cilium             Running: 2
                       cilium-operator    Running: 2
Cluster Pods:          15/15 managed by Cilium
Helm chart version:    
Image versions         cilium             public.ecr.aws/isovalent/cilium:v1.11.15-eksa.1: 2
                       cilium-operator    public.ecr.aws/isovalent/operator-generic:v1.11.15-eksa.1: 2

And the cilium agent pods are running in the kube-system namespace with the k8s-app=cilium label:

 ❯ ks get pods --show-labels
NAME                                 READY   STATUS    RESTARTS   AGE    LABELS
cilium-operator-55c4778776-gqrtn     1/1     Running   0          130m   io.cilium/app=operator,name=cilium-operator,pod-template-hash=55c4778776
cilium-operator-55c4778776-nx4q2     1/1     Running   0          130m   io.cilium/app=operator,name=cilium-operator,pod-template-hash=55c4778776
cilium-q6sbk                         1/1     Running   0          129m   controller-revision-hash=5bd5bfbcc4,k8s-app=cilium,pod-template-generation=1
cilium-wjmk4                         1/1     Running   0          130m   controller-revision-hash=5bd5bfbcc4,k8s-app=cilium,pod-template-generation=1
<non-cilium pods omitted for brevity>

General Information

• Cilium CLI version (run cilium version) see above
• Orchestration system version in use (e.g. kubectl version, ...)

❯ k version 
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.4", GitCommit:"fa3d7990104d7c1f16943a67f11b154b71f6a132", GitTreeState:"clean", BuildDate:"2023-07-19T12:14:48Z", GoVersion:"go1.20.6", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1-eks-61789d8", GitCommit:"61789d8300fe03083861ba7fe9fd070cd06839d1", GitTreeState:"archive", BuildDate:"2023-04-14T13:14:41Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/arm64"}

• Platform / infrastructure information (e.g. AWS / Azure / GCP, image / kernel versions)
  EKS-Anywhere running on Docker on Mac

• Link to relevant artifacts (policies, deployments scripts, ...) see reproduction below

• Generate and upload a system zip: cilium sysdump
  cilium-sysdump-20231122-135352.zip

How to reproduce the issue
Following instructions from EKS Anywhere to install on Docker:

CLUSTER_NAME=mgmt
eksctl anywhere generate clusterconfig $CLUSTER_NAME \
   --provider docker > $CLUSTER_NAME.yaml
eksctl anywhere create cluster -f $CLUSTER_NAME.yaml

[Tefnet]

Not sure if it's a EKS specific, but have the same issue on EKS (1.28)

cilium-cli: v0.15.14 compiled with go1.21.4 on darwin/arm64
cilium image (default): v1.14.2
cilium image (stable): v1.14.4
cilium image (running): unknown. Unable to obtain cilium version, no cilium pods found in namespace "kube-system"

kubectl -n kube-system get pods|grep cilium
cilium-dkkvv                                   1/1     Running   0          10h
cilium-kpr7d                                   1/1     Running   0          10h
cilium-operator-f5dcdcc8d-czqcx                1/1     Running   0          10h
cilium-operator-f5dcdcc8d-dv59f                1/1     Running   0          10h
cilium-qmw76                                   1/1     Running   0          10h

[lianghao208]

I checked the code, protentially cause could be the helm.Get() returns error.
https://github.com/cilium/cilium-cli/blob/main/k8s/client.go#L990

Can you turn off the helm mode by setting env CLIModeVariableName=classic to verify it?

[lizrice]

Sorry it took so long to respond on this. I think you are right @lianghao208

❯ CILIUM_CLI_MODE=classic cilium version 
cilium-cli: v0.15.0 compiled with go1.20.4 on darwin/arm64
cilium image (default): v1.13.4
cilium image (stable): v1.14.5
cilium image (running): v1.11.15-eksa.1
❯ CILIUM_CLI_MODE=helm cilium version    
cilium-cli: v0.15.0 compiled with go1.20.4 on darwin/arm64
cilium image (default): v1.13.4
cilium image (stable): v1.14.5
cilium image (running): unknown. Unable to obtain cilium version, no cilium pods found in namespace "kube-system"