Cannot find path 'C:\hpc\var\lib\kube-proxy\kubeconfig-win.conf' because it does not exist

[pbdiazam]

Describe the bug
When I deploy kube-proxy which matches my k8s version, it enters into CrashLoopBackoff:

I can see kube-proxy-windows starts running and after a little time it fails and restarts because it cannot find kubeconfig file path, and these are the logs I see:

kubectl logs -f kube-proxy-windows-6l7s4 -n kube-system

WARNING: The names of some imported commands from the module 'hns' include unapproved verbs that might make them less
discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose
parameter. For a list of approved verbs, type Get-Verb.
Running kub-proxy service.
Waiting for HNS network Calico to be created...
HNS network Calico found.
kubeproxy version Kubernetes v1.30.4
Write files so the kubeconfig points to correct locations


    Directory: C:\var\lib


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----         9/23/2024   7:53 AM                kube-proxy
Get-Content : Cannot find path 'C:\hpc\var\lib\kube-proxy\kubeconfig.conf' because it does not exist.
At C:\hpc\kube-proxy\start.ps1:56 char:3
+ ((Get-Content -path $env:CONTAINER_SANDBOX_MOUNT_POINT/var/lib/kube-p ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\hpc\var\lib\...kubeconfig.conf:String) [Get-Content], ItemNotFoundEx
   ception
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetContentCommand

cp : Cannot find path 'C:\hpc\var\lib\kube-proxy\kubeconfig-win.conf' because it does not exist.
At C:\hpc\kube-proxy\start.ps1:57 char:1
+ cp $env:CONTAINER_SANDBOX_MOUNT_POINT/var/lib/kube-proxy/kubeconfig-w ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\hpc\var\lib\...config-win.conf:String) [Copy-Item], ItemNotFoundExce
   ption
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.CopyItemCommand

Requires 2019 with KB4580390 (Oct 2020)
Detected VXLAN network, waiting for Calico host endpoint to be created...
Host endpoint found.
Enabling feature gates: WinDSR=true WinOverlay=true.
Start to run C:\hpc\/kube-proxy/kube-proxy.exe --hostname-override=ip-100-90-2-213.ec2.internal --v=4 --proxy-mode=kernelspace --kubeconfig=C:\hpc\/var/lib/kube-proxy/kubeconfig-win.conf --enable-dsr=true --source-vip=172.16.181.2 --feature-gates=WinDSR=true,WinOverlay=true
I0923 07:54:51.665208    2664 flags.go:64] FLAG: --bind-address="0.0.0.0"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --bind-address-hard-fail="false"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --cleanup="false"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --cluster-cidr=""
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --config=""
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --config-sync-period="15m0s"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-max-per-core="32768"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-min="131072"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-tcp-be-liberal="false"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-tcp-timeout-close-wait="1h0m0s"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-tcp-timeout-established="24h0m0s"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-udp-timeout="0s"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --conntrack-udp-timeout-stream="0s"
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --detect-local-mode=""
I0923 07:54:51.733435    2664 flags.go:64] FLAG: --enable-dsr="true"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --feature-gates="WinDSR=true,WinOverlay=true"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --forward-healthcheck-vip="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --healthz-bind-address="0.0.0.0:10256"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --healthz-port="10256"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --help="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --hostname-override="ip-100-90-2-213.ec2.internal"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --init-only="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --iptables-localhost-nodeports="true"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --iptables-masquerade-bit="14"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --iptables-min-sync-period="1s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --iptables-sync-period="30s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-exclude-cidrs="[]"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-min-sync-period="0s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-scheduler=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-strict-arp="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-sync-period="30s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-tcp-timeout="0s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-tcpfin-timeout="0s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --ipvs-udp-timeout="0s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --kube-api-burst="10"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --kube-api-qps="5"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --kubeconfig="C:\\hpc\\/var/lib/kube-proxy/kubeconfig-win.conf"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --log-flush-frequency="5s"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --log-json-info-buffer-size="0"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --log-json-split-stream="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --log-text-info-buffer-size="0"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --log-text-split-stream="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --logging-format="text"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --masquerade-all="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --master=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --metrics-bind-address="127.0.0.1:10249"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --metrics-port="10249"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --network-name=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --nodeport-addresses="[]"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --oom-score-adj="-999"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --pod-bridge-interface=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --pod-interface-name-prefix=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --profiling="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --proxy-mode="kernelspace"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --proxy-port-range=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --root-hnsendpoint-name="cbr0"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --show-hidden-metrics-for-version=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --source-vip="172.16.181.2"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --v="4"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --version="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --vmodule=""
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --windows-service="false"
I0923 07:54:51.737319    2664 flags.go:64] FLAG: --write-config-to=""
I0923 07:54:51.737319    2664 feature_gate.go:254] feature gates: {map[WinDSR:true WinOverlay:true]}
E0923 07:54:51.738038    2664 server.go:558] "Error running ProxyServer" err="CreateFile C:\\hpc\\/var/lib/kube-proxy/kubeconfig-win.conf: The system cannot find the file specified."
E0923 07:54:51.738038    2664 run.go:74] "command failed" err="CreateFile C:\\hpc\\/var/lib/kube-proxy/kubeconfig-win.conf: The system cannot find the file specified."

To Reproduce

EKS Setup with Linux and Calico Networking
eksctl create cluster --name cluster --region us-east-1 --vpc-private-subnets=subnet-1,subnet-2 --node-private-networking --without-nodegroup

kubectl delete daemonset -n kube-system aws-node

kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.28.1/manifests/tigera-operator.yaml

kubectl create -f - <<EOF
kind: Installation
apiVersion: operator.tigera.io/v1
metadata:
  name: default
spec:
  kubernetesProvider: EKS
  cni:
    type: Calico
  calicoNetwork:
    bgp: Disabled
EOF

cat <<EOF  | kubectl apply -f -
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
  name: default
spec: {}
EOF

eksctl create nodegroup   --cluster cluster  --name linux-nodegroup   --subnet-ids subnet-3,subnet-4 --node-type t3.medium   --nodes 2   --nodes-min 0   --nodes-max 4   --managed=false   --region us-east-1 --node-private-networking

WINDOWS
aws iam list-attached-role-policies --role-name eksClusterRole
{
    "AttachedPolicies": [
        {
            "PolicyName": "AmazonEKSClusterPolicy",
            "PolicyArn": "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
        },
        {
            "PolicyName": "AmazonEKSVPCResourceController",
            "PolicyArn": "arn:aws:iam::aws:policy/AmazonEKSVPCResourceController"
        }
    ]
}

aws iam attach-role-policy \
  --role-name eksClusterRole \
  --policy-arn arn:aws:iam::aws:policy/AmazonEKSVPCResourceController

vi vpc-resource-controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: amazon-vpc-cni
  namespace: kube-system
data:
  enable-windows-ipam: "true"

kubectl apply -f vpc-resource-controller-configmap.yaml

eksctl create nodegroup --cluster=cluster --name windows-nodegroup --node-ami-family=WindowsServer2022FullContainer --subnet-ids subnet-3,subnet-4 --node-type m5.xlarge --nodes 1   --nodes-min 1   --nodes-max 4   --managed=false   --region us-east-1 --node-private-networking

curl -O https://s3.us-west-2.amazonaws.com/amazon-eks/cloudformation/2020-10-29/aws-auth-cm.yaml

sed -i.bak -e 's|<ARN of instance role (not instance profile)>|<windows_nodegrouo_instance_role_arn>|' aws-auth-cm.yaml

kubectl apply -f aws-auth-cm.yaml

kubectl edit configmap aws-auth -n kube-system
Add the following to the config map:
- eks:kube-proxy-windows

kubectl patch ipamconfigurations default --type merge --patch='{"spec": {"strictAffinity": true}}'

kubectl patch installation default --type=merge -p '{"spec": {"calicoNetwork": {"bgp": "Disabled"}}}'

APISERVER_ADDR=<eks_server_endpoint>
APISERVER_PORT=443

kubectl apply -f - << EOF
kind: ConfigMap
apiVersion: v1
metadata:
  name: kubernetes-services-endpoint
  namespace: tigera-operator
data:
  KUBERNETES_SERVICE_HOST: "${APISERVER_ADDR}"
  KUBERNETES_SERVICE_PORT: "${APISERVER_PORT}"
EOF

kubectl patch installation default --type merge --patch='{"spec": {"serviceCIDRs": ["172.20.0.0/16"], "calicoNetwork": {"windowsDataplane": "HNS"}}}'

curl -L  https://raw.githubusercontent.com/kubernetes-sigs/sig-windows-tools/master/hostprocess/calico/kube-proxy/kube-proxy.yml | sed "s/KUBE_PROXY_VERSION/v1.30.4/g" | kubectl apply -f -

Expected behavior
I would expect kube-proxy to work without any additional configuration and to be able to find the kubeconfig.

Kubernetes (please complete the following information):

• Windows Server version: Windows Server 2022 Datacenter (AMI:WindowsServer2022FullContainer) - 10.0.20348.2700
• Kubernetes Version: v1.30.4-eks-a737599
• CNI: Calico v3.28.1

Additional context
These are the nodes I currently have in my cluster with the containerd version:

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kaspertoq]

We experienced the same issue.
The script that the kube-proxy-windows runs is looking for a kubeconfig.conf file. That file is supposed to be mounted from the kube-proxy ConfigMap. In our case that ConfigMap did only contain a kubeconfig file but no kubeconfig.conf. After modifying the ConfigMap the kube-proxy-windows did start up.
We did not dig deeper into the root cause of the file name mismatch yet.