helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution) #11095

andrewfung729 · 2024-04-18T01:29:07Z

What happened?

Hi Kubespray Team. I am new to k8s. Glad to use kubespray!

I found that when installing kubelet-csr-approver, due to the lack of 'no_proxy' env, helm uses the http proxy to access kube-apiserver. Our corporate proxy logged the requests like CONNECT lb-apiserver.kubernetes.local:6443.

For the security and privacy, I have hidden the username, IPs, proxy URL.

Source shortcut: https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/roles/kubernetes-apps/kubelet-csr-approver/meta/main.yml

playbook log:

TASK [helm-apps : Update Helm repositories] ************************************************************************************************************************
task path: /home/user/kubespray/roles/helm-apps/tasks/main.yml:30
fatal: [cp-1]: FAILED! => {
    "changed": false,
    "command": "/usr/local/bin/helm list --output=yaml --filter dummy",
    "invocation": {
        "module_args": {
            "api_key": null,
            "atomic": false,
            "binary_path": "/usr/local/bin/helm",
            "ca_cert": null,
            "chart_ref": null,
            "chart_repo_url": null,
            "chart_version": null,
            "context": null,
            "create_namespace": false,
            "dependency_update": false,
            "disable_hook": false,
            "force": false,
            "history_max": null,
            "host": null,
            "kubeconfig": null,
            "post_renderer": null,
            "purge": true,
            "release_name": "dummy",
            "release_namespace": "kube-system",
            "release_state": "absent",
            "release_values": {},
            "replace": false,
            "set_values": null,
            "skip_crds": false,
            "state": "absent",
            "timeout": null,
            "update_repo_cache": true,
            "validate_certs": true,
            "values_files": [],
            "wait": false,
            "wait_timeout": null
        }
    },
    "msg": "Failure when executing Helm command. Exited 1.\nstdout: \nstderr: Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr": "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr_lines": [
        "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
    ],
    "stdout": "",
    "stdout_lines": []
}

What did you expect to happen?

It should be passed normally

TASK [helm-apps : Update Helm repositories] ************************************************************************************************************************
task path: /home/user/test/kubespray/roles/helm-apps/tasks/main.yml:6
ok: [cp-1] => {
    "changed": false,
    "command": "/usr/local/bin/helm",
    "invocation": {
        "module_args": {
            "api_key": null,
            "atomic": false,
            "binary_path": "/usr/local/bin/helm",
            "ca_cert": null,
            "chart_ref": null,
            "chart_repo_url": null,
            "chart_version": null,
            "context": null,
            "create_namespace": false,
            "dependency_update": false,
            "disable_hook": false,
            "force": false,
            "history_max": null,
            "host": null,
            "kubeconfig": null,
            "post_renderer": null,
            "purge": true,
            "release_name": "dummy",
            "release_namespace": "kube-system",
            "release_state": "absent",
            "release_values": {},
            "replace": false,
            "set_values": null,
            "skip_crds": false,
            "state": "absent",
            "timeout": null,
            "update_repo_cache": true,
            "validate_certs": true,
            "values_files": [],
            "wait": false,
            "wait_timeout": null
        }
    },
    "status": null,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}

How can we reproduce it (as minimally and precisely as possible)?

Using kube-vip for HA (https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/docs/kube-vip.md)
Enable kubelet_csr_approver_enabled (hardening: https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/docs/hardening.md)

kubelet_rotate_server_certificates: true

Enable proxy (using proxy: https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/docs/proxy.md)

http_proxy: "http://foobar"
https_proxy: "http://foobar"
# https_proxy_cert_file: ""
# no_proxy: ""
# download_validate_certs: False
# additional_no_proxy: ""

OS

PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Version of Ansible

ansible [core 2.15.9]
config file = /home/user/test/kubespray/ansible.cfg
configured module search path = ['/home/user/test/kubespray/library']
ansible python module location = /home/user/miniconda3/envs/kubespray/lib/python3.11/site-packages/ansible
ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections
executable location = /home/user/miniconda3/envs/kubespray/bin/ansible
python version = 3.11.8 (main, Feb 26 2024, 21:39:34) [GCC 11.2.0] (/home/user/miniconda3/envs/kubespray/bin/python)
jinja version = 3.1.2
libyaml = True

Version of Python

Python 3.11.8

Version of Kubespray (commit)

2cb8c85 (v2.24.1)

Network plugin used

calico

Full inventory with variables

provided in the reproduce method before

Command used to invoke ansible

ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root --ask-become-pass -vvvvvv cluster.yml

Output of ansible run

playbook output:

TASK [helm-apps : Update Helm repositories] ************************************************************************************************************************
task path: /home/user/kubespray/roles/helm-apps/tasks/main.yml:30
fatal: [cp-1]: FAILED! => {
    "changed": false,
    "command": "/usr/local/bin/helm list --output=yaml --filter dummy",
    "invocation": {
        "module_args": {
            "api_key": null,
            "atomic": false,
            "binary_path": "/usr/local/bin/helm",
            "ca_cert": null,
            "chart_ref": null,
            "chart_repo_url": null,
            "chart_version": null,
            "context": null,
            "create_namespace": false,
            "dependency_update": false,
            "disable_hook": false,
            "force": false,
            "history_max": null,
            "host": null,
            "kubeconfig": null,
            "post_renderer": null,
            "purge": true,
            "release_name": "dummy",
            "release_namespace": "kube-system",
            "release_state": "absent",
            "release_values": {},
            "replace": false,
            "set_values": null,
            "skip_crds": false,
            "state": "absent",
            "timeout": null,
            "update_repo_cache": true,
            "validate_certs": true,
            "values_files": [],
            "wait": false,
            "wait_timeout": null
        }
    },
    "msg": "Failure when executing Helm command. Exited 1.\nstdout: \nstderr: Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr": "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr_lines": [
        "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
    ],
    "stdout": "",
    "stdout_lines": []
}

verbose output show the proxy env

<10.XX.XX.XX> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/user/.ssh/user-ansible"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="user"' -o ConnectTimeout=10 -o 'ControlPath="/home/user/.ansible/cp/2d48a40eb9"'10.XX.XX.XX '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-hrfrnpvqtsxvhxtnqdnfjewoftnqwsdo ; ALL_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' FTP_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' HTTPS_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' HTTP_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' NO_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' all_proxy='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' ftp_proxy='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' http_proxy=http://foobar https_proxy=http://foobar no_proxy='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' /usr/bin/python3'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<10.XX.XX.XX> (1, b'\n{"stdout": "", "stderr": "Error: Kubernetes cluster unreachable: Get \\"https://lb-apiserver.kubernetes.local:6443/version\\": tls: failed to verify certificate: x509: certificate signed by unknown authority\\n", "command": "/usr/local/bin/helm list --output=yaml --filter kubelet-csr-approver", "failed": true, "msg": "Failure when executing Helm command. Exited 1.\\nstdout: \\nstderr: Error: Kubernetes cluster unreachable: Get \\"https://lb-apiserver.kubernetes.local:6443/version\\": tls: failed to verify certificate: x509: certificate signed by unknown authority\\n", "invocation": {"module_args": {"atomic": true, "binary_path": "/usr/local/bin/helm", "name": "kubelet-csr-approver", "namespace": "kube-system", "chart_ref": "kubelet-csr-approver/kubelet-csr-approver", "chart_version": "0.2.8", "wait": true, "values": {}, "release_name": "kubelet-csr-approver", "release_namespace": "kube-system", "release_values": {}, "validate_certs": true, "dependency_update": false, "release_state": "present", "values_files": [], "update_repo_cache": false, "disable_hook": false, "force": false, "purge": true, "create_namespace": false, "replace": false, "skip_crds": false, "context": null, "kubeconfig": null, "host": null, "ca_cert": null, "api_key": null, "chart_repo_url": null, "wait_timeout": null, "timeout": null, "post_renderer": null, "history_max": null, "set_values": null}}}\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.XX.XX.XX is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 418983\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n")

Anything else we need to know

My solution is adding back no_proxy env (https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/roles/kubernetes-apps/kubelet-csr-approver/meta/main.yml)

---
dependencies:
  - role: helm-apps
    when:
      - inventory_hostname == groups['kube_control_plane'][0]
      - kubelet_csr_approver_enabled
    environment:
      http_proxy: "{{ http_proxy | default('') }}"
      https_proxy: "{{ https_proxy | default('') }}"
      no_proxy: "{{ no_proxy | default('') }}" # <-------  add this
    release_common_opts: {}
    releases:
      - name: kubelet-csr-approver
        namespace: "{{ kubelet_csr_approver_namespace }}"
        chart_ref: "{{ kubelet_csr_approver_chart_ref }}"
        chart_version: "{{ kubelet_csr_approver_chart_version }}"
        wait: true
        values: "{{ kubelet_csr_approver_values }}"
    repositories:
      - name: "{{ kubelet_csr_approver_repository_name }}"
        url: "{{ kubelet_csr_approver_repository_url }}"

I also found a similar structure at custom_cni (https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/roles/network_plugin/custom_cni/meta/main.yml)

The text was updated successfully, but these errors were encountered:

KubeKyrie · 2024-05-08T13:09:23Z

Could u show the kube-vip setting?

andrewfung729 · 2024-05-09T06:02:13Z

Thanks for your reply. For your information, after I added back the no_proxy env on playbook, the issue no longer occurred again.

## `inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml`
kube_proxy_strict_arp: true

## `inventory/mycluster/group_vars/k8s_cluster/addons.yml`
# Kube VIP
kube_vip_enabled: true
kube_vip_arp_enabled: true
kube_vip_controlplane_enabled: true
kube_vip_address: 10.XX.XX.XX
loadbalancer_apiserver:
  address: "{{ kube_vip_address }}"
  port: 6443
kube_vip_interface: enp6s19
kube_vip_lb_enable: true
kube_vip_services_enabled: false
kube_vip_enableServicesElection: true

andrewfung729 added the kind/bug Categorizes issue or PR as related to a bug. label Apr 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution) #11095

helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution) #11095

andrewfung729 commented Apr 18, 2024

KubeKyrie commented May 8, 2024

andrewfung729 commented May 9, 2024 •

edited

helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution) #11095

helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution) #11095

Comments

andrewfung729 commented Apr 18, 2024

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

OS

Version of Ansible

Version of Python

Version of Kubespray (commit)

Network plugin used

Full inventory with variables

Command used to invoke ansible

Output of ansible run

Anything else we need to know

KubeKyrie commented May 8, 2024

andrewfung729 commented May 9, 2024 • edited

andrewfung729 commented May 9, 2024 •

edited