Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cilium health check fails towards control plane node #15329

Closed
maximumG opened this issue Apr 17, 2023 · 7 comments · May be fixed by #15339
Closed

Cilium health check fails towards control plane node #15329

maximumG opened this issue Apr 17, 2023 · 7 comments · May be fixed by #15339
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@maximumG
Copy link

maximumG commented Apr 17, 2023 •
edited

/kind bug

1. What kops version are you running?

Client version: 1.26.2 (git-v1.26.2)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

v1.25.9

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

Trying to follow kOps Getting Started with cilium enabled

kops create cluster --name=${NAME} --cloud=aws --zones=us-west-1a --discovery-store=s3://xxx-xxx-xxx/${NAME}/discovery

5. What happened after the commands executed?

cilium-agent running on worker nodes reports that the health check towards control-plane nodes fails.

$ cilium status --verbose
  Name                              IP                Node          Endpoints
  i-018a7b5c13a2332ae               10.4.4.191        unreachable   reachable
  i-08674cf4137ca05a5               10.4.5.111        unreachable   reachable
  i-0a2ccc6a8bc4dc881               10.4.6.163        unreachable   reachable
  
  
$ cilium-health status
  [output omitted]
  i-08674cf4137ca05a5:
    Host connectivity to 10.4.5.111:
      ICMP to stack:   Connection timed out
      HTTP to agent:   OK, RTT=298.557us
    Endpoint connectivity to 100.96.47.26:
      ICMP to stack:   OK, RTT=1.46466ms
      HTTP to agent:   OK, RTT=344.044us

However on cilium-agent running on control plane nodes, cilium health checks are working not matter which nodes the target is. Basically, the AWS security group on control plane nodes does not allow inbound ICMP request from worker nodes.

image

6. What did you expect to happen?

Kops should have configured the control plane Security group to allow inbound ICMP request from worker nodes.

**7. Please provide your cluster manifest. Execute

N/A

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

N/A

9. Anything else do we need to know?

N/A
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Apr 17, 2023
@olemarkus
Copy link
Member

Do you actually need these health checks to pass? Doesn't the HTTP checks supersede the ICMP checks?

@maximumG
Copy link
Author

maximumG commented Apr 18, 2023 •
edited

Doesn't the HTTP checks supersede the ICMP checks?

Based on our testing, cilium reports the node as unreachable if one of the 2 health check fails. In our case, because ICMP health check fails the node is considered as unreachable no matter if the HTTP check succeeds.

The code which test node connectivity can be found here (cilium v1.12.9). Based on this code, its clear that both ICMP and HTTP health check must pass for a node to be considered as reachable.

As we are building monitoring and included the cilium metric unreachable_nodes (:link: here), we always have some false positive unreachable nodes in our graph when dealing with kops cluster.

Basically our baseline for a healthy clusters means that every worker node has 3 unreachable nodes.

image

Would it be acceptable to open ICMP traffic from worker SG to control-plane SG ? I can handle a PR to open ICMP flow from node to master only if Cilium is enabled if you'd like.

@maximumG maximumG mentioned this issue Apr 20, 2023
Open
@olemarkus
Copy link
Member

As mentioned in the PR, we'll open ICMP to allow these checks to pass.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 19, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle rotten
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 18, 2024
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

@k8s-ci-robot k8s-ci-robot closed this as not planned Won't fix, can't repro, duplicate, stale Mar 19, 2024
@k8s-ci-robot
Copy link
Contributor

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue with /reopen
  • Mark this issue as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(firewall): open node to master ICMP security group maximumG/kops
4 participants
@olemarkus @maximumG @k8s-ci-robot @k8s-triage-robot