Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ICMP messages are not forwarded back to the backing pod of a workload in ebpf #8854

Closed
ehsan310 opened this issue May 23, 2024 · 0 comments · Fixed by #8858
Closed

ICMP messages are not forwarded back to the backing pod of a workload in ebpf #8854

ehsan310 opened this issue May 23, 2024 · 0 comments · Fixed by #8858
Assignees
@tomastigera
Labels
area/bpf eBPF Dataplane issues impact/high kind/bug likelihood/high

Comments

@ehsan310
Copy link

ehsan310 commented May 23, 2024 •
edited by tomastigera
Loading

Expected Behavior

When traffic from the backing pod it hitting smaller MTU on the way back to the client, ICMP messages signalling that fragmentation is required do not make it back to the backing pod if that pod is on a different node than where the service is being resolved.

Current Behavior

due to bigger mtu packets are dropped and connection is closed.

Possible Solution

Steps to Reproduce (for bugs)

  1. enable ebpf
  2. peer with router
  3. disable node to node mesh
  4. create a service
    5.reach pod via service ip on the host not backing the pod so you get nat'ed
    6.you need to have big payload that pod returns.

Context

having a problem with Service IP and have to reduce the mtu.

Your Environment

  • Calico version : 3.27.3
  • Orchestrator version (e.g. kubernetes, mesos, rkt): : Kubernetes via Kubespray
  • Operating System and version: Debian 12
  • Link to your project (optional):
@tomastigera tomastigera self-assigned this May 23, 2024
tomastigera added a commit to tomastigera/project-calico-calico that referenced this issue May 25, 2024
@tomastigera
[BPF] fix forwarding and NATing of ICMP related messages
31f6338 
When a service (nodeport) is accessed from outside the cluster and
traffic from backend generates ICMP error outside the cluster, we need
to deliver the ICMP error (e.g. MTU too big) back to the backend
workload so that it can adjust. These packets were not sent back into
the vxlan tunnel neither we did DNAT on them if they were.

fixes projectcalico#8854
@tomastigera tomastigera mentioned this issue May 25, 2024
Merged
3 tasks
tomastigera added a commit to tomastigera/project-calico-calico that referenced this issue Jun 3, 2024
@tomastigera
[BPF] fix forwarding and NATing of ICMP related messages
4488a49 
When a service (nodeport) is accessed from outside the cluster and
traffic from backend generates ICMP error outside the cluster, we need
to deliver the ICMP error (e.g. MTU too big) back to the backend
workload so that it can adjust. These packets were not sent back into
the vxlan tunnel neither we did DNAT on them if they were.

* add test for small MTU internet link

* check ICMP checksum

* markes packets with resolved ICMP nat so that we do not do it again in
  revers on the next hop.

fixes projectcalico#8854
tomastigera added a commit to tomastigera/project-calico-calico that referenced this issue Jun 4, 2024
@tomastigera
[BPF] fix forwarding and NATing of ICMP related messages
58855b1 
When a service (nodeport) is accessed from outside the cluster and
traffic from backend generates ICMP error outside the cluster, we need
to deliver the ICMP error (e.g. MTU too big) back to the backend
workload so that it can adjust. These packets were not sent back into
the vxlan tunnel neither we did DNAT on them if they were.

* add test for small MTU internet link

* check ICMP checksum

* markes packets with resolved ICMP nat so that we do not do it again in
  revers on the next hop.

fixes projectcalico#8854
tomastigera added a commit to tomastigera/project-calico-calico that referenced this issue Jun 4, 2024
@tomastigera
[BPF] fix forwarding and NATing of ICMP related messages
0aa4fc0 
When a service (nodeport) is accessed from outside the cluster and
traffic from backend generates ICMP error outside the cluster, we need
to deliver the ICMP error (e.g. MTU too big) back to the backend
workload so that it can adjust. These packets were not sent back into
the vxlan tunnel neither we did DNAT on them if they were.

* add test for small MTU internet link

* check ICMP checksum

* markes packets with resolved ICMP nat so that we do not do it again in
  revers on the next hop.

fixes projectcalico#8854
tomastigera added a commit to tomastigera/project-calico-calico that referenced this issue Jun 4, 2024
@tomastigera
[BPF] fix forwarding and NATing of ICMP related messages
d1fc509 
When a service (nodeport) is accessed from outside the cluster and
traffic from backend generates ICMP error outside the cluster, we need
to deliver the ICMP error (e.g. MTU too big) back to the backend
workload so that it can adjust. These packets were not sent back into
the vxlan tunnel neither we did DNAT on them if they were.

* add test for small MTU internet link

* check ICMP checksum

* markes packets with resolved ICMP nat so that we do not do it again in
  revers on the next hop.

fixes projectcalico#8854
@sfudeus sfudeus mentioned this issue Jun 5, 2024
Closed
tomastigera added a commit to tomastigera/project-calico-calico that referenced this issue Jun 12, 2024
@tomastigera
fix forwarding and NATing of ICMP related messages
f9c9ce2 
When a service (nodeport) is accessed from outside the cluster and
traffic from backend generates ICMP error outside the cluster, we need
to deliver the ICMP error (e.g. MTU too big) back to the backend
workload so that it can adjust. These packets were not sent back into
the vxlan tunnel neither we did DNAT on them if they were.

* add test for small MTU internet link

* check ICMP checksum

* markes packets with resolved ICMP nat so that we do not do it again in
  revers on the next hop.

fixes projectcalico#8854
@tomastigera tomastigera mentioned this issue Jun 12, 2024
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomastigera tomastigera

Labels
area/bpf eBPF Dataplane issues impact/high kind/bug likelihood/high
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[BPF] fix forwarding and NATing of ICMP related messages tomastigera/project-calico-calico
3 participants
@caseydavenport @ehsan310 @tomastigera