Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enable-nat46x64-gateway breaks network connectivity over tun interfaces #32457

Open
3 tasks done
farcaller opened this issue May 10, 2024 · 2 comments
Open
3 tasks done

enable-nat46x64-gateway breaks network connectivity over tun interfaces #32457

farcaller opened this issue May 10, 2024 · 2 comments
Labels
feature/lb-only Impacts cilium running in lb-only datapath mode info-completed The GH issue has received a reply from the author kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.

Comments

@farcaller
Copy link
Contributor

farcaller commented May 10, 2024 •
edited

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

When I enable enable-nat46x64-gateway, tcp and icmp traffic over nodes with cilium stops immediately upon the agents applying the new configuration. If it's disabled, the traffic resumes. This was tested with tailscale providing tailscale0 interface by trying to ping the node using the ip address of the tailscale0 interface and trying to access the apiserver that normally listens on that interface. Tailscale ssh works, because it doesn't materialize as physical packets on the interface.

The functionality on physical links (e.g. eth0) is not affected. The functionality of accessing the services from the node itself is not affected (but I suppose that doesn't really go through the same flow anyway).

Cilium Version

1.15.4 9b3f9a8

Kernel Version

6.6.28

Kubernetes Version

v1.29.3+k3s1

Regression

No response

Sysdump

cilium-sysdump-20240510-105545.zip

Relevant log output

No response

Anything else?

No response

Cilium Users Document

  • Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

  • I agree to follow this project's Code of Conduct
@farcaller farcaller added kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. labels May 10, 2024
@farcaller farcaller changed the title enable-nat46x64-gateway breaks network connectivity over tap interfaces enable-nat46x64-gateway breaks network connectivity over tun interfaces May 10, 2024
@squeed
Copy link
Contributor

squeed commented May 14, 2024

@farcaller are you able to capture a pwru of traffic that is being dropped?

@squeed squeed added the need-more-info More information is required to further debug or fix the issue. label May 14, 2024
@farcaller
Copy link
Contributor Author

@squeed:

Is this enough?

$ pwru 'ip dst 100.92.130.52 and tcp port 6443'
2024/05/14 11:26:07 Attaching kprobes (via kprobe)...
1455 / 1455 [-------------------------------------------------------------------------------------------------------------------------------------] 100.00% 256 p/s
2024/05/14 11:26:13 Attached (ignored 108)
2024/05/14 11:26:13 Listening for events..
               SKB    CPU          PROCESS                     FUNC
0xffff888008a94100      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]        netif_receive_skb
0xffff888008a94100      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]      __netif_receive_skb
0xffff888008a94100      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581] __netif_receive_skb_one_core
0xffff888008a94100      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                   tc_run
0xffff888008a94100      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]             tcf_classify
0xffff888008a94100      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]      skb_ensure_writable
0xffff888008a94100      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]     bpf_skb_generic_push
0xffff888008a94100      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                 skb_push
^C

$ pwru 'ip dst 100.92.130.52 and icmp'
2024/05/14 11:27:17 Attaching kprobes (via kprobe)...
1455 / 1455 [-------------------------------------------------------------------------------------------------------------------------------------] 100.00% 264 p/s
2024/05/14 11:27:22 Attached (ignored 108)
2024/05/14 11:27:22 Listening for events..
               SKB    CPU          PROCESS                     FUNC
0xffff88800d38f400      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]        netif_receive_skb
0xffff88800d38f400      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]      __netif_receive_skb
0xffff88800d38f400      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581] __netif_receive_skb_one_core
0xffff88800d38f400      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                   tc_run
0xffff88800d38f400      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]             tcf_classify
0xffff88800d38f400      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]      skb_ensure_writable
0xffff88800d38f400      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]     bpf_skb_generic_push
0xffff88800d38f400      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                 skb_push

^C

Here's how the same ping looks with the option disabled:

$ pwru 'ip dst 100.92.130.52 and icmp'
2024/05/14 11:28:55 Attaching kprobes (via kprobe)...
1455 / 1455 [-------------------------------------------------------------------------------------------------------------------------------------] 100.00% 261 p/s
2024/05/14 11:29:00 Attached (ignored 108)
2024/05/14 11:29:00 Listening for events..
               SKB    CPU          PROCESS                     FUNC
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]        netif_receive_skb
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]      __netif_receive_skb
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581] __netif_receive_skb_one_core
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                   tc_run
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]             tcf_classify
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]      skb_ensure_writable
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                   ip_rcv
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]              ip_rcv_core
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]               sock_wfree
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]             nf_hook_slow
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]           nf_ip_checksum
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]  __skb_checksum_complete
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]     ip_route_input_noref
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]      ip_route_input_slow
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]      fib_validate_source
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]    __fib_validate_source
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]         ip_local_deliver
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]             nf_hook_slow
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]  ip_local_deliver_finish
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]  ip_protocol_deliver_rcu
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]        raw_local_deliver
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                skb_clone
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                  raw_rcv
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                 skb_push
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]     ipv4_pktinfo_prepare
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581] sock_queue_rcv_skb_reason
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]       sk_filter_trim_cap
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581] __cgroup_bpf_run_filter_skb
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]    security_sock_rcv_skb
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581] selinux_socket_sock_rcv_skb
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581] selinux_sock_rcv_skb_compat
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581] selinux_netlbl_sock_rcv_skb
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581] bpf_lsm_socket_sock_rcv_skb
0xffff888023841300      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]     __sock_queue_rcv_skb
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                 icmp_rcv
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]                icmp_echo
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]               icmp_reply
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]        __ip_options_echo
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]     fib_compute_spec_dst
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]              consume_skb
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]   skb_release_head_state
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]         skb_release_data
0xffff888023841000      0 [/nix/store/a52sm7708nvdp8v4ngin9msfgrz7s04p-tailscale-1.64.2/bin/.tailscaled-wrapped:581]             kfree_skbmem
0xffff888023841300      0 [cilium-agent:29469]        skb_free_datagram
0xffff888023841300      0 [cilium-agent:29469]              consume_skb
0xffff888023841300      0 [cilium-agent:29469]   skb_release_head_state
0xffff888023841300      0 [cilium-agent:29469]               sock_rfree
0xffff888023841300      0 [cilium-agent:29469]         skb_release_data
0xffff888023841300      0 [cilium-agent:29469]            skb_free_head
0xffff888023841300      0 [cilium-agent:29469]             kfree_skbmem

@github-actions github-actions bot added info-completed The GH issue has received a reply from the author and removed need-more-info More information is required to further debug or fix the issue. labels May 14, 2024
@squeed squeed added sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. feature/lb-only Impacts cilium running in lb-only datapath mode labels May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature/lb-only Impacts cilium running in lb-only datapath mode info-completed The GH issue has received a reply from the author kind/bug This is a bug in the Cilium logic. kind/community-report This was reported by a user in the Cilium community, eg via Slack. needs/triage This issue requires triaging to establish severity and next steps. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@farcaller @squeed