Stateful packet filtering in nftables drops non-Tailscale traffic (1.66)

[aenron]

What is the issue?

I have an OpenWRT router（x86） with Tailscale installed, configured as an exit node and subnet. After updating to version 1.66, the devices under the router cannot ping public IP addresses such as 1.1.1.1, but can still ping Tailscale's private IP address. Additionally, the router itself can ping public IP addresses, but only the devices under the router cannot ping them.

When I run a "tailscale netcheck" command, it showed that UDP, IPv6, DERP were all false.

However, after rolling back to version 1.64.0, everything returned to normal.

Steps to reproduce

No response

Are there any recent changes that introduced the issue?

No response

OS

Other

OS version

openwrt 23.05.02

Tailscale version

1.66.0

Other software

No response

Bug report

No response

[icio]

Hi @aenron, can you please share some more details about the network flow that's being disrupted here? Are these LAN devices unable to connect to the public internet through the router, e.g. a PC on 192.168.1.7 connecting to the router via eth0 cannot reach 1.1.1.1? Do these LAN devices have Tailscale installed, or is Tailscale only installed on the router?

1.66 included a change related to security-bulletin https://tailscale.com/security-bulletins#ts-2024-005 which might be affecting your setup:

Site-to-site networking now also requires --stateful-filtering=false in addition to --snat-subnet-routes=false on new subnet routers. Existing subnet routers with --snat-subnet-routes=false will default to --stateful-filtering=false.
— https://tailscale.com/changelog#2024-05-08-client

I wonder whether your router is running with --snat-subnet-routes=true and what happens if you set --stateful-filtering=false?

If you can also grab a tailscale bugreport bug ID that will help us look at your setup.

Thanks

Update: we're investigating whether this is a bug - the above should help mitigate it in the mean time.

[knyar]

Thank you for the report! This is a regression introduced in 1.66 as part of stateful packet filtering aimed to migitate the TS-2024-005 issue.

I think this only impacts packet-forwarding nodes (exit nodes, subnet routers) running on Linux with nftables firewall mode.

We expect to build a stable release with a fix shortly. Meanwhile, you can do one of the following:

• Disable stateful packet filtering via tailscale up --stateful-filtering=false
• On systems that support iptables CLI, install it and run tailscaled with TS_DEBUG_FIREWALL_MODE environment variable set to iptables.

[aenron]

Hi @aenron, can you please share some more details about the network flow that's being disrupted here? Are these LAN devices unable to connect to the public internet through the router, e.g. a PC on 192.168.1.7 connecting to the router via eth0 cannot reach 1.1.1.1? Do these LAN devices have Tailscale installed, or is Tailscale only installed on the router?

1.66 included a change related to security-bulletin https://tailscale.com/security-bulletins#ts-2024-005 which might be affecting your setup:

Site-to-site networking now also requires --stateful-filtering=false in addition to --snat-subnet-routes=false on new subnet routers. Existing subnet routers with --snat-subnet-routes=false will default to --stateful-filtering=false.
— https://tailscale.com/changelog#2024-05-08-client

I wonder whether your router is running with --snat-subnet-routes=true and what happens if you set --stateful-filtering=false?

If you can also grab a tailscale bugreport bug ID that will help us look at your setup.

Thanks

Update: we're investigating whether this is a bug - the above should help mitigate it in the mean time.

tailscale bugreport：BUG-eed0c23a1df9f3a1671a9ca2be015fdce42e5cd9096d380fe4927b7e3c187c9d-20240509142755Z-69de24189f9b173b
my startup is: tailscale up --advertise-routes=192.168.2.0/24 --accept-routes --advertise-exit-node --accept-dns=false, and the rest of the parameters are kept as default. When I use tailscale up --stateful-filtering=false, the problem is solved.It is nftables used on Linux, and I hope this information helps resolve the issue.

[plEase939]

tailscale up --stateful-filtering=false Resolved my issue for now. Got panic got multiple alerts of services being down and rushed here. All exit nodes are showing on devices.

[awly]

We released 1.66.1 with the fix: https://tailscale.com/changelog#2024-05-09

[shladek]

Can confirm the fix is working on my simple nftables router that has tailscale installed.

[keslerm]

This is still an issue in 1.66.1 it would seem, our nodes that were created and running 1.66.1 still need --stateful-filtering=false to restore connectivity in docker containers to anything outside of tailscale.

[awly]

@keslerm that is a separate issue, see #12070