Cilium 1.16 BGP LBs timing out

[JJGadgets]

• clearing all Cilium state from node (including making sure Cilium agent doesn't start during this), then draining (and manually deleting all podCIDR pods) and rebooting nodes: doesn't fix.

• During a while true; do curl https://ingress-nginx; sleep 1; done, Hubble shows that during the TCP ACK,PSH timeout period (PCAP from FortiGate (main router & Cilium BGP peer) shows TCP Previous Segment Not Captured), TCP traffic from outside cluster destined for ingress-nginx (LoadBalancer service) ends up as ICMP traffic for VMAgent (headless ClusterIP service) during the blip, then a new TCP session is started.

• if VMAgent pod is stopped, pre-xlate-rev (TRACED) TCP shows up where the ICMP would, presumably because there is no backend "VMAgent" pod left to route to.

• when on Cilium 1.16, even with existing 1.16 state, downgrading to 1.15.7-1.15.11 immediately resolves all timeouts with no drain/reboot or state clearing needed.

• Cilium preflight netpols validator and apiserver server apply validation status all show as Pass for all netpols.

• Talos 1.7.6, Kubernetes 1.30.1, 3x M720q e1000e CPs + 1 R730xd BCM5719 Proxmox worker VM, Cilium 1.16.4 and 1.16.2 tested and both have said issues.

[JJGadgets]

Enabling bpf.masquerade and bpf.hostLegacyRouting fixes it entirely. Not sure why that's the fix, but it works now lol.