Submariner Route Agent fails to populate custom PBR tables created by AWS CNI, causing intermittent cross-cluster connectivity loss

[jakkwis]

What happened:

I am experiencing intermittent, one-way cross-cluster connectivity failures in EKS multi-cluster environment. Connection to specific pods fails (times out), while other pods on the same node remain reachable.

The root cause is a Policy-Based Routing (PBR) integration failure between the Submariner Route Agent and the AWS CNI.

1. The AWS CNI creates dedicated PBR rules and custom routing tables (e.g., table 2) for pods assigned IPs from secondary ENIs (e.g., ens6).
2. The Submariner Route Agent correctly populates the main routing table with the necessary vx-submariner routes for remote clusters.
3. However, the Route Agent fails to detect or populate these custom, CNI-created tables (e.g., table 2).
4. As a result, any egress traffic (including replies to ping or traceroute) from a pod forced to use this custom table is not routed to the vx-submariner tunnel. Instead, it follows the table's default route (the VPC gateway via ens6), causing the packet to be blackholed.

Recreating the failing pod temporarily resolves the issue because the PBR rule is torn down, and the new pod instance (by chance) often uses the main table.

What you expected to happen:

The Submariner Route Agent should detect all active routing tables used by routable pods, including custom PBR tables dynamically created by the AWS CNI.

It should ensure that all necessary vx-submariner routes (for remote cluster/service CIDRs) are replicated to all relevant tables (i.e., table main and any custom tables like table 2) to guarantee consistent cross-cluster egress routing for all pods, regardless of which ENI or routing table they use.

How to reproduce it (as minimally and precisely as possible):

1. Deploy Submariner in an EKS cluster using AWS CNI in its default PBR-enabled mode.
2. Create enough pods on a single node to force the AWS CNI to allocate IPs from a secondary ENI (e.g., ens6).
3. Identify a pod that has been assigned an IP on this secondary ENI.
4. Confirm this by logging into the node and finding a specific PBR rule for the pod's IP (e.g., ip rule show | grep <pod_ip>).
5. Verify that the custom table (e.g., ip route show table 2) is missing the vx-submariner routes, while ip route show table main contains them.
6. Attempt to ping or traceroute this specific pod from a remote (Submariner-connected) cluster.
7. Observe the traceroute failing (timing out) after reaching the destination cluster's gateway.

Anything else we need to know?:

Environment:

• Subctl version: 0.20
• Cloud provider or hardware configuration: AWS EKS 1.32

[dfarrell07]

@aswinsuryan @yboaron Any thoughts here?

[yboaron]

Hi @jakkwis , two relevant points for your case :

1. Pod CIDR discovery
   If the IP addresses assigned to Pods via the secondary ENIs come from a different network range than the default cluster CIDR, Submariner's currently detect and publish to other clusters the default cluster CIDR , in this case the remote clusters will not know the route to reach Pods located on those specific secondary ENI subnets, leading to black-holed traffic.

2. Routing Table Interaction

As you noted, If the EKS CNI connects secondary ENIs to a routing table other than the default (main) current submariner-routeagent code will not function correctly. Submariner currently only adds rules describing how to reach remote cluster Gateway to main routing table.

Maybe you can try validating the second point (routing table interaction) and quickly determine a path forward,
by manually adding necessary inter-cluster routing rules (you can duplicate routes from main table) to the non-main table (in addition to the main table rules) and check if inter-cluster communication is restored.