HyperPod: cannot resolve pod ENI when VPC CNI prefix delegation is enabled

[guessi]

Bug Description

AWS Load Balancer Controller fails to resolve pod ENI for SageMaker HyperPod pods when VPC CNI prefix delegation is enabled. The controller continuously logs cannot resolve pod ENI for pods and requeues the TargetGroupBinding reconciliation every 15 seconds.

The root cause is that resolveViaVPCENIs queries DescribeNetworkInterfaces using the addresses.private-ip-address filter with the pod IP. When prefix delegation is enabled, pod IPs are allocated from /28 prefixes (Ipv4Prefixes) on the ENI rather than appearing as secondary private IPs (PrivateIpAddresses). The EC2 filter does not match IPs within prefixes, so the query returns no results and the pod remains unresolved.

This only affects SageMaker HyperPod. EC2 pods use resolveViaNodeENIs which already handles prefix delegation via isPodSupportedByNodeENI. Fargate pods use resolveViaVPCENIs but Fargate does not use prefix delegation. Hybrid Nodes bypass ENI resolution entirely.

Steps to Reproduce

1. Create an EKS cluster with a SageMaker HyperPod node group where VPC CNI has prefix delegation enabled (or where HyperPod enables it by default)
2. Deploy a workload (e.g., nginx deployment) scheduled on a HyperPod node
3. Create a TargetGroupBinding with targetType: ip pointing to the service
4. Observe controller logs:

{"level":"error","ts":"...","msg":"Requesting network requeue due to error from ReconcileForPodEndpoints","tgb":{"name":"my-tgb","namespace":"default"},"error":"cannot resolve pod ENI for pods: [default/nginx-deployment-xxxxxxxxxx-xxxxx]"}

5. Verify the pod IP is from a prefix range, not a secondary private IP:

# Pod IP is within a /28 prefix (e.g., 10.x.x.x)
kubectl get pod <pod-name> -o wide

# ENI has the IP in Ipv4Prefixes, not in PrivateIpAddresses
aws ec2 describe-network-interfaces --filters Name=addresses.private-ip-address,Values=<node-ip>

Expected Behavior

The controller should resolve the pod ENI by matching the pod IP against the ENI's Ipv4Prefixes/Ipv6Prefixes when the direct addresses.private-ip-address lookup fails.

Actual Behavior

The controller fails to resolve the pod ENI and continuously requeues the TargetGroupBinding reconciliation with FailedNetworkReconcile warning events. This occurs on every reconciliation cycle (every 15 seconds). The TargetGroupBinding never reaches a healthy state.

• Impact: Pod endpoints on HyperPod nodes with prefix delegation cannot be registered as targets
• Frequency: Always, when prefix delegation is enabled on HyperPod nodes

Regression

No — SageMaker HyperPod support was added without prefix delegation handling in resolveViaVPCENIs. The EC2 path (resolveViaNodeENIs) has always supported prefix delegation.

Current Workarounds

Disable prefix delegation on the VPC CNI (ENABLE_PREFIX_DELEGATION=false) if the HyperPod configuration allows it. This forces pod IPs to be allocated as secondary private IPs, which resolveViaVPCENIs can resolve. If HyperPod enables prefix delegation by default and it cannot be disabled, no workaround is available.

Environment

• AWS Load Balancer controller version: v3.1.0 (Any)
• Kubernetes version: Any
• Using EKS (yes/no), if so version?: Yes, with SageMaker HyperPod
• Using Service or Ingress: TargetGroupBinding
• Instance type: Any
• Node label: sagemaker.amazonaws.com/compute-type=hyperpod

Possible Solution (Optional)

Add a prefix-based fallback resolution path for HyperPod pods. When resolveViaVPCENIs fails to resolve a pod via direct IP lookup, look up the ENI using the node's primary IP (available from node.Status.Addresses), then check if the pod IP falls within any of the ENI's Ipv4Prefixes or Ipv6Prefixes.

The fix involves:

• classifyPodsByComputeType: collect node InternalIPs into PodsByComputeType.nodeIPByNodeName (reuses existing node fetch)
• resolvePodsViaCascadedLookup: for unresolved HyperPod pods, call new resolveViaVPCENIsByNodeIPAndPrefix
• resolveViaVPCENIsByNodeIPAndPrefix: queries ENIs by node IP, matches pod IPs against prefixes

No changes to resolveViaVPCENIs, resolveViaCascadedLookup, or any other existing function signatures. No impact on EC2, Fargate, or Hybrid Node paths.

Contribution Intention (Optional)

• Yes, I'm willing to submit a PR to fix this issue
• No, I cannot work on a PR at this time

Additional Context

The ENI for the HyperPod node shows prefix delegation is active:

{
    "NetworkInterfaceId": "eni-xxxxxxxxxxxxxxxxx",
    "PrivateIpAddresses": [{"PrivateIpAddress": "10.x.x.x"}],
    "Ipv4Prefixes": [
        {"Ipv4Prefix": "10.x.x.x/28"},
        {"Ipv4Prefix": "10.x.x.x/28"}
    ],
    "VpcId": "vpc-xxxxxxxxxxxxxxxxx"
}

The pod IP falls within one of the /28 prefixes but is not in PrivateIpAddresses, causing the addresses.private-ip-address EC2 filter to return no results.

Note: AWS documentation for HyperPod (Getting started with Amazon EKS support in SageMaker HyperPod) does not explicitly document whether prefix delegation is supported or unsupported. The max pods table lists values consistent with secondary-IP mode, but the actual HyperPod ENI configuration uses prefix delegation.