Bug Report: AWS Security Group Leakage when manage-backend-security-group-rules is false

[sss-ng]

Bug Description
The AWS Load Balancer Controller incorrectly triggers ec2:AuthorizeSecurityGroupIngress on Node/Pod security groups even when the Ingress annotation alb.ingress.kubernetes.io/manage-backend-security-group-rules is set to "false".

I believe this occurs because the TargetGroupBinding (TGB) reconciler unconditionally invokes the NetworkingManager's reconciliation logic, regardless of whether a networking specification is present in the TGB spec. When the annotation is set to "false", the generated TGB has a nil Spec.Networking, but the controller proceeds to track it in its internal security group reconciliation lifecycle. In environments with strict IAM Permissions Boundaries (denying ec2:AuthorizeSecurityGroupIngress), this leads to recurring 403 UnauthorizedOperation errors and prevents successful reconciliation.

Steps to Reproduce

1. Deploy an Ingress with the following annotations:

       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: ip
       alb.ingress.kubernetes.io/security-groups: sg-xxxxxx # Custom SG
       alb.ingress.kubernetes.io/manage-backend-security-group-rules: "false"

2. Ensure the controller's IAM role has a Permissions Boundary that denies ec2:AuthorizeSecurityGroupIngress (or simply lack the permission).
3. Observe the controller logs.

• Manifests applied while reproducing the issue:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: repro-ingress
     annotations:
       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: ip
       alb.ingress.kubernetes.io/security-groups: sg-xxxxxx
       alb.ingress.kubernetes.io/manage-backend-security-group-rules: "false"
   spec:
     ingressClassName: alb
     rules:
     - http:
         paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: repro-svc
               port:
                 number: 80

• Controller logs/error messages while reproducing the issue:

{"level": "error",
"ts": "2026-03-27T14:56:07Z",
"msg": "Requesting network requeue due to error from ReconcileForPodEndpoints",
"tgb": {"name":"k8s-REDACTED", "namespace": "REDACTED"},
"error": "operation error EC2: AuthorizeSecurityGroupIngress, https response error StatusCode: 403, RequestID: ..."

Expected Behavior
When manage-backend-security-group-rules is set to "false", the controller should skip all security group mutations for the associated TargetGroupBinding resources. Specifically, it should not call into the NetworkingManager for resources that have opted out.

Actual Behavior
The controller attempts to reconcile security group rules for the TGB despite the opt-out.

• The bug causes reconciliation failures and 403 errors in logs, though target group registration might still succeed if permissions for that are present.
• This happens always when the annotation is set to "false" but the global flag --enable-backend-security-group is true (default).

Regression
Was the functionality working correctly in a previous version ? [No / Unknown]
Verified in v2.17.1.

Current Workarounds
Setting the global flag --enable-backend-security-group=false resolves the issue by disabling backend SG management cluster-wide. However, this is not a viable solution if other Ingresses in the cluster do require managed security groups.

Environment

• AWS Load Balancer controller version: v2.17.1
• Kubernetes version: v1.30+
• Using EKS (yes/no), if so version?: Yes, 1.33
• Using Service or Ingress: Both
• AWS region: us-east-1
• How was the aws-load-balancer-controller installed: Helm

Possible Solution (Optional)
Add a nil guard in pkg/targetgroupbinding/resource_manager.go to check if tgb.Spec.Networking is non-nil before invoking the NetworkingManager.

In reconcileWithIPTargetType around line 209:

	if tgb.Spec.Networking != nil {
		if err := m.networkingManager.ReconcileForPodEndpoints(ctx, tgb, endpoints); err != nil {
			// ...
		}
	}

And similarly in reconcileWithInstanceTargetType.

Contribution Intention (Optional)

• Yes, I'm willing to submit a PR to fix this issue

Additional Context
Testing with a reproduction unit test confirmed that ReconcileForPodEndpoints is called even when Spec.Networking is nil. Implementing the guard locally fixed the issue and resolved the unauthorized API calls.

[shraddhabang]

@sss-ng Thank you for reaching out and providing the detailed analysis. I think it makes sense. I will try to reproduce this and will work on fixing this.

[shraddhabang]

I have tried to reproduce this configuration and confirmed that the resulting TargetGroupBinding has spec.networking: null. No AuthorizeSecurityGroupIngress calls were observed for this.
Could you confirm:

1. Are there other TargetGroupBindings in your cluster that have spec.networking populated?
2. Are the 403 errors referencing the same SGs that your opted-out pods are running on?

If your IAM permissions boundary denies ec2:AuthorizeSecurityGroupIngress entirely, then any TGB in the cluster with networking enabled will trigger those errors — not just the ones associated with your specific Ingress. The manage-backend-security-group-rules: "false" annotation only controls whether a given Ingress's TGBs get networking rules; it can't prevent other TGBs from managing rules on shared SGs.

[sss-ng]

I was afraid that I ran into some edge case that wasn't easily replicable.

Just to clarify a few things, we also saw that spec.networking was null, and yet the AWS LBC pod was continually logging the AuthorizeSecurityGroupIngress ... error.

I took a look at cloudtrail and saw that, it was trying to edit a security group attached to our EKS nodes, this I believe is the "backend" security group. However, this SG did not match the SG in our annotation alb.ingress.kubernetes.io/security-groups: sg-xxxxxx.

Indeed, we do have other Ingress objects which we are expecting to have this issue, as they were still "managing" their own SGs/Rules prior to us disabling all SG/Rule management.

However, we are seeing this issue for NEW Ingress objects being deployed that explicitly have manage-backend-security-group-rules: "false". The log explicitly shows that this new ingress is the reason.

It's as if the older Ingress objects are somehow tainting the new objects that I'd expect to not have this issue - since after all, the new ones are configured correctly.

In any case, we are planning to migrate the old ones away from trying to manage their own security groups, as this would likely fix the issue, but we thought we'd report this issue we saw during the transition period.

After some more research, I'm pretty convinced that adding that nil guard to the initial ReconcileForPodEndpoints function call will solve this, but not totally sure. What are your thoughts?

[zac-nixon]

I think the misunderstanding is how the various security groups interact:

Internet
   |
   v
+=======================+
|   Frontend SG (top)   |
+=======================+
          ||
          \/ 
   +-----------------+
   |  Load Balancer  |
   +-----------------+
          ||
          \/
+=======================+
| Backend SG (bottom)   |
+=======================+
          ||
          \/
        [Targets]

(Generated w/ Chat GPT)

The annotation alb.ingress.kubernetes.io/security-groups influences the Frontend SG. The SG that is being influenced, like you said, is the backend sg that controls traffic from your LB to inside the cluster.

The Frontend SG can is usually dedicated to a single load balancer (you can share it -- not recommended). The backend SG is always shared, this is what protects ingress to your worker nodes. So it doesn't make sense to revoke AuthorizeSecurityGroupIngress to the backend SG, if not all of your workloads disable backend sg management.

After some more research, I'm pretty convinced that adding that nil guard to the initial ReconcileForPodEndpoints function call will solve this, but not totally sure. What are your thoughts?

This doesn't work because we need to generate a full picture of the Backend SG rules to ensure we can remove rules safely.

[sss-ng]

Hi @zac-nixon, thanks for the clarification on that. I see the distinction between frontend/backend. However, I still have one thing I'm hung up on. Here's a scenario to illustrate:

Lets say I have a VPC that's allowed to create and manage security groups (young startup!)

We deploy App-A which has the following annotations:

       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: ip

Then, a year later, we get acquired by an enterprise (mega corp!) who says, "we are disabling your ability to manage security groups via IAM." They perform the disabling, but App-A keeps working (for now) because the rules already exist.

Then, we get a new feature to develop App-B. We do that with the following annotations to comply with the new rules:

       alb.ingress.kubernetes.io/scheme: internet-facing
       alb.ingress.kubernetes.io/target-type: ip
       alb.ingress.kubernetes.io/security-groups: sg-xxxxxx # Custom SG provided by Mega Corp
       alb.ingress.kubernetes.io/manage-backend-security-group-rules: "false"

Now, we expected this to work, and the traffic does actually flow. However, what we saw in the logs is a constant stream of errors specifically for App-B:

{"level": "error",
"ts": "2026-03-27T14:56:07Z",
"msg": "Requesting network requeue due to error from ReconcileForPodEndpoints",
"tgb": {"name":"k8s-AppB", "namespace": "AppB"},
"error": "operation error EC2: AuthorizeSecurityGroupIngress, https response error StatusCode: 403..."

This is the core of my confusion. Since App-B has manage-backend-security-group-rules: "false", why is the controller attempting a write operation (AuthorizeSecurityGroupIngress) for this specific TGB?

If a specific Ingress has explicitly opted out of management, shouldn't the controller treat its associated TGB as a "no-op" for security group mutations?

[zac-nixon]

The backend SG is shared by all TGBs. On each reconcile run, the code is generating what it believes the state of the ingress rules should be, based off what is put into each TGB networking block.

Ref: https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/pkg/networking/networking_manager.go#L229-L241

So, just because App-B has no rules, the code is still attempting to correct the state of the backend SG because it is missing rules for another TGB.