Added logic to the target_group_synthesizer to ensure the controller can recover when the maximum unique target groups per ALB is reached

[dominikhei]

Issue

Fixes #4848

Description

Previously stale target groups were deleted by PostSynthesize as target groups can't be deleted while listener rules still reference them, so deletion is deferred until after the listener rule synthesizer has already cleaned up its stale rules. However if Synthesize hits the quota error and returns early, PostSynthesize never runs, so the stale TGs never get deleted, so the slot never gets freed, so the next reconcile hits the exact same error again.

This PR introduces logic to fix this by adding an inline delete to Synthesize. When the quota error is hit, instead of returning immediately, one stale TG is deleted within Synthesize to free a slot and retry the create in the same reconcile cycle. Because the inline delete runs before the listener rule synthesizer has cleaned up stale rules in the current cycle, the delete may fail with ResourceInUse on the first attempt. In that case the controller requeues. On the next cycle the listener rules from the previous cycle are already gone, so the stale TG delete succeeds and the create goes through / the current deadlock is resolved. The controller recovers in at most two reconcile cycles instead of being stuck forever.

Compared to the ListenerRuleSynthesizer this one for target groups handles quota logic inside Synthesize on the struct and not via a separate function. Due to this there needs to be the full environment that Synthesize expects
and tests for the new functionality became a bit complex.

Note: The logic for deleting and creating TGs could be made easier to understand and tests could be simplified by rewriting the range loop as an index-based loop with the same three-branch structure as the listener rule synthesizer. Would be interested on some thoughts on that?

Checklist

• Added tests that cover your change (if possible)
• Added/modified documentation as required (such as the README.md, or the docs directory)
• Manually tested
• Made sure the title of the PR is a good description that can go into the release notes

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

• Backfilled missing tests for code in same general area 🎉
• Refactored something and made the world a better place 🌟

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dominikhei
Once this PR has been reviewed and has the lgtm label, please assign shraddhabang for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubernetes-prow]

Hi @dominikhei. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 78.94737% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.57%. Comparing base (2fcd20d) to head (6faf733).
⚠️ Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/deploy/elbv2/target_group_synthesizer.go	78.94%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4882      +/-   ##
==========================================
+ Coverage   57.44%   57.57%   +0.12%     
==========================================
  Files         396      396              
  Lines       31619    31695      +76     
==========================================
+ Hits        18164    18248      +84     
+ Misses      12398    12382      -16     
- Partials     1057     1065       +8     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[zac-nixon]

P1 — The PR handles the error at the wrong AWS operation. The new code catches TooManyUniqueTargetGroupsPerLoadBalancer from tgManager.Create(), which calls CreateTargetGroup. But AWS does not return that error from CreateTargetGroup; its documented quota error is TooManyTargetGroups, the account-level TG quota. TooManyUniqueTargetGroupsPerLoadBalancer is produced when a TG becomes associated with the load balancer, e.g. through CreateRule—which is exactly where issue #4848 reports the failure. So in the reported production scenario, this new branch is never reached. The new test masks this by having mockTargetGroupManager.Create() synthesize an error that the real CreateTargetGroup API doesn't produce.
P1 — The claimed two-reconcile recovery path cannot work when the TG is still referenced. The PR says that if inline target-group deletion gets ResourceInUse, the next reconcile will find that the listener rules from the previous cycle are already gone. But TargetGroupSynthesizer runs before ListenerRuleSynthesizer, and the stack deployer immediately returns when any Synthesize() call errors. Therefore, if the inline TG delete fails with ResourceInUse, listener-rule synthesis never runs in that reconcile. On the next reconcile you reach the same TG first and fail again. There is no mechanism in this patch that makes the stale listener rule disappear between those attempts.

[dominikhei]

P1 — The PR handles the error at the wrong AWS operation. The new code catches TooManyUniqueTargetGroupsPerLoadBalancer from tgManager.Create(), which calls CreateTargetGroup. But AWS does not return that error from CreateTargetGroup; its documented quota error is TooManyTargetGroups, the account-level TG quota. TooManyUniqueTargetGroupsPerLoadBalancer is produced when a TG becomes associated with the load balancer, e.g. through CreateRule—which is exactly where issue #4848 reports the failure. So in the reported production scenario, this new branch is never reached. The new test masks this by having mockTargetGroupManager.Create() synthesize an error that the real CreateTargetGroup API doesn't produce. P1 — The claimed two-reconcile recovery path cannot work when the TG is still referenced. The PR says that if inline target-group deletion gets ResourceInUse, the next reconcile will find that the listener rules from the previous cycle are already gone. But TargetGroupSynthesizer runs before ListenerRuleSynthesizer, and the stack deployer immediately returns when any Synthesize() call errors. Therefore, if the inline TG delete fails with ResourceInUse, listener-rule synthesis never runs in that reconcile. On the next reconcile you reach the same TG first and fail again. There is no mechanism in this patch that makes the stale listener rule disappear between those attempts.

Thank you for the detailed feedback. I will have a look at it once I am back from holidays on Sunday!