New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Thundering herd causes canary to be stuck in finalising #1600
Comments
Use Docs: https://docs.flagger.app/usage/deployment-strategies#canary-release |
@stefanprodan Thanks for the response. Would it make sense to set the new primary replicas to be the same as canary when |
Describe the bug
When a canary for a deployment doing high rps is promoted to primary, it fails because it doesn't have enough replicas to handle the load.
To Reproduce
stepWeight: 3, interval: 1 (3% every min)
minReplicas: "3", maxReplicas: "30"
Expected behavior
The deployment should succeed
Actual behavior
Finalising
stateThis is because the new primary deployment replicas is only set when hpa ref is nil. This means the new primary deployment replica count will be set to hpa's min and since this is a small value, it cannot handle the load.
Additional context
Workarounds
stepWeightPromotion
to make sure it does a partial traffic shift - Since this already done as part of canary, it seems redundantstepWeightPromotion: 100 (or have another variable like promotionReplicas)
, primary replicas should be set to canary replicas - This seems logical but not sure how the hpa will react.The text was updated successfully, but these errors were encountered: