Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] avoid etcd updates when AWs do not have preemption condition generated #632

Open
asm582 opened this issue Sep 13, 2023 · 0 comments

Comments

@asm582
Copy link
Member

asm582 commented Sep 13, 2023

Describe the Bug

AWs were sending etcd updates for every run of the preemption thread. The update should only be send the AWs have preempt condition. the side effect of this is that the current AW will be sent to back-off queue and wont be dispatched for next 20 seconds (default) backoff time.

Codeflare Stack Component Versions

Please specify the component versions in which you have encountered this bug.

Codeflare SDK:
MCAD: Release 1.34.1 and main

Steps to Reproduce the Bug

Submit an AW that takes a very long time to spawn pods, the AW will get preempted and will be sent to the backoff queue.

What Have You Already Tried to Debug the Issue?

Same as above

Expected Behavior

AWs should only be preempted and backed off when they have a valid preemption condition.

Screenshots, Console Output, Logs, etc.

NA

Affected Releases

Release 1.34.1

Additional Context

NA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

2 participants