Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: don't use replicaset informer write back #3332

Draft
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

zachaller
Copy link
Collaborator

Checklist:

  • Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
  • The title of the PR is (a) conventional with a list of types and scopes found here, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
  • I've signed my commits with DCO
  • I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
  • My builds are green. Try syncing with master if they are not.
  • My organization is added to USERS.md.

@zachaller zachaller marked this pull request as draft January 26, 2024 22:31
Copy link

codecov bot commented Jan 26, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.20%. Comparing base (8405f2e) to head (1265f0c).
Report is 99 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #3332      +/-   ##
==========================================
- Coverage   81.83%   78.20%   -3.64%     
==========================================
  Files         135      158      +23     
  Lines       20688    18378    -2310     
==========================================
- Hits        16931    14372    -2559     
- Misses       2883     3102     +219     
- Partials      874      904      +30     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

github-actions bot commented Jan 26, 2024

Go Published Test Results

2 159 tests   2 159 ✅  2m 50s ⏱️
  119 suites      0 💤
    1 files        0 ❌

Results for commit 1265f0c.

♻️ This comment has been updated with latest results.

Copy link

sonarcloud bot commented Jan 26, 2024

Quality Gate Passed Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

1 New issue
0 Security Hotspots
No data about Coverage
8.5% Duplication on New Code

See analysis details on SonarCloud

Copy link
Contributor

github-actions bot commented Jan 26, 2024

E2E Tests Published Test Results

  4 files    4 suites   3h 28m 14s ⏱️
110 tests  96 ✅  6 💤  8 ❌
462 runs  417 ✅ 24 💤 21 ❌

For more details on these failures, see this check.

Results for commit 1265f0c.

♻️ This comment has been updated with latest results.

@jessesuen
Copy link
Member

I thought the informer writeback solved a lot of our issues. Has this caused problems?

@zachaller zachaller force-pushed the dont-use-rs-informer-writeback branch from 4dce3b2 to 2e37889 Compare April 1, 2024 22:22
@edmondshtogu
Copy link

edmondshtogu commented Apr 24, 2024

Hi guys, I am facing the below issue in a specific cluster, while rollout fails to scale the RS and I believe this might fix it.

msg="failed to reconcileCanaryReplicaSets in syncReplicasOnly: failed to scaleReplicaSetAndRecordEvent in reconcileCanaryStableReplicaSet:L failed to scaleReplicaSet in scaleReplicaSetAndRecordEvent: error updating replicaset <rs-name>: Operation cannot be fulfilled on replicasets.apps \"<rs-name>:\": the object has been modified; please apply your changes to the latest version and try again" error="<nil>"

It seems that cache is keeping an old state of the newly created RS. From the audit logs it is only rollouts controller that manipulates it. I found that wen it happens, all the patches after create operation do fail.

Is there any plan to push this forward?

I am planing to build an image out of your code and see if will fix the issue.

@zachaller
Copy link
Collaborator Author

I am 98% sure this code won't fix that issue, I am however now looking into the root cause so hopefully will find something soon. The main issue is that within a single reconcile we try to update the replicaset multiple times and do not keep the in memory state updated.

Copy link

sonarcloud bot commented May 1, 2024

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
9.6% Duplication on New Code

See analysis details on SonarCloud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants