Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BREAKING] Only store hash of distributions over time #12

Merged
merged 3 commits into from
Feb 18, 2025

Conversation

dee-kryvenko
Copy link
Member

Evaluation resource has been storing list of replicas to each history
record. It appears to still blow out of the waters the size of the
object on big number of replicas. This change removes the distribution
from history records on evaluations. The idea is that we store the last
known projected winning distribution and compare it to the current one,
and if current one becomes the winner - we store it insead. At no point
in time we should need to know any other non-winning distribution, other
than what was its hash, how many times we've seen it and when was the
last time we've seen it.

There would be an edge case when a while ago there was a very much
wanted distribution that no longer wanter but it's total seen count is
higher than any of the current distributions. When that huge part of the
history goes out of the bounds and getting erased - we might not have
current projection anymore. In that case evaluator will go in not ready
state for a while until the new projected winning distribution is clear.

This change updates CRDs in-place with removal of fields, which is ok
because this is still beta.

Evaluation resource has been storing list of replicas to each history
record. It appears to still blow out of the waters the size of the
object on big number of replicas. This change removes the distribution
from history records on evaluations. The idea is that we store the last
known projected winning distribution and compare it to the current one,
and if current one becomes the winner - we store it insead. At no point
in time we should need to know any other non-winning distribution, other
than what was its hash, how many times we've seen it and when was the
last time we've seen it.

There would be an edge case when a while ago there was a very much
wanted distribution that no longer wanter but it's total seen count is
higher than any of the current distributions. When that huge part of the
history goes out of the bounds and getting erased - we might not have
current projection anymore. In that case evaluator will go in not ready
state for a while until the new projected winning distribution is clear.

This change updates CRDs in-place with removal of fields, which is ok
because this is still beta.
@dee-kryvenko dee-kryvenko merged commit b0a7e00 into main Feb 18, 2025
6 checks passed
@dee-kryvenko dee-kryvenko deleted the compact-evaluation-history branch February 18, 2025 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant