[fix][broker] Fix out-of-order issues with ConsistentHashingStickyKeyConsumerSelector #23327
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lhotari
requested review from
merlimat,
eolivelli,
equanz,
codelipenghui,
hrsakai and
poorbarcode
This was referenced Sep 20, 2024
[Bug] Hash range collision causes out-of-order cases between existing consumers in Key_Shared
#23315
Open
lhotari
force-pushed
the
lh-fix-ConsistentHashingStickyKeyConsumerSelector-select-bug
branch
from
ac7fdeb
to
cc78aec
Compare
lhotari
changed the title
|
I renamed this to "reduce" out-of-order issues since I'm not sure if the current model in this PR prevents the case that a consumer gets swapped when an unrelated consumer leaves. It seems that another hash ring layer is needed for the colliding consumers. That would prevent unnecessary changes. A hash ring inside a hash ring... |
lhotari
force-pushed
the
lh-fix-ConsistentHashingStickyKeyConsumerSelector-select-bug
branch
from
cc78aec
to
6a1be71
Compare
lhotari
force-pushed
the
lh-fix-ConsistentHashingStickyKeyConsumerSelector-select-bug
branch
2 times, most recently
from
04c751b
to
12577d5
Compare
lhotari
force-pushed
the
lh-fix-ConsistentHashingStickyKeyConsumerSelector-select-bug
branch
from
b28a199
to
cdf3490
Compare
lhotari
force-pushed
the
lh-fix-ConsistentHashingStickyKeyConsumerSelector-select-bug
branch
from
527e431
to
e5eb2ef
Compare
lhotari
force-pushed
the
lh-fix-ConsistentHashingStickyKeyConsumerSelector-select-bug
branch
from
06db6e3
to
bae30bc
Compare
lhotari
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #23315
Fixes #23321
Motivation
See #23315 and #23321. This PR fixes a bug in ConsistentHashingStickyKeyConsumerSelector where the selected consumer for a hash could change across 2 existing consumers when a consumer was removed.
In addition it fixes the problem that ConsistentHashingStickyKeyConsumerSelector.getConsumerKeyHashRanges() returned invalid information. These are fixed together since they are in the same code area and fixing getConsumerKeyHashRanges is required for testing the consistent selection of a consumer when there are hash collisions.
Modifications
Add a solution that contains a solution to distribute the selected consumer for consumers with the same name across the hash ring, but in a way where it prevents from the consumer switching across 2 existing consumers.
Documentation
docdoc-requireddoc-not-neededdoc-complete