Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RoundRobin Assignor doesn't acknowledge Member subscribed topics in assignment #1679

Open
edeesis opened this issue Apr 17, 2024 · 0 comments

Comments

@edeesis
Copy link

edeesis commented Apr 17, 2024

Describe the bug
Consumers in a Consumer Groups can subscribe to different topics. When a group rebalances, the elected leader uses Assigner#assign to determine what topic-partitions to assign to which group members.

However, the current implementation only looks at the leader's subscribed topics when it does this assignment, rather than the subscribed topics of each of the members.

Here's an example:

Give 2 topics, Topic A with 2 partitions, Topic B with 2 partitions

Consumer Group ID (Leader?) Subscribed Topics Client ID (used to generate Group Member ID) Assigned Topic Partitions
group-1 (Leader) Topic A group-1-1 Topic A Partition 1
group-1 Topic B group-1-2 No assignments
group-1 Topic A group-1-3 No Assignments
group-1 Topic B group-1-0 Topic A Partition 0

The client IDs are just to show that there's a alphanumeric sort applied to member IDs to keep partition assignment relatively stable across rebalances.

To Reproduce

  1. Create two topics, each with 1 partition, called topic-a, and topic-b
  2. Run two consumers, with the same group ID. The first consumer should subscribe to topic-a and the client ID of "client-id-1" and the second should subscribe to topic-b with the client ID "client-id-0". The first consumer should become the leader when it starts up.
  3. Use the KafkaAdmin describeGroup to list the group members and assignments. You should see the second consumer assigned topic-a, even though its subscribed to topic-b.

Expected behavior
The round robin assigner should assign topic-partitions only to members that are subscribed to those topics.

Observed behavior
The round robin assigner assigns topics to consumers that they aren't subscribed to.

Environment:

  • OS: Mac OS 14.4.1 (in Docker)
  • KafkaJS version: 2.2.4
  • Kafka version: 7.3.1-ccs (Confluent)
  • NodeJS version: 20.10.0

Additional context
The Java RoundRobinAssignor more intelligently maps partition assignments via subscribed topics as described in the JavaDocs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant