Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KAFKA-16625: Reverse lookup map from topic partitions to members #15974

Merged
merged 60 commits into from
May 25, 2024
Merged

KAFKA-16625: Reverse lookup map from topic partitions to members #15974

merged 60 commits into from
May 25, 2024

Conversation

rreddy-22
Copy link
Contributor

During performance testing we observed that calculating the unassigned partitions by first initializing the set with all of the topic partitions within the Uniform assignor is a costly process, this can be improved by using a reverse lookup map between topicIdPartition and the member.

This patch adds the following:

  1. A new map to the consumer group state called partitionAssignments which is a reverse lookup map from each partition to the member it is assigned to. This map is updated every time the target assignment is updated.
  2. A new interface called GroupSpec which is implemented by the existing assignmentSpec.
  3. The GroupSpec interface contains a new method called isPartitionAssigned.

The most performance improvements should be noticeable in the Uniform Optimized Assignment Builder.

rreddy-22 and others added 30 commits May 3, 2024 16:06
@rreddy-22
Add isSubscriptionHomogeneous value in ConsumerGroup
7d92a43
@rreddy-22
Add isSubscriptionHomogeneous flag to TargetAssignmentBuilder
f4ec728
@rreddy-22
Add isSubscriptionHomogeneous flag to AssignmentSpec
0cb1a5f
@rreddy-22
Edit existing tests to incorporate changes
1b7b693
@rreddy-22
add getter method in AssignmentSpec
e7c68b5
@rreddy-22
Modify UniformAssignor to use flag to decide which assignment builder…
11b7b1f 
… to use
@rreddy-22
Add optimization to the Range Assignor using flag
e710fca
@rreddy-22
Change flag to TimelineObject in Consumer Group
b74f713
@rreddy-22
Incorporate updating subs model into update subs topic names method
0785f3b
@rreddy-22
minor
b7c2a24
@rreddy-22
Revert to brute force check for subsModelUpdate
db1c601
@rreddy-22
Convert group subscription model to enum from isSubscriptionHomogenen…
2afb1bc 
…ous flag
@rreddy-22
Enum added in a separate file
50812d5
@rreddy-22
Add unit test
9dada36
@rreddy-22
minor
d63e491
@rreddy-22
Rename subsModel to subsType, PR comments
e3a8b95
@rreddy-22
Use enum in benchmarks
9fa0ab3
@rreddy-22
move out subs updates into the group metadata manager
5c4c4ed
@rreddy-22
minor
0d25f3a
@rreddy-22
minor
a4ce616
@rreddy-22
Address comments
9cdd729
@rreddy-22
minor
9ca54c3
@rreddy-22
address comments
a36d59c
@rreddy-22
Initialize target assignment map in range assignor
9d4fbfe
@rreddy-22
Update numOfMembers in heartbeat
01a7b6e
@rreddy-22
Revert initialising new map in range assignor
d0a0ef9
@rreddy-22
Merge remote-tracking branch 'upstream/trunk' into KAFKA-16587_Add_Su…
7e6a7eb 
…bcription_Model_to_Group_State
@AyoubOm @rreddy-22
MINOR: Fix streams javadoc links (apache#15900)
ca53724 
Reviewers: Chia-Ping Tsai <[email protected]>
@FrankYang0529 @rreddy-22
MINOR: use classic consumer with ZK mode for DeleteOffsetsConsumerGro…
fa9f388 
…upCommandIntegrationTest (apache#15872)

Reviewers: Chia-Ping Tsai <[email protected]>
@AndrewJSchofield @rreddy-22
MINOR: Add missing RPCs to security.html (apache#15878)
fa22563 
Reviewers: Chia-Ping Tsai <[email protected]>
dajac
dajac reviewed May 21, 2024
View reviewed changes
Copy link
Contributor

@dajac dajac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rreddy-22 Thanks for the update. I left a few minor comments.

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupSpec.java Outdated Show resolved Hide resolved
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupSpecImpl.java Show resolved Hide resolved
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroup.java Outdated Show resolved Hide resolved
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroup.java Outdated Show resolved Hide resolved
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroup.java Show resolved Hide resolved
...coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroupTest.java Outdated Show resolved Hide resolved
...coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroupTest.java Show resolved Hide resolved
dajac
dajac reviewed May 22, 2024
View reviewed changes
Copy link
Contributor

@dajac dajac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rreddy-22 Thanks for the update. I left a few small comments for consideration.

group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupSpecImpl.java Show resolved Hide resolved
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupSpecImpl.java Show resolved Hide resolved
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupSpecImpl.java Outdated Show resolved Hide resolved
group-coordinator/src/test/java/org/apache/kafka/coordinator/group/AssignmentTestUtil.java Outdated Show resolved Hide resolved
...coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroupTest.java Show resolved Hide resolved
...coordinator/src/test/java/org/apache/kafka/coordinator/group/consumer/ConsumerGroupTest.java Outdated Show resolved Hide resolved
...benchmarks/src/main/java/org/apache/kafka/jmh/assignor/TargetAssignmentBuilderBenchmark.java Outdated Show resolved Hide resolved
jeffkbkim
jeffkbkim reviewed May 24, 2024
View reviewed changes
group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GroupSpecImpl.java
public AssignmentSpec(
/**
* Reverse lookup map representing topic partitions with
* their current member assignments.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think i'm slightly confused again on the types of assignments.

Target Assignment: the assignment we will reconcile to.

Current Assignment: the previous group epoch's target assignment.

is this correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes

dajac
dajac approved these changes May 24, 2024
View reviewed changes
Copy link
Contributor

@dajac dajac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dajac dajac merged commit a8d166c into apache:trunk May 25, 2024
1 check failed
apourchet added a commit to apourchet/kafka that referenced this pull request May 29, 2024
@apourchet
Squashed commit of the following:
572377b 
commit cc269b0
Author: Antoine Pourchet <[email protected]>
Date:   Wed May 29 14:15:02 2024 -0600

    KAFKA-15045: (KIP-924 pt. 14) Callback to TaskAssignor::onAssignmentComputed (apache#16123)

    This PR adds the logic and wiring necessary to make the callback to
    TaskAssignor::onAssignmentComputed with the necessary parameters.

    We also fixed some log statements in the actual assignment error
    computation, as well as modified the ApplicationState::allTasks method
    to return a Map instead of a Set of TaskInfos.

    Reviewers: Anna Sophie Blee-Goldman <[email protected]>

commit 862ea12
Author: Eugene Mitskevich <[email protected]>
Date:   Wed May 29 16:14:37 2024 -0400

    MINOR: Fix rate metric spikes (apache#15889)

    Rate reports value in the form of sumOrCount/monitoredWindowSize. It has a bug in monitoredWindowSize calculation, which leads to spikes in result values.

    Reviewers: Jun Rao <[email protected]>

commit 0f0c9ec
Author: gongxuanzhang <[email protected]>
Date:   Thu May 30 01:08:17 2024 +0800

    KAFKA-16771 First log directory printed twice when formatting storage (apache#16010)

    Reviewers: Chia-Ping Tsai <[email protected]>

commit 2d9994e
Author: Andrew Schofield <[email protected]>
Date:   Wed May 29 16:31:52 2024 +0100

    KAFKA-16722: Introduce ConsumerGroupPartitionAssignor interface (apache#15998)

    KIP-932 introduces share groups to go alongside consumer groups. Both kinds of group use server-side assignors but it is unlikely that a single assignor class would be suitable for both. As a result, the KIP introduces specific interfaces for consumer group and share group partition assignors.

    This PR introduces only the consumer group interface, `o.a.k.coordinator.group.assignor.ConsumerGroupPartitionAssignor`. The share group interface will come in a later release. The existing implementations of the general `PartitionAssignor` interface have been changed to implement `ConsumerGroupPartitionAssignor` instead and all other code changes are just propagating the change throughout the codebase.

    Note that the code in the group coordinator that actually calculates assignments uses the general `PartitionAssignor` interface so that it can be used with both kinds of group, even though the assignors themselves are specific.

    Reviewers: Apoorv Mittal <[email protected]>, David Jacot <[email protected]>

commit 0b75cf7
Author: gongxuanzhang <[email protected]>
Date:   Wed May 29 22:38:00 2024 +0800

    KAFKA-16705 the flag "started" of RaftClusterInstance is false even though the cluster is started (apache#15946)

    Reviewers: Chia-Ping Tsai <[email protected]>

commit 8d11d95
Author: Loïc GREFFIER <[email protected]>
Date:   Wed May 29 14:09:22 2024 +0200

    KAFKA-16448: Add ProcessingExceptionHandler interface and implementations (apache#16090)

    This PR is part of KAFKA-16448 which aims to bring a ProcessingExceptionHandler to Kafka Streams in order to deal with exceptions that occur during processing.

    This PR brings ProcessingExceptionHandler interface and default implementations.

    Co-authored-by: Dabz <[email protected]>
    Co-authored-by: sebastienviale <[email protected]>

    Reviewer: Bruno Cadonna <[email protected]>

commit b73f479
Author: Ramin Gharib <[email protected]>
Date:   Wed May 29 13:12:54 2024 +0200

    KAFKA-16362: Fix type-unsafety in KStreamKStreamJoin caused by isLeftSide (apache#15601)

    The introduced changes provide a cleaner definition of the join side in KStreamKStreamJoin. Before, this was done by using a Boolean flag, which led to returning a raw LeftOrRightValue without generic arguments because the generic type arguments depended on the boolean input.

    Reviewers: Greg Harris <[email protected]>, Bruno Cadonna <[email protected]>

commit 897cab2
Author: Luke Chen <[email protected]>
Date:   Wed May 29 15:30:18 2024 +0800

    KAFKA-16399: Add JBOD support in tiered storage (apache#15690)

    After JBOD is supported in KRaft, we should also enable JBOD support in tiered storage. Unit tests and Integration tests are also added.

    Reviewers: Satish Duggana <[email protected]>, Kamal Chandraprakash <[email protected]>, Igor Soarez <[email protected]>, Mickael Maison <[email protected]>

commit eefd114
Author: Dongnuo Lyu <[email protected]>
Date:   Wed May 29 02:21:30 2024 -0400

    KAFKA-16832; LeaveGroup API for upgrading ConsumerGroup (apache#16057)

    This patch implements the LeaveGroup API to the consumer groups that are in the mixed mode.

    Reviewers: Jeff Kim <[email protected]>, David Jacot <[email protected]>

commit 9562143
Author: A. Sophie Blee-Goldman <[email protected]>
Date:   Tue May 28 21:35:02 2024 -0700

    HOTFIX: remove unnecessary list creation (apache#16117)

    Removing a redundant list declaration in the new StickyTaskAssignor implementation

    Reviewers: Antoine Pourchet <[email protected]>

commit d64e3fb
Author: Antoine Pourchet <[email protected]>
Date:   Tue May 28 20:43:30 2024 -0600

    KAFKA-15045: (KIP-924 pt. 13) AssignmentError calculation added (apache#16114)

    This PR adds the post-processing of the TaskAssignment to figure out if the new assignment is valid, and return an AssignmentError otherwise.

    Reviewers: Anna Sophie Blee-Goldman <[email protected]>

commit 8d243df
Author: Antoine Pourchet <[email protected]>
Date:   Tue May 28 19:01:18 2024 -0600

    KAFKA-15045: (KIP-924 pt. 12) Wiring in new assignment configs and logic (apache#16074)

    This PR creates the new public config of KIP-924 in StreamsConfig and uses it to instantiate user-created TaskAssignors. If such a TaskAssignor is found and successfully created we then use that assignor to perform the task assignment, otherwise we revert back to the pre KIP-924 world with the internal task assignors.

    Reviewers: Anna Sophie Blee-Goldman <[email protected]>, Almog Gavra <[email protected]>

commit 56ee139
Author: Antoine Pourchet <[email protected]>
Date:   Tue May 28 18:05:51 2024 -0600

    KAFKA-15045: (KIP-924 pt. 11) Implemented StickyTaskAssignor (apache#16052)

    This PR implements the StickyTaskAssignor with the new KIP 924 API.

    Reviewers: Anna Sophie Blee-Goldman <[email protected]>

commit 59ba555
Author: Nick Telford <[email protected]>
Date:   Wed May 29 00:23:23 2024 +0100

    KAFKA-15541: Add oldest-iterator-open-since-ms metric (apache#16041)

    Part of [KIP-989](https://cwiki.apache.org/confluence/x/9KCzDw).

    This new `StateStore` metric tracks the timestamp that the oldest
    surviving Iterator was created.

    This timestamp should continue to climb, and closely track the current
    time, as old iterators are closed and new ones created. If the timestamp
    remains very low (i.e. old), that suggests an Iterator has leaked, which
    should enable users to isolate the affected store.

    It will report no data when there are no currently open Iterators.

    Reviewers: Matthias J. Sax <[email protected]>

commit 4eb60b5
Author: Frederik Rouleau <[email protected]>
Date:   Tue May 28 23:56:47 2024 +0200

    KAFKA-16507 Add KeyDeserializationException and ValueDeserializationException with record content (apache#15691)

    Implements KIP-1036.

    Add raw ConsumerRecord data to RecordDeserialisationException to make DLQ implementation easier.

    Reviewers: Kirk True <[email protected]>, Andrew Schofield <[email protected]>, Matthias J. Sax <[email protected]>

commit 4d04eb8
Author: PoAn Yang <[email protected]>
Date:   Wed May 29 03:13:33 2024 +0800

    KAFKA-16796 Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder (apache#16064)

    Reviewers: Chia-Ping Tsai <[email protected]>

commit a649bc4
Author: Luke Chen <[email protected]>
Date:   Wed May 29 00:05:49 2024 +0800

    KAFKA-16711: Make sure to update highestOffsetInRemoteStorage after log dir change (apache#15947)

    Reviewers: Kamal Chandraprakash<[email protected]>, Satish Duggana <[email protected]>

commit 64f699a
Author: Omnia Ibrahim <[email protected]>
Date:   Tue May 28 15:22:54 2024 +0100

    KAFKA-15853: Move general configs out of KafkaConfig (apache#16040)

    Reviewers: Mickael Maison <[email protected]>, Chia-Ping Tsai <[email protected]>

commit 699438b
Author: Sanskar Jhajharia <[email protected]>
Date:   Tue May 28 16:34:44 2024 +0530

    MINOR: Fix the config name in ProducerFailureHandlingTest (apache#16099)

    When moving from KafkaConfig.ReplicaFetchMaxBytesProp we used ReplicationConfigs.REPLICA_LAG_TIME_MAX_MS_CONFIG instead of ReplicationConfigs.REPLICA_FETCH_MAX_BYTES_CONFIG. This PR patches the same.

    Reviewers: Omnia Ibrahim <[email protected]>, Manikumar Reddy <[email protected]>

commit a57c05b
Author: Ken Huang <[email protected]>
Date:   Tue May 28 17:42:33 2024 +0900

    KAFKA-16805 Stop using a ClosureBackedAction to configure Spotbugs reports (apache#16081)

    Reviewers: Chia-Ping Tsai <[email protected]>

commit 91284d8
Author: Luke Chen <[email protected]>
Date:   Tue May 28 12:23:34 2024 +0800

    KAFKA-16709: abortAndPauseCleaning only when future log is not existed (apache#15951)

    When doing alter replica logDirs, we'll create a future log and pause log cleaning for the partition( here). And this log cleaning pausing will resume after alter replica logDirs completes (here). And when in the resuming log cleaning, we'll decrement 1 for the LogCleaningPaused count. Once the count reached 0, the cleaning pause is really resuming. (here). For more explanation about the logCleaningPaused state can check here.

    But, there's still one factor that could increase the LogCleaningPaused count: leadership change (here). When there's a leadership change, we'll check if there's a future log in this partition, if so, we'll create future log and pauseCleaning (LogCleaningPaused count + 1). So, if during the alter replica logDirs:

    1. alter replica logDirs for tp0 triggered (LogCleaningPaused count = 1)
    2. tp0 leadership changed (LogCleaningPaused count = 2)
    3. alter replica logDirs completes, resuming logCleaning (LogCleaningPaused count = 1)
    4. LogCleaning keeps paused because the count is always >  0

    This PR fixes this issue by only abortAndPauseCleaning when future log is not existed. We did the same check in alterReplicaLogDirs. So this change can make sure there's only 1 abortAndPauseCleaning for either abortAndPauseCleaning or maybeAddLogDirFetchers. Tests also added.

    Reviewers: Chia-Ping Tsai <[email protected]>, Igor Soarez <[email protected]>

commit adab48d
Author: Greg Harris <[email protected]>
Date:   Mon May 27 18:33:01 2024 -0700

    MINOR: Disable JDK 11 and 17 tests on PRs (apache#16051)

    Signed-off-by: Greg Harris <[email protected]>
    Reviewers: Justine Olshan <[email protected]>, David Arthur <[email protected]>, Ismael Juma <[email protected]>, Luke Chen <[email protected]>, Chia-Ping Tsai <[email protected]>

commit bac8df5
Author: Colin P. McCabe <[email protected]>
Date:   Mon May 27 08:53:53 2024 -0700

    MINOR: fix typo in KAFKA-16515

commit da3304e
Author: David Jacot <[email protected]>
Date:   Mon May 27 17:10:37 2024 +0200

    KAFKA-16371; fix lingering pending commit when handling OFFSET_METADATA_TOO_LARGE (apache#16072)

    This patch was initially created in apache#15536.

    When there is a commit for multiple topic partitions and some, but not all, exceed the offset metadata limit, the pending commit is not properly cleaned up leading to UNSTABLE_OFFSET_COMMIT errors when trying to fetch the offsets with read_committed. This change makes it so the invalid commits are not added to the pendingOffsetCommits set.

    Co-authored-by: Kyle Phelps <[email protected]>

    Reviewers: Chia-Ping Tsai <[email protected]>, Justine Olshan <[email protected]>

commit 524ad1e
Author: Kamal Chandraprakash <[email protected]>
Date:   Mon May 27 15:14:23 2024 +0530

    KAFKA-16452: Don't throw OOORE when converting the offset to metadata (apache#15825)

    Don't throw OFFSET_OUT_OF_RANGE error when converting the offset to metadata, and next time the leader should increment the high watermark by itself after receiving fetch requests from followers. This can happen when checkpoint files are missing and being elected as a leader.

    Reviewers: Luke Chen <[email protected]>, Jun Rao <[email protected]>

commit d9ee9c9
Author: Nick Telford <[email protected]>
Date:   Sat May 25 20:22:56 2024 +0100

    KAFKA-15541: Use LongAdder instead of AtomicInteger (apache#16076)

    `LongAdder` performs better than `AtomicInteger` when under contention
    from many threads. Since it's possible that many Interactive Query
    threads could create a large number of `KeyValueIterator`s, we don't
    want contention on a metric to be a performance bottleneck.

    The trade-off is memory, as `LongAdder` uses more memory to space out
    independent counters across different cache lines. In practice, I don't
    expect this to cause too many problems, as we're only constructing 1
    per-store.

    Reviewers: Matthias J. Sax <[email protected]>

commit a8d166c
Author: Ritika Reddy <[email protected]>
Date:   Sat May 25 09:06:15 2024 -0700

    KAFKA-16625; Reverse lookup map from topic partitions to members (apache#15974)

    This patch speeds up the computation of the unassigned partitions by exposing the inverted target assignment. It allows the assignor to check whether a partition is assigned or not.

    Reviewers: Jeff Kim <[email protected]>, David Jacot <[email protected]>

commit d585a49
Author: Jeff Kim <[email protected]>
Date:   Fri May 24 16:33:57 2024 -0400

    KAFKA-16831: CoordinatorRuntime should initialize MemoryRecordsBuilder with max batch size write limit (apache#16059)

    CoordinatorRuntime should initialize MemoryRecordsBuilder with max batch size write limit. Otherwise, we default the write limit to the min buffer size of 16384 for the write limit. This causes the coordinator to threw RecordTooLargeException even when it's under the 1MB max batch size limit.

    Reviewers: David Jacot <[email protected]>

commit 8eea6b8
Author: Edoardo Comar <[email protected]>
Date:   Fri May 24 20:33:00 2024 +0100

    MINOR: mention KAFKA-15905 in docs "Notable changes in 3.7.1" (apache#16070)

    * MINOR: mention KAFKA-15905 in docs "Notable changes in 3.7.1/3.8.0"

    Co-Authored-By: Adrian Preston <[email protected]>

commit 4f55786
Author: Colin P. McCabe <[email protected]>
Date:   Mon May 20 15:41:52 2024 -0700

    KAFKA-16515: Fix the ZK Metadata cache confusion between brokers and controllers

    ZkMetadataCache could theoretically return KRaft controller information from a call to
    ZkMetadataCache.getAliveBrokerNode, which doesn't make sense. KRaft controllers are not part of the
    set of brokers. The only use-case for this functionality was in MetadataCacheControllerNodeProvider
    during ZK migration, where it allowed ZK brokers in migration mode to forward requests to
    kcontrollers when appropriate. This PR changes MetadataCacheControllerNodeProvider to simply
    delegate to quorumControllerNodeProvider in this case.

    Reviewers: José Armando García Sancio <[email protected]>

commit 90892ae
Author: Colin P. McCabe <[email protected]>
Date:   Mon May 20 16:23:27 2024 -0700

    KAFKA-16516: Fix the controller node provider for broker to control channel

    Fix the code in the RaftControllerNodeProvider to query RaftManager to find Node information,
    rather than consulting a static map. Add a RaftManager.voterNode function to supply this
    information. In KRaftClusterTest, add testControllerFailover to get more coverage of controller
    failovers.

    Reviewers: José Armando García Sancio <[email protected]>

commit 2432a18
Author: KrishVora01 <[email protected]>
Date:   Fri May 24 22:21:02 2024 +0530

    KAFKA-16373: KIP-1028:  Adding code to support Apache Kafka Docker Official Images (apache#16027)

    This PR aims to add JVM based Docker Official Image for Apache Kafka as per the following KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka

    This PR adds the following functionalities:
    Introduces support for Apache Kafka Docker Official Images via:

    GitHub Workflows:

    - Workflow to prepare static source files for Docker images
    - Workflow to build and test Docker official images
    - Scripts to prepare source files and perform Docker image builds and tests

    A new directory for Docker official images, named docker/docker_official_images. This is the new directory to house all Docker Official Image assets.

    Co-authored-by: Vedarth Sharma <[email protected]>

    Reviewers: Manikumar Reddy <[email protected]>, Vedarth Sharma <[email protected]>

commit 0143c72
Author: Lianet Magrans <[email protected]>
Date:   Fri May 24 14:19:43 2024 +0200

    KAFKA-16815: Handle FencedInstanceId in HB response (apache#16047)

    Handle FencedInstanceIdException that a consumer may receive in the heartbeat response. This will be the case when a static consumer is removed from the group by and admin client, and another member joins with the same group.instance.id (allowed in). The first member will receive a FencedInstanceId on its next heartbeat. The expectation is that this should be handled as a fatal error.

    There are no actual changes in logic with this PR, given that without being handled, the FencedInstanceId was being treated as an "unexpected error", which are all treated as fatal errors, so the outcome remains the same. But we're introducing this small change just for accuracy in the logic and the logs: FencedInstanceId is expected during heartbeat, a log line is shown describing the situation and why it happened (and it's treated as a fatal error, just like it was before this PR).

    This PR also improves the test to ensure that the error propagated to the app thread matches the one received in the HB.

    Reviewers: Andrew Schofield <[email protected]>, David Jacot <[email protected]>

commit c5cd190
Author: Gantigmaa Selenge <[email protected]>
Date:   Fri May 24 11:50:47 2024 +0100

    MINOR: Refactor SSL/SASL admin integration tests to not use a custom authorizer (apache#15377)

    Reviewers: Mickael Maison <[email protected]>

commit 520aa86
Author: Jeff Kim <[email protected]>
Date:   Fri May 24 03:51:50 2024 -0400

    KAFKA-16626; Lazily convert subscribed topic names to topic ids (apache#15970)

    This patch aims to remove the data structure that stores the conversion from topic names to topic ids which was taking time similar to the actual assignment computation. Instead, we reuse the already existing ConsumerGroupMember.subscribedTopicNames() and do the conversion to topic ids when the iterator is requested.

    Reviewers: David Jacot <[email protected]>

commit 6941598
Author: Krishna Agarwal <[email protected]>
Date:   Fri May 24 12:16:01 2024 +0530

    KAFKA-16826: Integrate Native Docker Image with github actions (apache#16045)

    This PR integrates the Native docker image with the existing github action jobs for the jvm docker image of AK.

    The integration is done to the following actions:

    docker_build_and_test.yml: Builds the docker image and runs sanity tests and CVE scan
    docker_rc_release.yml: Builds the RC docker image for both amd and arm platform and pushes it to the dockerhub.
    docker_promote.yml: Promotes the RC docker image to the released image tag

    Reviewers: Manikumar Reddy <[email protected]>, Vedarth Sharma <[email protected]>

commit de32028
Author: Kuan-Po (Cooper) Tseng <[email protected]>
Date:   Fri May 24 05:25:53 2024 +0800

    KAFKA-16828 RackAwareTaskAssignorTest failed (apache#16044)

    Reviewers: Chia-Ping Tsai <[email protected]>

commit 11ad5e8
Author: Greg Harris <[email protected]>
Date:   Thu May 23 13:23:18 2024 -0700

    MINOR: Refactor Values class to fix checkstyle, add benchmark, optimize exceptions (apache#15469)

    Signed-off-by: Greg Harris <[email protected]>
    Reviewers: Mickael Maison <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dajac dajac dajac approved these changes

@jeffkbkim jeffkbkim jeffkbkim approved these changes

Assignees
No one assigned
Labels
KIP-848
Projects
None yet
Milestone
No milestone
6 participants
@rreddy-22 @dajac @jeffkbkim @AyoubOm @FrankYang0529 @AndrewJSchofield