feat: update Produce and Fetch APIs to support zstd compression #619
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since the
kafka_protocol 4.2.1
supports thezstd
compression protocol, for convenience I also added it to brod.I had to upgrade the versions of the
Produce
andFetch APIs
to those that support the required compression type. Unfortunately, I couldn't find a normal changelog, so I relied on the comments in the source code.Produce API
Version
5 -> 7
5
is the same as version 36
is the same as version 37
, records can be produced using ZStandard compression5
added LogStartOffset to filter out spurious. OutOfOrderSequenceExceptions on the client6
no changes7
no changesFetch API
Version
7 -> 10
7
adds incremental fetch request support8
is the same as version 79
adds CurrentLeaderEpoch, as described in KIP-320 (parameter is ignorable and has default value)10
indicates that we can use the ZStd compression algorithm, as described in KIP-1107
adds incremental fetch request support8
, on quota violation, brokers send out responses before throttling9
is the same as version 810
indicates that the response data can use the ZStd compression algorithm, as described in KIP-110There don't seem to be any significant changes between versions, so there are no other changes in the source code.
I also uncommented the tests for
lz4
, since this compression type is also supported in kafka_protocol.