Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update Produce and Fetch APIs to support zstd compression #619

Merged

Conversation

ShapovalovaIrina
Copy link
Contributor

@ShapovalovaIrina ShapovalovaIrina commented Feb 17, 2025

Since the kafka_protocol 4.2.1 supports the zstd compression protocol, for convenience I also added it to brod.

I had to upgrade the versions of the Produce and Fetch APIs to those that support the required compression type. Unfortunately, I couldn't find a normal changelog, so I relied on the comments in the source code.

Produce API

Version 5 -> 7

  • Request
    • Version 5 is the same as version 3
    • Version 6 is the same as version 3
    • Starting in version 7, records can be produced using ZStandard compression
  • Response
    • Version 5 added LogStartOffset to filter out spurious. OutOfOrderSequenceExceptions on the client
    • Version 6 no changes
    • Version 7 no changes

Fetch API

Version 7 -> 10

  • Request
    • Version 7 adds incremental fetch request support
    • Version 8 is the same as version 7
    • Version 9 adds CurrentLeaderEpoch, as described in KIP-320 (parameter is ignorable and has default value)
    • Version 10 indicates that we can use the ZStd compression algorithm, as described in KIP-110
  • Response
    • Version 7 adds incremental fetch request support
    • Starting in version 8, on quota violation, brokers send out responses before throttling
    • Version 9 is the same as version 8
    • Version 10 indicates that the response data can use the ZStd compression algorithm, as described in KIP-110

There don't seem to be any significant changes between versions, so there are no other changes in the source code.

I also uncommented the tests for lz4, since this compression type is also supported in kafka_protocol.

@ShapovalovaIrina
Copy link
Contributor Author

@zmstone Hi! Sorry to bother you, but could you please leave some feedback about the PR?

@zmstone zmstone merged commit 8af2314 into kafka4beam:master Feb 27, 2025
14 checks passed
@zmstone
Copy link
Contributor

zmstone commented Feb 27, 2025

Thank you @ShapovalovaIrina

@zmstone
Copy link
Contributor

zmstone commented Feb 27, 2025

Will tag 4.4.0 after #620 is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants