feat: add default checksums for appendable writer (excludes appendable takeover writer)

[krishnamd-jkp]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @krishnamd-jkp, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces default checksumming capabilities for gRPC appendable writers, enhancing data integrity during object writes. It also includes important documentation regarding the performance implications of automatic checksum computation and provides a mechanism to disable it. A temporary measure has been implemented to disable checksums for "takeover" writers, with a clear indication for future development.

Highlights

• Checksumming for Appendable Writers: Implemented default checksum calculation and sending for gRPC appendable writers, addressing a previous TODO to enhance data integrity.
• Performance Documentation: Added a note to the storage/doc.go documentation warning about potential CPU overhead from gRPC client's default auto checksumming and how users can disable it.
• Takeover Writer Handling: Introduced a takeoverWriter flag to explicitly disable checksums for 'takeover' appendable writers, with a TODO indicating future work to enable validation for this specific scenario.
• Checksum Logic Refactoring: Refactored the gRPCAppendBidiWriteBufferSender to use functional getters for checksum settings (checksumSettings and fullObjectChecksum), improving modularity and flexibility in how checksums are determined and applied.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds support for default checksumming for appendable writers in the gRPC client. It introduces logic to calculate and send checksums for chunks and for the full object, while also providing an option to disable this for performance reasons. A temporary flag is added to disable checksums for takeover writers, with a TODO to implement it later. The changes are well-documented and tested.

My review includes a suggestion to refactor how checksum settings are passed to the gRPCAppendBidiWriteBufferSender. Instead of using a function that is called on every send, I recommend storing the immutable settings as direct fields on the struct for better clarity and minor performance improvement.

[cpriti-os]

Can you change the title to explicitly call out the limitation: add default checksums for appendable writer except takeover writer. Feel free to rephrase, just want to ensure if this is released, the release notes should not mislead users.

[cpriti-os]

Have we done benchmarking and profiling for appendable writes checksum as well @krishnamd-jkp ?

[krishnamd-jkp]

Have we done benchmarking and profiling for appendable writes checksum as well @krishnamd-jkp ?

@cpriti-os Because this PR introduces checksums only for appendable writer only, the code path is similar to a resumable or one shot writers so there won't be any additional latency. The CPU overhead will also be the same because computation of checksum is done by the same function.

I'd assume things would be a little different for takeover writer as it has a slightly different logic where writer needs to accept the crc32c from server and then calculate checksums for further writes