Separate part sizes for GET and PUT

[jamesbornholt]

Tell us more about this new feature.

Sometimes we might want to use different part sizes for uploads vs downloads, for cost or performance reasons. For example, you might want a large PUT part size to support large file uploads, but a smaller GET part size for maximum throughput. Today they're configured with the same --part-size argument. It would be nice if they could be configured separately.

[crrow]

Hi, can I work on this ?

[dannycjones]

Hey @crrow, yes - we'd be delighted!

To recap, we're wanting to make it possible to specify the part size for reading (GET) separately from writing/uploading (PUT). While the default 8MiB works well for both, sometimes the best value can be different. Plus, PUTs have a minimum size of 5MiB meaning today the floor on part size with Mountpoint is effectively 5MiB if you're uploading as well.

The AWS CRT team added an option to specify part size per-request in awslabs/aws-c-s3#393. I hope it's a matter of plumbing in the logic between Mountpoint CLI arguments and where the S3 meta request is created, including a small update to the CRT bindings.

Let us know if you get started on this, we'd be happy to help!

[crrow]

@dannycjones Sure, I will get started on this. Thanks for recapping, I'll let you know if I have any questions.

[dannycjones]

I've set #949 for merging, so this can be closed once we release (and I'll add some updates to the docs just before release).

[dannycjones]

This feature was released in Mountpoint v1.8.0! 🎉

The new arguments are --read-part-size <BYTES> and --write-part-size <BYTES>. All the arguments are optional, so you can leave read part size to default for example while configuring the write part size.

Thank you to @crrow for the contribution (#949)! 🙌