Does changing the chunks of a zarr Array affect compression ratio? #1627
-
|
Thank you for any help on this |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
In general, yes. Lossless compression fundamentally relies on the presence of repeated values in the data. In the limit of chunks with only one element, there can be no repeated values, and thus no compression. So generally speaking, as you make your chunks larger, you will have more opportunities for compression. The chunk shape can also make a big difference. To go beyond this very general statement, you would need to do some experiments with your actual dataset to see how compression ratio depends on chunk size and shape. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for the swift reply :) |
Beta Was this translation helpful? Give feedback.
In general, yes. Lossless compression fundamentally relies on the presence of repeated values in the data. In the limit of chunks with only one element, there can be no repeated values, and thus no compression. So generally speaking, as you make your chunks larger, you will have more opportunities for compression. The chunk shape can also make a big difference.
To go beyond this very general statement, you would need to do some experiments with your actual dataset to see how compression ratio depends on chunk size and shape.