Optimizing getsize_prefix further

[rabernat]

One great thing about Icechunk is that the manifests store info about all of the chunks, making it in theory fast to know the on-disk size of very large arrays. (This is slow with vanilla Zarr because it requires listing the object store.)

Today I discovered we are not quite there yet though.

import icechunk as ic
import xarray as xr

storage = ic.s3_storage(
    bucket="icechunk-public-data",
    prefix="v01/era5_weatherbench2",
    region="us-east-1",
    anonymous=True,
)

repo = ic.Repository.open(storage=storage)
session = repo.readonly_session(branch="main")

group = zarr.open_group(session.store, zarr_format=3, mode="r")
a = group['1x721x1440/2m_temperature']
a.info_complete()

I never had the patience to let the last line finish after waiting minutes.

From @paraseba on slack

basically we implemented getsize that takes a key, so python is still listing all keys and calling this ... which is slow because python <-> rust...
it's the switching between python and rust back and forth for every key that is slow

To fully optimize this, I think we need to implement the store.getsize_prefix method in Rust world.