arrays Dynamically generated from other arrays in group e.g. for Remote sensing indices.

[benritchie]

Hey All

I wondered if there was any plans for or interest in dynamically generated arrays in a column group.

I think this could have significant use cases in geo/remote sensing data at least, where its common to want to have easy access to indicies derived from other columns. An example would be ndvi as a dynamic column, defined as (near-IR - red)/ (near-IR + red). (where Near-IR and Red are other arrays in the group).
Although it would be possible to dynamically calculate this on the client side, I think here's lots of cases where its desirable to have this data as a separate column in the zarr array. - E.g. either because the ZARR's are tied to a tiling engine for visualisation, or for ease of providing ML inputs. E.g. currently we are storing about 0.3 PetaBytes of Zarr arrays, and about 20% of these is bands (arrays) such as these.

I had a look to see if there was a way to implement this in the current extension points, but I couldn't see one. I suspect the group sharding extension might bring this closer to feasability though, e.g. something along the lines of the following:

• dynamic arrays would be read only. - and would likely need to mandate that the source arrays were also read only.
• we then either need a codec, or a storage transform that can access from multiple arrays, not just 1. Maybe the mooted groups support in storage transformers could provide that?
• then the developer could write a transformer (or codec) that generated the needed dynamic array from the source arrays.
• the store method could either be special cased to understand this array should not be stored, or could store an array of None values (which would compress to nothing).

Anyway, mostly just wondering if anything like this is a possibility. I'd be happy to help work on it if it was.