Resilient zarr codecs

[emfdavid]

When reading very large zarr aggregations (I work with petabyte NODD HRRR aggregations comprised of ~50k blocks) it would be really helpful to allow the codec to NAN fill missing or bad values. For instance, I have found some of the grib files are truncted resulting in a ECCODES ERROR : grib_handle_new_from_message: No final 7777 in message!

I found a solution by hacking zarr core chunk_getitem but it feels like this belongs in the codec? I couldn't implement it there though because the codec doesn't have access to the context (shape, type & fillval).

Any standard solution for Zarr3 would be most welcome - making the behavior configurable would be great (fail fast or fault tolerant)!

[rabernat]

What sort of API would you like to see here @emfdavid? Most users probably want an error to be raised if the underlying data are corrupted and undecodable. How would you like to "tell" Zarr to treat corrupted chunks as missing data?

[emfdavid]

I think the suggestion on the pangeo call last week was to use the new zarr3 python config object which might be globally accessible down in the error handler at the right point in the stack?

[rabernat]

How would you want to specify which errors to translate to missing data vs. raise? I imagine, for example, you'd still want to see an error if the chunk were not available due to HTTP connection errors, permission errors, etc. What about different codecs? Do you want to see errors from Blosc but not from eccodes?

[emfdavid]

That would be nice to have, but don't let perfect get in the way of functional.
Starting with two modes: Fail fast, or log and nan fill, would be a good first step.
Configurable exception handling would be nice to have, but I agree it gets complicated.