slow array/metadata/sub_group creation with GCS #2545
Unanswered
AndrewAnnex
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am using zarr 2.18.3 on google cloud to create empty zarr groups with seeded metadata and arrays in the following manner:
Unfortunately the performance isn't very good, at about 5 seconds per iteration.
Without simply writing to the local disk and manually syncing to google cloud storage, is there a more performant method to be attempted here? I think I've seen references to async methods but I haven't seen a worked out example in python in the docs, or I could use the parallel methodology but I am not sure if it's possible to pickle the top level group object, but maybe I could get that top level group made and then re-open the group with zarr.open?
tips and ideas would be appreciated
Beta Was this translation helpful? Give feedback.
All reactions