Conversation
CoordinateIndexer (general path and sorted-1D fast path) and IntArrayDimIndexer built a dense per-chunk histogram via np.bincount(..., minlength=nchunks) / np.zeros(nchunks) plus a full-length cumsum. For arrays with very many chunks this allocates memory proportional to the total chunk count regardless of how few points are selected — a 3-point write to an array with 1.4e11 chunks tried to allocate 1 TiB and raised MemoryError. Replace the dense histogram with run boundaries computed directly on the sorted raveled chunk ids (sorted_run_ends), storing a compressed cumsum aligned with the occupied chunks, and index it positionally in __iter__. Memory and time now scale with the number of selected points. Fixes zarr-developers#4174 Assisted-by: ClaudeCode:claude-fable-5
…rties The compressed per-occupied-chunk cumsum introduced for zarr-developersgh-4174 changed the observable semantics of chunk_nitems_cumsum (and removed chunk_nitems on IntArrayDimIndexer). Although zarr.core is documented as private API, external code is known to introspect these indexers, so be conservative: store the compressed offsets under a new name (chunk_run_ends, aligned with chunk_rixs / dim_chunk_ixs) and restore chunk_nitems / chunk_nitems_cumsum as properties that lazily rebuild the original dense arrays, warning with ZarrDeprecationWarning. The O(nchunks) cost is now only paid if someone actually accesses them — which was the status quo before the fix. Assisted-by: ClaudeCode:claude-fable-5
Merge current main, preserve sparse occupied-chunk storage, and verify arbitrary large-coordinate projection roundtrips. Keep uniform chunk counts in integer arithmetic to avoid rejecting valid near-end coordinates. Assisted-by: Codex:GPT-6
Documentation build overview
24 files changed ·
|
…tions Review follow-ups for the sparse-selection change: - Fix ceildiv itself rather than bypassing it at one call site. It went through float division, so ceildiv(2**62 - 1, 1) returned 2**62; four other callers (slice arithmetic, the regular-grid check, dask-style chunk sizes) had the same latent error. Integers now divide exactly; floats keep the ceil-of-quotient path. FixedDimension goes back to calling it. - BoolArrayDimIndexer still allocated a dense per-chunk count array and ran a Python loop over every chunk, so an orthogonal boolean selection on a finely chunked axis paid O(nchunks) in time and memory on top of the mask. It now derives the occupied chunks and run ends from the selected positions with sorted_run_ends, like the other two indexers, and exposes the same deprecated dense properties. A boolean axis with 2**22 chunks goes from a multi-second loop to ~2 ms. - Changelog: state precisely which selection kinds are covered, that reading the deprecated properties rebuilds the dense array, and the ceildiv correction. Assisted-by: ClaudeCode:claude-fable-5-1
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4356 +/- ##
==========================================
+ Coverage 94.34% 94.35% +0.01%
==========================================
Files 92 92
Lines 12935 12966 +31
==========================================
+ Hits 12203 12234 +31
Misses 732 732
🚀 New features to boost your workflow:
|
Retain sparse coordinate and integer indexing plus exact ceildiv, while restoring the existing boolean-axis implementation to avoid dense-mask regressions. Assisted-by: Codex:GPT-6
Merging this PR will not alter performance
Comparing 
Footnotes
|
Assisted-by: Codex:GPT-6
Preserve dense attribute identity, mutation behavior, and dataclass fields without deprecation warnings while keeping normal indexing sparse. Assisted-by: Codex:GPT-6
Restore the original ceildiv behavior and exports, and use ceildiv_int for integer-only chunk and slice calculations without type-based dispatch. Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
Contributor
Author
|
@sehoffmann have a look |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
this is an AI-authored PR that makes coordinate indexers much more memory efficient, resolving #4174
🤖 AI text below 🤖
Fixes #4174. Coordinate reads and writes and orthogonal integer-array selections now store occupied chunk IDs and run-end offsets instead of allocating count arrays for the entire chunk grid. Selecting a few points from an array with billions of chunks therefore avoids the previous full-grid allocation. This covers both the general coordinate path and the sorted 1D fast path; mask selections routed through
CoordinateIndexeralso benefit.Run-boundary calculation is linear in the number of selected points. Unsorted selections can still require sorting, and dense selections can use additional temporary memory. The legacy dense attributes on
CoordinateIndexerandIntArrayDimIndexerare materialized on first access and cached without deprecation warnings. Repeated reads return the same array, in-place mutations persist, and iteration honors edits to materialized cumulative offsets. The legacy dataclass fields remain available;dataclasses.asdict()materializes them, whilerepromits the dense arrays to keep inspection sparse. Accessing these attributes still requires full-grid memory.The new
ceildiv_inthelper uses exact Python integer arithmetic without type-based routing. All eight production call sites for chunk counts and slice calculations use it. The existingzarr.core.common.ceildivhelper retains its original floating-point behavior.The orthogonal boolean-axis optimization is deferred: expanding dense masks into integer positions caused a memory and runtime regression.
BoolArrayDimIndexerretains its existing implementation and attributes.Validation: 1,195 tests passed across
test_common.py,test_indexing.py,test_unified_chunk_grid.py, andtest_codecs/test_sharding.pyin the Hatch Python 3.12 minimal environment, with 2 skips and 2 expected failures. Coverage includes huge sparse grids, sorted and unsorted selections, duplicates, projection reconstruction, lazy attribute caching, mutation behavior, dataclass introspection, and exact large-integer division.