feat: unified chunk grid with rectilinear chunk/shard support

docs/user-guide/arrays.md

@@ -599,6 +599,171 @@ In this example a shard shape of (1000, 1000) and a chunk shape of (100, 100) is
 This means that `10*10` chunks are stored in each shard, and there are `10*10` shards in total.
 Without the `shards` argument, there would be 10,000 chunks stored as individual files.
 
+## Rectilinear (variable) chunk grids
+
+!!! warning "Experimental"
+    Rectilinear chunk grids are an experimental feature and may change in
+    future releases. This feature is expected to stabilize in Zarr version 3.3.
+
+    Because the feature is still stabilizing, it is disabled by default and
+    must be explicitly enabled:
+
+    ```python
+    import zarr
+    zarr.config.set({"array.rectilinear_chunks": True})
+    ```
+
+    Or via the environment variable `ZARR_ARRAY__RECTILINEAR_CHUNKS=True`.
+
+    The examples below assume this config has been set.
+
+By default, Zarr arrays use a regular chunk grid where every chunk along a
+given dimension has the same size (except possibly the final boundary chunk).
+Rectilinear chunk grids allow each chunk along a dimension to have a different
+size. This is useful when the natural partitioning of the data is not uniform —
+for example, satellite swaths of varying width, time series with irregular
+intervals, or spatial tiles of different extents.
+
+### Creating arrays with rectilinear chunks
+
+To create an array with rectilinear chunks, pass a nested list to the `chunks`
+parameter where each inner list gives the chunk sizes along one dimension:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+zarr.config.set({"array.rectilinear_chunks": True})
+z = zarr.create_array(
+    store=zarr.storage.MemoryStore(),
+    shape=(60, 100),
+    chunks=[[10, 20, 30], [50, 50]],
+    dtype='int32',
+)
+print(z.info)
+```
+
+In this example the first dimension is split into three chunks of sizes 10, 20,
+and 30, while the second dimension is split into two equal chunks of size 50.
+
+### Reading and writing data
+
+Rectilinear arrays support the same indexing interface as regular arrays.
+Reads and writes that cross chunk boundaries of different sizes are handled
+automatically:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+import numpy as np
+data = np.arange(60 * 100, dtype='int32').reshape(60, 100)
+z[:] = data
+# Read a slice that spans the first two chunks (sizes 10 and 20) along axis 0
+print(z[5:25, 0:5])
+```
+
+### Inspecting chunk sizes
+
+The `.write_chunk_sizes` property returns the actual data size of each storage
+chunk along every dimension. It works for both regular and rectilinear arrays
+and returns a tuple of tuples (matching the dask `Array.chunks` convention).
+When sharding is used, `.read_chunk_sizes` returns the inner chunk sizes instead:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+print(z.write_chunk_sizes)
+```
+
+For regular arrays, this includes the boundary chunk:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+z_regular = zarr.create_array(
+    store=zarr.storage.MemoryStore(),
+    shape=(100, 80),
+    chunks=(30, 40),
+    dtype='int32',
+)
+print(z_regular.write_chunk_sizes)
+```
+
+Note that the `.chunks` property is only available for regular chunk grids. For
+rectilinear arrays, use `.write_chunk_sizes` (or `.read_chunk_sizes`) instead.
+
+### Resizing and appending
+
+Rectilinear arrays can be resized. When growing past the current edge sum, a
+new chunk is appended covering the additional extent. When shrinking, the chunk
+edges are preserved and the extent is re-bound (chunks beyond the new extent
+simply become inactive):
+
+```python exec="true" session="arrays" source="above" result="ansi"
+z = zarr.create_array(
+    store=zarr.storage.MemoryStore(),
+    shape=(30,),
+    chunks=[[10, 20]],
+    dtype='float64',
+)
+z[:] = np.arange(30, dtype='float64')
+print(f"Before resize: chunk_sizes={z.write_chunk_sizes}")
+z.resize((50,))
+print(f"After resize:  chunk_sizes={z.write_chunk_sizes}")
+```
+
+The `append` method also works with rectilinear arrays:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+z.append(np.arange(10, dtype='float64'))
+print(f"After append:  shape={z.shape}, chunk_sizes={z.write_chunk_sizes}")
+```
+
+### Compressors and filters
+
+Rectilinear arrays work with all codecs — compressors, filters, and checksums.
+Since each chunk may have a different size, the codec pipeline processes each
+chunk independently:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+z = zarr.create_array(
+    store=zarr.storage.MemoryStore(),
+    shape=(60, 100),
+    chunks=[[10, 20, 30], [50, 50]],
+    dtype='float64',
+    filters=[zarr.codecs.TransposeCodec(order=(1, 0))],
+    compressors=[zarr.codecs.BloscCodec(cname='zstd', clevel=3)],
+)
+z[:] = np.arange(60 * 100, dtype='float64').reshape(60, 100)
+np.testing.assert_array_equal(z[:], np.arange(60 * 100, dtype='float64').reshape(60, 100))
+print("Roundtrip OK")
+```
+
+### Rectilinear shard boundaries
+
+Rectilinear chunk grids can also be used for shard boundaries when combined
+with sharding. In this case, the outer grid (shards) is rectilinear while the
+inner chunks remain regular. Each shard dimension must be divisible by the
+corresponding inner chunk size:
+
+```python exec="true" session="arrays" source="above" result="ansi"
+z = zarr.create_array(
+    store=zarr.storage.MemoryStore(),
+    shape=(120, 100),
+    chunks=(10, 10),
+    shards=[[60, 40, 20], [50, 50]],
+    dtype='int32',
+)
+z[:] = np.arange(120 * 100, dtype='int32').reshape(120, 100)
+print(z[50:70, 40:60])
+```
+
+Note that rectilinear inner chunks with sharding are not supported — only the
+shard boundaries can be rectilinear.
+
+### Metadata format
+
+Rectilinear chunk grid metadata uses run-length encoding (RLE) for compact
+serialization. When reading metadata, both bare integers and `[value, count]`
+pairs are accepted:
+
+- `[10, 20, 30]` — three chunks with explicit sizes
+- `[[10, 3]]` — three chunks of size 10 (RLE shorthand)
+- `[[10, 3], 5]` — three chunks of size 10, then one chunk of size 5
+
+When writing, Zarr automatically compresses repeated values into RLE format.
+
 ## Missing features in 3.0
 
 The following features have not been ported to 3.0 yet.

docs/user-guide/config.md

@@ -30,6 +30,7 @@ Configuration options include the following:
 - Default Zarr format `default_zarr_version`
 - Default array order in memory `array.order`
 - Whether empty chunks are written to storage `array.write_empty_chunks`
+- Enable experimental rectilinear chunk grids `array.rectilinear_chunks`
 - Async and threading options, e.g. `async.concurrency` and `threading.max_workers`
 - Selections of implementations of codecs, codec pipelines and buffers
 - Enabling GPU support with `zarr.config.enable_gpu()`. See GPU support for more.

mkdocs.yml

@@ -76,6 +76,8 @@ nav:
         - Creation sub-module: api/zarr/deprecated/creation.md
   - release-notes.md
   - contributing.md
+  - Design documents:
+    - design/chunk-grid.md
 watch:
   - src/zarr
   - docs

src/zarr/abc/codec.py

@@ -17,10 +17,10 @@
 
     from zarr.abc.store import ByteGetter, ByteSetter, Store
     from zarr.core.array_spec import ArraySpec
-    from zarr.core.chunk_grids import ChunkGrid
     from zarr.core.dtype.wrapper import TBaseDType, TBaseScalar, ZDType
     from zarr.core.indexing import SelectorTuple
     from zarr.core.metadata import ArrayMetadata
+    from zarr.core.metadata.v3 import ChunkGridMetadata
 
 __all__ = [
     "ArrayArrayCodec",
@@ -140,7 +140,7 @@ def validate(
         *,
         shape: tuple[int, ...],
         dtype: ZDType[TBaseDType, TBaseScalar],
-        chunk_grid: ChunkGrid,
+        chunk_grid: ChunkGridMetadata,
     ) -> None:
         """Validates that the codec configuration is compatible with the array metadata.
         Raises errors when the codec configuration is not compatible.
@@ -151,8 +151,8 @@ def validate(
             The array shape
         dtype : np.dtype[Any]
             The array data type
-        chunk_grid : ChunkGrid
-            The array chunk grid
+        chunk_grid : ChunkGridMetadata
+            The array chunk grid metadata
         """
 
     async def _decode_single(self, chunk_data: CodecOutput, chunk_spec: ArraySpec) -> CodecInput:
@@ -357,7 +357,7 @@ def validate(
         *,
         shape: tuple[int, ...],
         dtype: ZDType[TBaseDType, TBaseScalar],
-        chunk_grid: ChunkGrid,
+        chunk_grid: ChunkGridMetadata,
     ) -> None:
         """Validates that all codec configurations are compatible with the array metadata.
         Raises errors when a codec configuration is not compatible.
@@ -368,8 +368,8 @@ def validate(
             The array shape
         dtype : np.dtype[Any]
             The array data type
-        chunk_grid : ChunkGrid
-            The array chunk grid
+        chunk_grid : ChunkGridMetadata
+            The array chunk grid metadata
         """
         ...
 

src/zarr/api/synchronous.py

@@ -13,7 +13,7 @@
 from zarr.errors import ZarrDeprecationWarning
 
 if TYPE_CHECKING:
-    from collections.abc import Iterable
+    from collections.abc import Iterable, Sequence
 
     import numpy as np
     import numpy.typing as npt
@@ -822,7 +822,7 @@ def create_array(
     shape: ShapeLike | None = None,
     dtype: ZDTypeLike | None = None,
     data: np.ndarray[Any, np.dtype[Any]] | None = None,
-    chunks: tuple[int, ...] | Literal["auto"] = "auto",
+    chunks: tuple[int, ...] | Sequence[Sequence[int]] | Literal["auto"] = "auto",
     shards: ShardsLike | None = None,
     filters: FiltersLike = "auto",
     compressors: CompressorsLike = "auto",
@@ -858,9 +858,13 @@ def create_array(
     data : np.ndarray, optional
         Array-like data to use for initializing the array. If this parameter is provided, the
         ``shape`` and ``dtype`` parameters must be ``None``.
-    chunks : tuple[int, ...] | Literal["auto"], default="auto"
+    chunks : tuple[int, ...] | Sequence[Sequence[int]] | Literal["auto"], default="auto"
         Chunk shape of the array.
         If chunks is "auto", a chunk shape is guessed based on the shape of the array and the dtype.
+        A nested list of per-dimension edge sizes creates a rectilinear grid.
+        Rectilinear chunk grids are experimental and must be explicitly enabled
+        with ``zarr.config.set({'array.rectilinear_chunks': True})`` while the
+        feature is stabilizing.
     shards : tuple[int, ...], optional
         Shard shape of the array. The default value of ``None`` results in no sharding at all.
     filters : Iterable[Codec] | Literal["auto"], optional
@@ -993,7 +997,7 @@ def from_array(
     data: AnyArray | npt.ArrayLike,
     write_data: bool = True,
     name: str | None = None,
-    chunks: Literal["auto", "keep"] | tuple[int, ...] = "keep",
+    chunks: Literal["auto", "keep"] | tuple[int, ...] | Sequence[Sequence[int]] = "keep",
     shards: ShardsLike | None | Literal["keep"] = "keep",
     filters: FiltersLike | Literal["keep"] = "keep",
     compressors: CompressorsLike | Literal["keep"] = "keep",
@@ -1025,13 +1029,17 @@ def from_array(
     name : str or None, optional
         The name of the array within the store. If ``name`` is ``None``, the array will be located
         at the root of the store.
-    chunks : tuple[int, ...] or "auto" or "keep", optional
+    chunks : tuple[int, ...] or Sequence[Sequence[int]] or "auto" or "keep", optional
         Chunk shape of the array.
         Following values are supported:
 
         - "auto": Automatically determine the chunk shape based on the array's shape and dtype.
-        - "keep": Retain the chunk shape of the data array if it is a zarr Array.
-        - tuple[int, ...]: A tuple of integers representing the chunk shape.
+        - "keep": Retain the chunk grid of the data array if it is a zarr Array.
+        - tuple[int, ...]: A tuple of integers representing the chunk shape (regular grid).
+        - Sequence[Sequence[int]]: Per-dimension chunk edge lists (rectilinear grid).
+          Rectilinear chunk grids are experimental and must be explicitly enabled
+          with ``zarr.config.set({'array.rectilinear_chunks': True})`` while the
+          feature is stabilizing.
 
         If not specified, defaults to "keep" if data is a zarr Array, otherwise "auto".
     shards : tuple[int, ...], optional