feat: N-dimensional raster type extension (Phase 1)

[james-willis]

Summary

Upgrades SedonaDB's raster type from 2D tiles to N-dimensional chunks with named dimensions, enabling support for climate model outputs, hyperspectral imagery, Zarr/NetCDF datacubes, and other multi-dimensional geospatial datasets.

This is a schema-and-API migration. All 33 existing RS_* functions are migrated mechanically and produce identical outputs on 2D inputs; no new SQL functions are added in this PR. GDAL-backed functions added in #787 are ported to read from the new schema and gate on BandRef::is_2d() — N-D bands return a clear Execution error pending MDArray/Zarr support.

This PR is rebased onto post-#787 main and is the destructive replacement of #787's band schema. Because the schema swap and the GDAL loader port have to land together to keep main compiling, the PR is structured as four review-friendly commits but only the tip is required to be CI-green — intermediate commits intentionally don't build and the PR is not bisectable across commits 1–3.

This PR depends on #812 (GDAL outdb-URI parser); merge order: #812 → #749.

Note on utils.rs — the GDAL indb-loader helpers added in #811 (append_as_indb_raster, dataset_to_indb_raster) are deleted in this PR because they target the legacy BandMetadata / StorageType / band_data_writer API that the N-D port removes. They are reintroduced against the canonical N-D schema in the follow-up PR #818, which is already drafted and stacked on top of this branch.

Commits

1. Schema swap — replace feat(sedona-raster-gdal) add GDAL foundation library for supporting GDAL-based RS functions #787's band schema with the canonical N-D schema. Removes nodata_value, storage_type, outdb_url, outdb_band_id; adds dim_names, source_shape, nullable view, outdb_uri, outdb_format. Top-level gains spatial_dims / spatial_shape.
2. Trait surface + is_2d — RasterRef and BandRef accessors for the N-D schema; BandRef::is_2d() default method returns true iff dim_names == ["y","x"] with the identity view (used by GDAL-backed paths to refuse N-D input cleanly).
3. Reader / builder / RS_* migration — identity-view reader and builder, identity-view default via null view row, all 33 RS_* functions ported. RS_BandPath keeps its existing fragment-stripping (format-agnostic display) and is unchanged.
4. Port GDAL loader to canonical schema — rewrite sedona-raster-gdal to read from outdb_uri + the feat(raster-gdal): add parse_outdb_source helper for the GDAL format driver #812 parser instead of storage_type / outdb_url / outdb_band_id. VSI normalization, dataset cache, and RasterIO bodies are byte-for-byte unchanged — only the schema-read sites move. Each GDAL-backed SQL function gates on band.is_2d() at entry.

Equivalence mapping (vs #787)

#787 field	New schema
storage_type (InDb / OutDbRef enum)	data.is_empty() — InDb bands carry their bytes in data; OutDb bands have an empty data buffer and resolve through outdb_uri + outdb_format.
outdb_url	outdb_uri (no rename, same string)
outdb_band_id	encoded inside outdb_uri (<uri>#band=N or GDAL native subdataset URI). Parsed only inside the GDAL format driver via #812's parse_outdb_source. Format-agnostic surfaces (incl. RS_BandPath) treat outdb_uri as opaque.
nodata_value	nodata: Binary (typed bytes; a null row means "no nodata")

Top-level adds spatial_dims: List<Utf8> and spatial_shape: List<UInt64> (e.g. ["y","x"] and [height, width]).

Schema (canonical)

raster: Struct<
  crs:           Utf8View (nullable),
  transform:     List<Float64>,            // 6-element GDAL GeoTransform
  spatial_dims:  List<Utf8View>,           // ["x","y"] today
  spatial_shape: List<Int64>,              // [width, height] today
  bands: List<Struct<
    name:         Utf8 (nullable),
    dim_names:    List<Utf8>,              // 2D bands: ["y","x"]
    source_shape: List<UInt64>,            // C-order extent of source
    data_type:    UInt32 (BandDataType discriminant),
    nodata:       Binary (nullable),
    view:         List<Struct<source_axis,start,step,steps: Int64>>
                  (nullable; null = canonical identity view),
    outdb_uri:    Utf8 (nullable),         // null = InDb; set = OutDbRef
    outdb_format: Utf8View (nullable),     // GDAL driver hint
    data:         BinaryView (non-nullable; empty for outdb-only)
  >>
>

The view field is defined in the schema so it is part of the canonical N-D layout and round-trips through Arrow IPC, but in this PR every writer emits a null view row (canonical identity) and the reader treats a non-null view row as a hard read error. Encoding identity as a null row keeps the common "no slice applied" case free in Arrow space; consumers see the same shape and bytes regardless of which path produced the band. The composition path that decodes non-null view rows lands in #813.

Traits

• RasterRef — spatial_dims, spatial_shape, transform, crs, num_bands, band(i), width/height/x_dim/y_dim (derived).
• BandRef — name, dim_names, source_shape, shape (visible, derived), view, data_type, nodata, outdb_uri, outdb_format, nd_buffer, contiguous_data returning Cow<[u8]>, is_2d (default).
• NdBuffer — zero-copy buffer view used at the numpy / Arrow C Data Interface boundary.

Builder

• start_raster / start_band for full N-D; start_raster_2d / start_band_2d for legacy 2D.
• start_band writes a null view (canonical identity) — no caller change.
• finish_band validates each band's visible shape against the raster's spatial_shape along the spatial dims.

Backward compatibility

Legacy 2D rasters are represented as bands with dim_names = ["y","x"], source_shape = [height, width], and an identity view. All existing RS_* outputs are byte-identical on 2D inputs; every test that passed against #787's GDAL functions passes against the ported loader.

Crates modified

• sedona-schema — N-D raster schema definition.
• sedona-raster — traits, Arrow array reader, builder, affine transform, display.
• sedona-testing — raster test helpers.
• sedona-raster-functions — all RS_* function implementations migrated; RS_BandPath unchanged.
• sedona-raster-gdal — loader reads outdb_uri + feat(raster-gdal): add parse_outdb_source helper for the GDAL format driver #812 parser; SQL functions gate on is_2d().

Out of scope (follow-ups)

• GDAL indb-loader port (feat(raster-gdal): port indb-raster loader utilities to canonical N-D schema #818) — reintroduces append_as_indb_raster / dataset_to_indb_raster (deleted here, see note above) against the N-D schema.
• View machinery — validate_view, start_band_with_view, view-decoding in the reader, view-aware contiguous_data, and the explicit-view test cluster — landing in [WIP] feat(raster): view machinery for non-identity band views #813 between this PR and feat: add N-D raster dimension query and manipulation functions #750.
• Functions that produce non-identity views (slice/crop, broadcast, axis reorder, heterogeneous-source stack) — coming in feat: add N-D raster dimension query and manipulation functions #750 on top of [WIP] feat(raster): view machinery for non-identity band views #813.
• N-D-aware GDAL via MDArray (lifts the is_2d() guard for selected functions).
• Zarr / non-GDAL N-D backends.
• View-aware fast paths in arithmetic functions.

Test plan

• sedona-schema: tests pass; verifies the view field is nullable and the view-struct field order matches the indices module.
• sedona-raster: tests pass — identity round-trip via start_band (null view row); on-disk schema invariant that identity bands serialise as null view rows; reader rejects non-null view rows with a clear error; BandRef::is_2d truth-table tests; builder visible-shape validation against spatial_shape.
• sedona-raster-functions: tests pass — all existing function tests produce identical outputs after migration.
• sedona-raster-gdal: 46/46 pass — every feat(sedona-raster-gdal) add GDAL foundation library for supporting GDAL-based RS functions #787 SQL-function test passes against the ported loader; new tests cover N-D rejection at raster_ref_to_gdal_mem and the VRT path; one end-to-end test confirms path#band=2 selects band 2 of a two-band GeoTIFF.
• cargo clippy --all-targets -- -D warnings clean on the affected crates.
• cargo fmt --all --check clean.

[paleolimbot]

I'll take a closer look at the implementation tomorrow, but wanted to get a preliminary review of the schema out!

[paleolimbot]

I did another round...there are a few places where tests or comments were removed that were still useful.

I do think it's worth iterating a tiny bit on the schema, and importantly, I think we should wait until @Kontinuation has finished the GDAL read portion of the previous set of raster refactoring because (1) this was a lot of work and Kristin's time is valuable and (2) these are important use cases to consider when making the new type work!

[paleolimbot]

Thank you for working on this! This is a solid schema design that will be the foundation of a solid nd spatial raster framework that will be flexible enough to deliver the type of performance we are hoping for in the native translation of Sedona raster functions to Rust.

Most of my comments are about minimizing the diff to avoid disrupting existing work. With a few small changes I think you avoid almost all disruption of existing + in progress work and minimize the footprint of this PR considerably. Basically, just keep RasterMetadata and BandMetadata and bands() around, even if they are just shims returning concrete structs you'll remove later. For what it's worth I frequently do this with PRs into larger projects using VSCode's pull request extension which nicely lets you roll through your PR diff to minimize it.

[james-willis]

Design note on raster_ref_to_gdal_mem lifetime plumbing

The GDAL backend in this PR uses the shim's BandRef::data() -> &[u8] rather than BandRef::contiguous_data() -> Result<Cow<'_, [u8]>>. That's deliberate, and it lets us drop the (Dataset, Vec<Vec<u8>>) tuple return / _owned_band_bytes: Vec<Vec<u8>> field on RasterDataset.

Why it works in this PR: every band is identity-viewed, so the bytes GDAL points at are a borrow into the StructArray's backing buffer. RasterDataset already keeps the source raster alive via _source_raster: PhantomData<&'a dyn RasterRef>, so the pointer is valid for the dataset's lifetime — no extra owned-bytes plumbing needed. Non-identity views error at read time here, so Cow::Owned cannot appear.

What changes when the view machinery lands: once non-identity views are supported, contiguous_data() may return Cow::Owned for sliced/permuted bands (the reader materializes a new buffer). That Vec<u8> will need to outlive the GDAL dataset that points into it. Bringing back the _owned_band_bytes: Vec<Vec<u8>> field on RasterDataset (and the matching tuple return from raster_ref_to_gdal_mem) is the right way to do that — earlier revisions of this PR carried that plumbing for exactly this reason.

Short version: if you're picking this code up later to add view support, expect to re-thread owned band bytes from the reader, through raster_ref_to_gdal_mem, and onto RasterDataset.

[paleolimbot]

A few optional nits probably worth considering now but in general this look great! Thank you!

[paleolimbot]

Thank you!