feat(raster): transparent OutDb byte loading via a GDAL loader hook

[james-willis]

Summary

Make BandRef::nd_buffer() and BandRef::contiguous_data() Just Work for OutDb bands (bands whose Arrow data column is empty and whose outdb_uri points elsewhere). Today those methods return NotYetImplemented, blocking every UDF and downstream consumer from reading OutDb pixel bytes — the entire reason OutDb references exist as a schema feature is moot without a working byte path.

Approach: a single statically-typed function-pointer hook in sedona-raster, populated by sedona-raster-gdal at session bootstrap. No trait, no HashMap-keyed registry, no async — "compiled in, not pluggable". The byte-access surface is caller-managed scratch: methods take a &mut Vec<u8> rather than allocating internally, so consumers in scalar-function loops can reuse a single buffer across rasters and (eventually) reserve memory through DataFusion's memory pool. The only abstraction introduced is the one the crate-boundary forces (sedona-raster-gdal depends on sedona-raster, not the reverse, so a direct call from BandRefImpl to GDAL is impossible).

Byte-access surface

trait BandRef {
    fn nd_buffer<'s>(&'s self, scratch: &'s mut Vec<u8>)
        -> Result<NdBuffer<'s>, ArrowError>;

    fn contiguous_data<'s>(&'s self, scratch: &'s mut Vec<u8>)
        -> Result<&'s [u8], ArrowError>;

    // ... metadata methods unchanged ...
}

InDb identity-view bands return a zero-copy slice of the Arrow column and leave scratch untouched. OutDb bands invoke the process-wide loader, which writes into scratch; the returned slice borrows from scratch. Callers don't branch on InDb-vs-OutDb — they pass the same Vec<u8> for every band, capacity grows to the largest band's source size and then stabilises.

The pre-N-D infallible accessor BandRef::data() -> &[u8] is removed: every existing call site was inside #[cfg(test)] or in sedona-testing's test helpers (no production callers), and its silent-&[]-on-error behaviour was the kind of trap the new scratch contract eliminates.

OutDb loader hook

pub type OutDbBandLoader =
    fn(&OutDbLoadRequest<'_>, &mut Vec<u8>) -> Result<(), ArrowError>;

The loader appends exactly Π source_shape × data_type.byte_size() bytes to the caller-provided scratch. The internal load_outdb dispatcher clears the scratch on entry and validates the loader's written length on return, so backend implementations only need to write — they don't have to inspect pre-call state or zero the buffer.

sedona-raster-gdal::register_outdb_loader() is called from SedonaContext::new_from_context() next to the existing function-set registration. Registration is idempotent (OnceLock::set is first-write-wins) so repeated SedonaContext::new() calls in one process are safe.

Changes

• New rust/sedona-raster/src/outdb_loader.rs — OutDbLoadRequest, OutDbBandLoader fn-pointer type, OnceLock, set_outdb_band_loader, internal load_outdb dispatcher.
• Modified rust/sedona-raster/src/array.rs — BandRefImpl byte-access methods rewritten to take scratch; new source_bytes(&mut scratch) helper unifies the InDb (zero-copy column) and OutDb (loader writes into scratch) paths. Overflow-safe expected-size check (checked_mul + try_into::<usize>). No per-band cache field — the lifetime anchor is the caller's scratch.
• Modified rust/sedona-raster/src/traits.rs — BandRef::is_indb() is now a required method (no default impl, since the previous default derived from nd_buffer() emptiness which misclassifies OutDb bands as InDb once a loader is installed). New scratch-aware nd_buffer / contiguous_data signatures; data() method removed.
• Modified rust/sedona-raster/src/lib.rs — re-exports set_outdb_band_loader, OutDbBandLoader, OutDbLoadRequest.
• New rust/sedona-raster-gdal/src/outdb_loader.rs — gdal_load fn + register_outdb_loader. 2-D [y,x] enforcement, pre-read pixel-type check (errors cleanly if the file's GDAL pixel type differs from the band metadata's claim), uses GDALDatasetCache::get_or_create_outdb_source. Writes the read bytes into the caller's scratch.
• Modified rust/sedona-raster-gdal/src/gdal_common.rs — raster_ref_to_gdal_mem migrated to contiguous_data(&mut scratch). The function still rejects OutDb above the loop, so the scratch path is never exercised — but a defensive if !scratch.is_empty() assert after each iteration catches any future regression loudly: if a relaxed OutDb check ever lets the loader write into scratch, the captured data_ptr would alias scratch and could dangle next iteration. The assert fires before that's possible.
• Modified rust/sedona/src/context.rs — calls sedona_raster_gdal::register_outdb_loader() from SedonaContext::new_from_context().
• Migrated ~40 test sites in sedona-raster, sedona-raster-gdal, sedona-raster-functions, and sedona-testing from band.data() / band.contiguous_data().unwrap() to the scratch-aware shape.

Open question — hard GDAL dependency on sedona

Heads-up for reviewers: this PR adds sedona-raster-gdal as a direct dependency of sedona (rust/sedona/Cargo.toml:84), so SedonaContext::new() can register the GDAL OutDb loader at bootstrap. Before this PR, the sedona crate did not link sedona-raster-gdal at all — GDAL was strictly opt-in. After this PR, building sedona (and anything that depends on it, e.g. the Python wheel and R bindings) requires libgdal-dev at link time, unconditionally.

The cleanest fix is to feature-gate the registration:

sedona-raster-gdal = { workspace = true, optional = true }

[features]
default = ["gdal"]
gdal = ["dep:sedona-raster-gdal"]

#[cfg(feature = "gdal")]
sedona_raster_gdal::register_outdb_loader();

That keeps the default behaviour (GDAL linked, OutDb works out of the box for SQL users) while letting downstream consumers --no-default-features build a slim sedona without GDAL — same flexibility they had before this PR.

Calling this out for explicit reviewer judgment rather than fixing it inline, because the answer depends on what level of optionality the project wants to preserve. Happy to apply the feature gate (or any variant) in a follow-up commit on this branch.

Tests

sedona-raster (mock loader, 9 unit tests): loader registration, round-trip via nd_buffer() / contiguous_data(), scratch reuse across bands, missing-uri error, undersized-loader-output error, loader-failure propagation, is_indb invariant after loader runs.

sedona-raster (process-isolated integration): tests/no_loader_registered.rs — separate test binary so OUTDB_BAND_LOADER stays unset; asserts the "no OutDb loader registered" diagnostic message verbatim. Catches regressions in SedonaContext::new_from_context()'s registration call without needing GDAL or a SQL engine to reproduce. Reviewers: see the "Hard GDAL dependency" note above — if we feature-gate, this test stays useful for the slim-build path.

sedona-raster-gdal (real GDAL, 7 integration tests): write real GeoTIFFs to temp dirs and verify end-to-end byte path, scratch reuse across calls, multi-band #band=N selection, non-[y,x] dim-name rejection, dtype mismatch rejection, missing-file error, band-index-out-of-range error.

End-to-end SQL coverage

A SQL-layer test along the lines of SELECT RS_Value(raster, x, y) FROM outdb_geotiff_table would also exercise this path, but no byte-reading RS_* function exists yet — the current RS_* surface (RS_BandPath, RS_NumBands, RS_BandPixelType, RS_Envelope, …) is all metadata, none of them call band.nd_buffer(). SQL-layer coverage is deferred until a byte-reading kernel lands. The 7 real-GDAL integration tests are the right test layer until then.

Relationship to PR #813 (view machinery)

This PR is independent of PR-D (#813 — view machinery / materialized cell / strided walk) and based directly on main. The view-composition path remains rejected at construction in RasterRefImpl::band(), so the OutDb byte path here only needs to handle the identity-view case.

The scratch-buffer convention here aligns with the review feedback on #813 about avoiding hidden allocations on band structures — the OutDb byte path no longer hides allocation in an OnceCell<Vec<u8>> on the band, it threads the caller's scratch through the byte-access methods.

Test plan

• cargo test -p sedona-raster --lib (72 passing)
• cargo test -p sedona-raster --tests no_loader_registered (process-isolated, 1 passing)
• cargo test -p sedona-raster-gdal --lib outdb_loader (7 passing)
• cargo test -p sedona-raster-functions --lib (143 passing)
• cargo test -p sedona-testing --lib (78 passing)
• cargo clippy --all-targets -p sedona-raster -p sedona-raster-gdal -p sedona-raster-functions -p sedona-testing -- -D warnings
• cargo fmt --all -- --check

[james-willis]

Converting back to draft while the design pivots. Three motivations driving the rewrite:

No magic loading. Today byte fetching happens transparently inside Band::nd_buffer / Band::contiguous_data. That hides I/O from EXPLAIN, hides it from the optimiser, and produces a callstack-rooted exception when a network fetch fails. The replacement is an explicit RS_EnsureLoaded(raster) -> raster UDF that the optimiser auto-injects ahead of RS_* functions that need pixel bytes. Band accessors return whatever's in data verbatim — no hidden side effects.

Async UDF perf gains. DataFusion 52.5's AsyncScalarUDFImpl + AsyncFuncExec give us cooperative scheduling, buffer_unordered intra-batch fan-out, bounded memory via ideal_batch_size, and cancellation for free. A sync transparent loader can't access any of that — every fetch blocks one tokio worker and there's no way to parallelise N row fetches into one batch.

Plugin story for additional formats. The single global function-pointer registry becomes a format-keyed trait-object registry living in sedona-raster. Format crates (this PR's GDAL, the in-flight Zarr work, future COG / Icechunk / …) register themselves under outdb_format keys; the sedona crate consumes the registry from RS_EnsureLoaded without depending on any specific format crate. Adding a backend becomes a new entry in the registry, not a change to consumer code.

Survives from this PR as-is: the GDAL backend impl and the session-bootstrap registration scaffolding. Goes away: the transparent dispatch hook on the Band accessor; the single-loader API. Will reopen as the rewrite once the new infra lands.

[james-willis]

Superseded by #886.

The design pivoted from a transparent-sync hook on BandRef accessors with a single fn-pointer registry to:

• An async AsyncByteLoader trait + format-keyed OutDbLoaderRegistry in sedona-raster
• An explicit RS_EnsureLoaded(raster) -> raster async UDF that materialises OutDb bytes on demand
• A needs_bytes annotation on SedonaScalarUDF + an analyzer rule that wraps raster args of needs_bytes UDFs with RS_EnsureLoaded, with DataFusion's CommonSubexprEliminate doing dedup
• BandRef::nd_buffer() / contiguous_data() / data() unchanged from main — accessors return whatever is in the data column verbatim, no hidden I/O

The GDAL byte-read core from this PR survives as sedona-raster-gdal::GdalLoader (Commit 5 of #886). The Zarr backend lands alongside (Commit 6) so both delivery modes — compiled-in and out-of-tree plugin — are exercised in the same PR.

Closing this PR rather than rebasing because the trait surface and registry shape differ enough that a fresh history reviews better than a morph.