Apache Iceberg read/write support

[hrodmn]

I am unsure if this repo is the right place for any of this, but I just want to get the idea out there!

I am interested in building operational pipelines that can stream new STAC records into a stac-geoparquet store. So far I have just built a pipeline that appends records via hive-partitioned parquet files but that solution would not scale well for a continuously updated metadata archive. Apache Iceberg could be a good option because it is simple to append an arrow dataframe to an Iceberg table in a transactional way and queries don't require potentially slow/expensive ListBucket requests.

Here is a rough pseudo-code Python example of how I have written STAC records to Iceberg:

import pyarrow.parquet as pq
from pyiceberg.catalog import load_catalog
from pyiceberg.partitioning import PartitionSpec

catalog = load_catalog()

# get arrow table with stac records
# could probably do this with rustac.arrow, too
df = pq.read_table("/path/to/stac.parquet")

table = catalog.create_table(
    "stac.{collection_id}",
    schema=df.schema,  # need to convert to an actual Iceberg schema but you get the idea
    partition_spec=PartitionSpec(),
    properties={"geo": ...},
)

table.append(df)

Then subsequent tasks can write items with the same schema to the table with more table.append(df) calls in some distributed context without worrying about manual partitioning or file locks.

In any case it could be nice to be able to read from an iceberg table using the duckdb client. The only change I think you need to make (compared to reading a parquet file) is iceberg_scan('/path/to/iceberg.metadata.json') instead of read_parquet('/path/to/stac.parquet').

[jjcfrancisco]

The read side looks very doable but the write may require a lot more work. @gadomski do you have any thoughts? I can tackle the reading of iceberg tables

[gadomski]

Reading would be a great start!

[gadomski]

@jjcfrancisco as a heads up, we're taking this on internally at Development Seed. You're correct that writing is the trickier bit, and we're initially going to build something downstream of rustac (but that uses rustac to handle that). It doesn't make too much sense to build the read story until the write exists, simply because we won't have an example to test against until we write something.

[jjcfrancisco]

@jjcfrancisco as a heads up, we're taking this on internally at Development Seed. You're correct that writing is the trickier bit, and we're initially going to build something downstream of rustac (but that uses rustac to handle that). It doesn't make too much sense to build the read story until the write exists, simply because we won't have an example to test against until we write something.

No worries, I’m almost finished on my side. I ran into a few issues generating test Iceberg data without exposing my own information. Happy to stash the work for now. Let me know when you think it’s a good time to resume development on reading Iceberg tables. I’m also happy to open a PR in the meantime with the reading changes for visibility, though I appreciate it may not be ideal to have code sitting around and going stale.

[gadomski]

Were you creating a native read implementation using the iceberg crate? I was planning just to offload the work to DuckDB via https://duckdb.org/docs/stable/core_extensions/iceberg/overview, but would be interested in looking a iceberg native solution as well.

[jjcfrancisco]

Were you creating a native read implementation using the iceberg crate? I was planning just to offload the work to DuckDB via https://duckdb.org/docs/stable/core_extensions/iceberg/overview, but would be interested in looking a iceberg native solution as well.

I was tempted to go with native implementation but ended up using DuckDB because format_parquet_href fitted well... I changed it to format_source_href which handles both parquet and iceberg. Presumably, you’re thinking of handling the writing natively?

[gadomski]

Presumably, you’re thinking of handling the writing natively?

Correct, in Python but using rustac to get the Arrow schema for a batch of items. We're also going to include collections in their own table (which will require sorting out radiantearth/stac-geoparquet-spec#17).

[jjcfrancisco]

Correct, in Python but using rustac to get the Arrow schema for a batch of items. We're also going to include collections in their own table (which will require sorting out radiantearth/stac-geoparquet-spec#17).

Makes sense. I’m happy to pause on the reading side or help out once the write side is ready.