Discussion: Should Comet add geospatial (ST_*) function support?

[andygrove]

Background

PR #4423 proposes adding 40 native geospatial SQL functions (ST_Contains, ST_Intersects, ST_Distance, etc.) directly into Comet, executed via DataFusion using new Rust dependencies (geo, geoarrow, geojson, geos, wkt). The functions are wired through CometSparkSessionExtensions so that users get them automatically once Comet is enabled — no Sedona dependency required.

This represents a potential shift in scope for Comet, which to date has focused on accelerating Spark's built-in expressions and operators. The discussion on the PR raised several open questions that deserve broader community input, ideally on the dev mailing list as well as here.

Questions for discussion

1. Should geospatial support be in scope for Comet at all? Comet has historically targeted Spark built-ins (math, string, datetime, aggregates, etc.). ST_* functions are not Spark built-ins — they come from Apache Sedona or other extensions. Adding them would expand Comet's surface area into a domain that already has dedicated projects.

2. If yes, what is the right implementation path?

   • In-tree, maintained by Comet — as proposed in feat: add 40 native geospatial SQL functions (ST_ expressions) #4423. Comet owns the function definitions, tests, and dependencies (including the GEOS C library via geos crate with static linking).
   • Wrap SedonaDB — @paleolimbot noted that SedonaDB has ~100 functions plus join/Parquet IO already implemented, tested, and benchmarked in Rust. Comet could wrap those, limiting Comet's maintenance burden to thin wrapper code.
   • Defer to Sedona / Wherobots — users who need geo today already have options (Sedona on Spark; Wherobots offers a Rust-accelerated path). Comet could choose not to enter this space.

3. Geometry representation. The PR uses WKT strings. @paleolimbot pointed out that Spark, Parquet, and SedonaDB all use WKB, which is significantly faster ("the equivalent of passing around doubles as strings"). If Comet adopts geo, what is the right representation, and does that depend on broader UDT support?

4. UDT / Spark geometry type support. @paleolimbot mentioned that full UDT support would require changing many DataType usages to FieldRef usages. Spark geometry has a type parameter that is dropped when represented as Utf8. Is this a prerequisite for doing geo "properly," and is it work the project wants to take on?

5. Build and runtime dependencies. The proposed approach adds a native dependency on GEOS (statically linked, so end users don't need it at runtime, but build machines do). How does the community feel about adding a C library dependency to the Comet build?

6. Maintenance burden. Geo functions are a large surface area (the PR adds 40; SedonaDB has ~100+). Who maintains them, who reviews changes, and who handles compatibility as Sedona/Spark evolve?

References

• PR feat: add 40 native geospatial SQL functions (ST_ expressions) #4423 — feat: add 40 native geospatial SQL functions (ST_ expressions) #4423
• SedonaDB — https://github.com/apache/sedona
• Wherobots — https://wherobots.com/

Next steps

@andygrove suggested taking this to the dev@ mailing list given the scope shift implications. This issue is intended to collect written input from contributors and users before/alongside that discussion. Please weigh in with your perspective, especially if you have a use case for geo in Comet or experience maintaining geospatial libraries.

[Shekharrajak]

Geo datatype is highlight of Iceberg v3 apache/iceberg#10260 spec, so prioritizing this feature ensures early adoption and alignment with the new standard.

[jiayuasu]

Hi, this is Jia from Apache Sedona / Wherobots. We were also a driving force of the Parquet Geo and Iceberg Geo design and implementation.

Here is my 2 cents.

Spark 4.2 will add the following 5 geospatial constructor and accessor functions to enable Parquet and Iceberg read/write (reference):

• ST_AsBinary
• ST_GeogFromWKB
• ST_GeomFromWKB
• ST_SRID
• ST_SetSRID

So if Comet wants to enable Iceberg V3 and Parquet read/write, we should add these 5 functions for Comet + Spark 4.2 compatibility, which unblocks Comet. Even just ST_GeogFromWKB and ST_SRID are already quite troublesome on their own, so I suggest we wrap SedonaDB for those.

Going beyond these 5 functions leads to two problems:

1. It changes the direction of Comet.

2. Maintaining ST functions is an enormous burden. Many people underestimate this. It is not as simple as wrapping GEOS, Rust GEO, etc. Three reasons:

   1. Function behavior differs across libraries. Do we want to follow SQL MM3, OGC, or PostGIS semantics?

   2. Lack of domain knowledge in code review. We have seen this play out in many projects. People push domain-specific functions into the main engine, then nobody can effectively review them or improve their performance. Users resort to other solutions, and the built-in component either lacks maintenance (HIVE Spatial SQL) or gets deprecated (Spark GraphX). Do we want to repeat this?

   3. If people want something quick for simple use cases, we should enable that via extensions or UDFs. For example, make it easy to use Comet together with SedonaDB, or implement UDF to wrap Rust GEO or GEOS. The main engine should not bear that burden and risk.

[paleolimbot]

Echoing all Jia's comments plus linking #4283, which seems like a practical route to adding translations for other functions not limited to just spatial. There are likely some other ways in which adding spatial support as an extension can strengthen extensibility for other cases (this has certainly played out in DuckDB, PostGIS, and DataFusion).

While I do think maintaining these externally is a better solution, I'm happy to review anything the Comet community agrees on (i.e., if you'd like to go that direction I'm happy to help get the details right!).

[zhangfengcdt]

In addition to what @jiayuasu has pointed out for the complexity of supporting those st functions compatible with other geospatial analytics engine, the combinations of them into joins or sub-queries would make it really challenge from the core itself. For example, using st predicates in a join condition without dedicated spatial partitioning would easily make the join impractical to run.

That said, I think it is much cleaner and reasonable to make st functions extensible to take advantage of other tools in the datafusion ecosystems, such as sedona-db.

[andygrove]

Thanks for all the feedback so far. I personally don't have any experience with geospatial, and review bandwidth in Comet is already an issue, so I would prefer an integration approach.

As @paleolimbot pointed out, we have an experimental PR (#4283) for supporting custom Rust UDFs, and geospatial support provides more justification for me to spend time on that.

[james-willis]

I built the UDT for spark 4.1 geom/geog support in lake sail: lakehq/sail#1325

Could be valuable for how comet wants to proceed if they are looking to support spark's (limited) new spatial functionality.

But in general I am aligned with the posters above that generally these domain specific functionalities are best served by extensions where they can be maintained primarily by folks with domain specific expertise, and have their own norms about release practices, code quality, etc.

[parthchandra]

+1 to what Jia said.
I would love to see Comet have the same functionality as Spark, and a clean integration with Sedona/Wherobots.

[sp-202]

I understand the concerns around maintenance and scope.

For the broader geo support, the extension approach via #4283 makes more sense long term.