Add utility support wrapping CrateDB's special table options

[amotl]

About

@wierdvanderhaar suggested to improve the ctk load table interface such that the target table can be created with CrateDB-specific features like partitioned table, and friends. Thanks.

This patch adds relevant support code, mostly for providing it to downstream libraries and applications. The first candidate is CrateDB Toolkit's I/O subsystem.

References

• [I/O] Enable using CrateDB-specific DDL statements on ctk load table interface cratedb-toolkit#149
• Add table_kwargs context manager to make pandas/Dask support CrateDB's special SQL DDL options #139

See also

• UX: Improve situation with weird __table_args__ syntax #142

[seut]

@amotl
Honestly, I do not like this approach as it would force use to maintain table options not only at CrateDB but also at this dialect. I'd really prefer to not add concrete options here. Why not just double quote CrateDB specific (all settings prefixed with crate_ when they get converted into the create table stmt?

[amotl]

Response

Why not just double quote CrateDB specific (all settings prefixed with crate_ when they get converted into the create table stmt?

This is probably about that issue?

• UX: Improve situation with weird __table_args__ syntax #142

This patch's aim is different, and beyond the UX improvements we would like to tackle there.

Idea

It is to have a map for shorthand versions of special table options to be conveyed via URL query parameters, like this:

export CRATEDB_SQLALCHEMY_URL='crate://crate@localhost:4200/testdrive/demo?if-exists=replace&partitioned-by=time&clustered-by="A"&replicas=0-2&shards=2&durability=async&max-columns=42&column-policy=dynamic'

The patch is exploring this area to have a proper metadata knowledge data structure in Python, to be used for different purposes. Currently, it is targeted for the ctk load table interface of CrateDB Toolkit.

It would force use to maintain table options not only at CrateDB but also at this dialect.

I hear you. About the maintenance details, it does not necessarily have to be the source of truth here. In a future iteration, both the documentation and API wrappers would be able to derive the information from CrateDB itself, either its sources, or by inquiring at runtime?