[DRAFT] SA20: Add compatibility adapters for psycopg3 and asyncpg dialects

[amotl]

About

The idea is to unlock the CrateDB dialect for SQLAlchemy with the psycopg3 and asyncpg drivers by introducing corresponding crate+psycopg://, crate+asyncpg://, and crate+urllib3:// dialect identifiers.

The asynchronous variant of psycopg is also supported and will be automatically selected when using create_async_engine() instead of create_engine().

Installation

git clone https://github.com/crate/crate-python --branch=amo/postgresql-async
cd crate-python
python3 -m venv .venv
source .venv/bin/activate
pip install --editable='.[postgresql,sqlalchemy]'

Examples

Those example programs exercise basic conversations using the specified drivers.

python examples/sync_table.py urllib3 psycopg
python examples/async_table.py psycopg asyncpg
python examples/async_streaming.py psycopg asyncpg

References

• Support asyncpg with SQLAlchemy 1.4 sqlalchemy-cratedb#80
• Support SQLAlchemy dialect with asyncpg sqlalchemy-cratedb#79
• SQLAlchemy backlog sqlalchemy-cratedb#74

[amotl]

Backlog

1. Missing test cases

C'est la vie. Please use the provided example programs until we have them.

2. Still using PGDDLCompiler

There is more work to do, to make the hybrid dialect use the CrateDDLCompiler instead of the PGDDLCompiler. Otherwise, when defining a column with autoincrement=True, it will obviously croak, because it will render the SERIAL type, which CrateDB does not understand.

sa.Column("x", sa.Integer, primary_key=True, autoincrement=True)

CREATE TABLE testdrive (
	x SERIAL NOT NULL,
	y INT,
	PRIMARY KEY (x)
)

sqlalchemy.exc.InternalError: (psycopg.errors.InternalError_) Cannot find data type: serial

3. Error Cannot find portal: c_10479c0a0_1 with psycopg3

This flaw has been observed with the psycopg driver in asynchronous mode when using result streaming based on server-side cursors. It can be exercised using the async_streaming.py example program.

python examples/async_streaming.py psycopg

The same program works fine when using the asyncpg driver.

python examples/async_streaming.py asyncpg

Introduction

crate-python/examples/async_streaming.py

Lines 2 to 15 in 387b19f

	About
	=====
	Example program to demonstrate how to connect to CrateDB using its SQLAlchemy
	dialect, and exercise a few basic examples using the low-level table API, this
	time in asynchronous mode.
	Specific to the asynchronous mode of SQLAlchemy is the streaming of results:
	> The `AsyncConnection` also features a "streaming" API via the `AsyncConnection.stream()`
	> method that returns an `AsyncResult` object. This result object uses a server-side cursor
	> and provides an async/await API, such as an async iterator.
	>
	> -- https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html#synopsis-core

Bug

crate-python/examples/async_streaming.py

Lines 42 to 47 in 387b19f

	Bugs
	====
	When using the `psycopg` driver, the program currently croaks like::
	sqlalchemy.exc.InternalError: (psycopg.errors.InternalError_) Cannot find portal: c_10479c0a0_1

References

• https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html#synopsis-core
• https://docs.sqlalchemy.org/en/20/_modules/examples/asyncio/basic.html

[amotl]

This patch has been replaced by two others.

• Dialect: Add support for asyncpg and psycopg3 drivers sqlalchemy-cratedb#11
• Python/SQLAlchemy: Demonstrate support for asyncpg and psycopg3 cratedb-examples#201

/cc @SStorm, @surister