Report stable physical catalog for DuckLake-backed sessions (single- and multi-tenant)#644
Open
fuziontech wants to merge 2 commits into
Open
Report stable physical catalog for DuckLake-backed sessions (single- and multi-tenant)#644fuziontech wants to merge 2 commits into
fuziontech wants to merge 2 commits into
Conversation
b11465b to
3c34fb7
Compare
3c34fb7 to
c11a930
Compare
A serverless duckling broke its nightly SQLMesh run after its connection
dbname changed (ducklake -> portola): current_database()/pg_database were
installed from the connection's startup dbname, and SQLMesh fully-qualifies
every model as catalog.schema.object and persists the catalog in its state.
A changed default catalog reads as a brand-new warehouse, so SQLMesh wants
to rebuild every model; mid-transition you also get "Catalog X does not
exist" when a reference under the other name isn't translated.
Install current_database()/pg_database from a stable catalog name chosen by
sessionmeta.ReportedDatabaseName, in precedence order:
1. a configured default catalog (e.g. "iceberg") — that session's
search_path/USE points there, so it is what current_database() must
report;
2. else, for a DuckLake-backed session, the physical catalog "ducklake"
(gated on the actual runtime DuckLake attachment);
3. else, the connection dbname unchanged (plain DuckDB / no default).
The connection dbname still works as an alias for DuckLake sessions: the
logical-catalog transform rewrites <dbname>.schema.table ->
ducklake.schema.table, the USE rewriter maps USE "<dbname>" -> ducklake.main,
and ducklake.* references resolve directly — so both names resolve and a
half-migrated SQLMesh state can't strand on "Catalog X does not exist".
Reporting "ducklake" for both single- and multi-tenant DuckLake sessions
makes the catalog identity consistent across deployments: an org graduating
from a single-tenant duckling to a multi-tenant worker pod keeps the same
catalog name and does not churn SQLMesh. iceberg-default sessions report
"iceberg" so their catalog identity is likewise stable and correct.
This intentionally changes the client-visible behavior of the logical
database catalog feature for DuckLake sessions: current_database()/
pg_database/information_schema/SHOW DATABASES now report the stable catalog
("ducklake", or "iceberg" for iceberg-default users) instead of the
connection/logical dbname. Org identity for observability is still available
via orgID (logged separately). Tests asserting the old contract are updated:
- tests/integration/logical_catalog_mapping_test.go
- tests/integration/logical_database_catalog_mapping_test.go
- tests/integration/logical_database_catalog_test.go
- tests/k8s/sni_test.go
The alias-execution assertions in those tests are kept (the connection
dbname still resolves for writes), only the metadata-reporting expectations
flip.
Also: use the shared physicalDuckLakeCatalog/PhysicalDuckLakeCatalog const
at the HasAttachedCatalog call sites and newTranspiler's PhysicalCatalogName
(they must stay in lockstep for the alias guarantee), and refresh the stale
control.go comment that claimed current_database() surfaces the per-org
routing database.
Both entry points covered: the control-plane connection handler and the
standalone serve() path; each detects DuckLake attachment before installing
session metadata. Adds sessionmeta.ReportedDatabaseName (+ shared
PhysicalDuckLakeCatalog const) with unit coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
c11a930 to
f689fd9
Compare
Now that current_database() reports the physical catalog name ("ducklake")
for DuckLake-backed sessions, pg-compat clients that build
catalog.schema.object from current_database() emit ducklake.public.<table>.
Previously the public->main mapping only ran for the logical catalog name
(PublicSchemaTransform deliberately leaves 3-part names alone, and
LogicalCatalogTransform only matched the logical name), so ducklake.public.*
fell through unrewritten and failed — DuckLake's default schema is "main",
not "public".
Extend LogicalCatalogTransform.rewriteRangeVar to also map "public" -> "main"
when a reference already uses the physical catalog name, mirroring the
logical-name case. Unrelated external catalogs (e.g. postgres.public.*) are
untouched because the match is gated on PhysicalCatalogName.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A serverless / single-tenant duckling failed its nightly SQLMesh run with
Catalog "portola" does not exist, and SQLMesh also wanted to rebuild the entire warehouse. Root cause:current_database()/pg_database) is installed from the connection's startup dbname (InitSessionDatabaseMetadata).ducklake→portola.catalog.schema.objectand persists the catalog in state, so a changed default catalog reads as a brand-new warehouse → full rebuild of ~140 models. Mid-transition you also getCatalog X does not existwhen a reference under the other name isn't translated.Fix
Install
current_database()/pg_databasefrom a stable catalog name (sessionmeta.ReportedDatabaseName), in precedence order:iceberg) — that session'ssearch_path/USEpoints there, so it's whatcurrent_database()must report;ducklake(gated on the actual runtime DuckLake attachment);The connection dbname still works as an alias for DuckLake sessions (logical-catalog transform rewrites
<dbname>.schema.table→ducklake.schema.table,USE "<dbname>"→ducklake.main, andducklake.*resolves directly) — so both names resolve and a half-migrated SQLMesh state can't strand onCatalog X does not exist.Reporting
ducklakefor single- and multi-tenant DuckLake sessions makes the catalog identity consistent across deployments, so an org graduating from a single-tenant duckling to a multi-tenant worker pod keeps the same catalog name and does not churn SQLMesh. iceberg-default sessions reporticeberg.This intentionally changes the client-visible behavior of the logical-catalog feature for DuckLake sessions:
current_database()/pg_database/information_schema/SHOW DATABASESnow report the stable catalog (ducklake, oricebergfor iceberg-default users) instead of the connection/logical dbname. Org identity for observability remains viaorgID(logged separately). The feature tests asserting the old contract are updated (alias-execution assertions kept; only metadata-reporting expectations flip):tests/integration/logical_catalog_mapping_test.go,logical_database_catalog_mapping_test.go,logical_database_catalog_test.gotests/k8s/sni_test.goSecond commit (
public→mainfor direct physical-catalog refs): now that clients readcurrent_database()=ducklakeand may emitducklake.public.<table>, theLogicalCatalogTransformmapspublic→mainfor the physical catalog too (it previously only did so for the logical name;PublicSchemaTransformleaves 3-part names alone). Gated onPhysicalCatalogName, sopostgres.public.*etc. are untouched.Scope / consistency
Gating on runtime attachment leaves non-DuckLake sessions (e.g. iceberg-only orgs with no
ducklakeattached) reporting their own dbname. Covers both entry points: control-plane handler + standaloneserve(). The change is control-plane behavior only — the worker just executes the macro SQL the control plane sends.Verified live on portola (single-tenant duckling)
Deployed
4d6fed5(currently-running base) + this fix via a graceful control-plane reload (systemctl reload→ SIGUSR1/tableflip), worker left untouched:dbname=portola→SELECT current_database():portola→ducklake✅dbname=ducklake→ducklake(unchanged) ✅dbname=portola→SHOW DATABASES→ducklake✅Deployment notes
systemctl reload duckgres.Tests
TestReportedDatabaseName: DuckLake →ducklake(single- and multi-tenant); iceberg default →iceberg(incl. wins-over-ducklake); non-DuckLake → connection dbname.TestTranspile_LogicalDatabaseCatalogMapping/physical_ducklake_public_schema_maps_to_mainfor thepublic→mainfix.go build/go vet; transpiler, server, sessionmeta suites pass.Test plan
go test ./server/sessionmeta/... ./transpiler/...go test ./server/ -run 'Logical|DirectQueryRewrite|SessionDatabase|InitSession'go build ./...,go vetdatabase=portola→current_database()=ducklake;SHOW DATABASES=ducklake(verified live, see above)Out of scope
database=portola(this PR makes the catalog name stable regardless).->/->>Conversion Error: Failed to cast value to numerical errors from the same run — separate issue (DuckDB JSON-path, likely 1.5.3 fallout).🤖 Generated with Claude Code