fix(lakekeeper): set ACCESS_DELEGATION_MODE 'none' on Iceberg ATTACH#600
Merged
Conversation
The Lakekeeper ATTACH omitted ACCESS_DELEGATION_MODE entirely, assuming that meant "no vending." It doesn't: DuckDB's iceberg extension defaults the mode to 'vended_credentials'. So the client still requested delegation, and since warehouse-side STS vending is disabled (#594, PackedPolicyTooLarge), Lakekeeper returned a per-table storage *config* (region/endpoint) with NO credentials. DuckDB materialized that as a path-scoped S3 secret with empty creds whose scope (the table's S3 prefix) is more specific than iceberg_sigv4's (s3://), so it shadowed the duckling's real brokered-creds secret. Every data read/write went out anonymous and S3 returned 403. Set ACCESS_DELEGATION_MODE 'none' explicitly on both ATTACH branches so DuckDB never creates that empty shadow secret and falls back to iceberg_sigv4. Verified live on `ben` in managed-warehouse-dev: with the option omitted, INSERT 403'd at the parquet write; with 'none', CREATE TABLE / INSERT / SELECT round-trip through Lakekeeper to S3 (data + metadata land in the warehouse bucket, rows read back). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Lakekeeper Iceberg
ATTACHomittedACCESS_DELEGATION_MODE, on the assumption that omitting it meant "don't vend." It doesn't — DuckDB's iceberg extension defaults the mode to'vended_credentials'. So the worker still requested credential delegation on every table load.Since warehouse-side STS vending is disabled (#594, to avoid
PackedPolicyTooLarge), Lakekeeper responded with a per-table storage config (region/endpoint) carrying no credentials. DuckDB materialized that into a path-scoped S3 secret with empty creds, and because its scope (the table's S3 prefix) is more specific thaniceberg_sigv4's (s3://), it shadowed the duckling's real brokered-creds secret. Every data read/write then went out anonymous → S3403 Forbidden.Catalog metadata ops (CREATE SCHEMA/TABLE via the Lakekeeper REST API) worked fine — only the client-side S3 data path was broken, which is why provisioning looked green.
Fix
Set
ACCESS_DELEGATION_MODE 'none'explicitly on both ATTACH branches (allowall + OAuth2). DuckDB then never creates the empty shadow secret and falls back to the ambienticeberg_sigv4secret, which holds the duckling's own brokered S3 creds for the warehouse bucket.Verification (live,
benin managed-warehouse-dev)INSERT→403 Forbiddenwriting the parquet data file;SELECTreturned 0 rows.ACCESS_DELEGATION_MODE 'none':CREATE SCHEMA/CREATE TABLE/INSERT/SELECTround-trip through Lakekeeper to S3 — parquet data + Iceberg metadata (metadata.json, manifest/snapshot.avro) land ins3://posthog-duckling-ben-mw-dev/lakekeeper/.../data/, and rows read back correctly.Unit tests updated to assert the delegation option on both branches.
🤖 Generated with Claude Code