-
Notifications
You must be signed in to change notification settings - Fork 304
Add API for source metadata snapshots #3749
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Add a new API endpoint `/v1/metadata-snapshot` that returns a snapshot of the PostgreSQL source's metadata for debugging purposes. The response includes: - xmin: Oldest visible transaction ID - xmax: Next transaction ID to be assigned - xip_list: List of in-progress transaction IDs at the time of the snapshot - database_lsn: Current WAL log sequence number (as string) This endpoint is protected by the same authentication as the shape API when ELECTRIC_SECRET is configured.
Significantly expand the /v1/metadata-snapshot endpoint to provide comprehensive debugging information: Global metadata: - database: PostgreSQL snapshot info (xmin, xmax, xip_list, lsn) - status: Service connection and shape subsystem status - shape_count: Total number of active shapes Per-shape metadata (in shapes array): - handle: Unique shape identifier - definition: Shape definition including: - table name and OID - primary key columns - where clause (if filtered) - selected columns (if specified) - replica mode and log mode - storage config - dependency handles (for subqueries) - status: snapshot_started and snapshot_completed flags - latest_offset: Current log offset for this shape - pg_snapshot: PostgreSQL snapshot at shape creation time This provides all the information needed to debug shape behavior, replication state, and storage issues.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Comment |
❌ 1 Tests Failed:
View the top 2 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
|
Found 5 test failures on Blacksmith runners: Failures
|
Plug to create a comprehensive snapshot of the source's metadata for debugging purposes.
Returns a JSON object containing:
Global Metadata
database: Current PostgreSQL snapshot informationxmin: Oldest visible transaction IDxmax: Next transaction ID to be assignedxip_list: List of in-progress transaction IDslsn: Current WAL log sequence number (as string)status: Service connection statusshape_count: Total number of active shapesPer-Shape Metadata (in
shapesarray)handle: Unique shape identifierdefinition: Shape definition including table, where clause, columnsstatus: Shape status (snapshot_started, snapshot_completed)latest_offset: Current log offset for this shapepg_snapshot: PostgreSQL snapshot at shape creation time (if available){ "database": { "xmin": 12345, "xmax": 12347, "xip_list": [12346], "lsn": "100" }, "status": { "connection": "up", "shape": "up" }, "shape_count": 2, "shapes": [ { "handle": "items-1234567890", "definition": { "table": "public.items", "root_table_id": 16385, "primary_key": ["id"], "replica": "default", "log_mode": "full", "where": "status = 'active'", "columns": ["id", "name", "status"], "storage": {"compaction": "disabled"}, "flags": {}, "dependency_handles": [] }, "status": { "snapshot_started": true, "snapshot_completed": true }, "latest_offset": "0_0", "pg_snapshot": { "xmin": 12340, "xmax": 12342, "xip_list": [12341] } } ] }