Skip to content

fix: Use consistent snapshot for ExportTableToPointInTime#80

Open
robinnsc wants to merge 2 commits into
mainfrom
fix/57-export-consistent-snapshot
Open

fix: Use consistent snapshot for ExportTableToPointInTime#80
robinnsc wants to merge 2 commits into
mainfrom
fix/57-export-consistent-snapshot

Conversation

@robinnsc
Copy link
Copy Markdown
Collaborator

What

With this change, the ExportTableToPointInTime full table scan will now be run inside a REPEATABLE READ transaction, guaranteeing every page reflects the table state at the time of the first read.

Why

Closes #57

ExportTableToPointInTime previously ran a paginated scan loop without a transaction, so items written or deleted between pages could alter the final export. The resulting file was not a consistent snapshot of the table.

Testing done

  • cargo test --workspace passes
  • cargo clippy --all-targets -- -D warnings clean
  • Verified the fix locally by running the manual concurrent-write scenario against a populated table — kicking off ExportTableToPointInTime on a 5,000-item table and writing additional items in parallel from a second client during the export, then confirming the resulting file contained only the original items. A dedicated unit/regression test for snapshot isolation is intentionally deferred to a follow-up PR that will introduce the direct-to-PostgreSQL test harness needed to drive this scenario deterministically; the existing test_import_export.py suite continues to cover the basic export happy-path.

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

ExportTableToPointInTime previously ran a paginated scan loop without a transaction, so items written or deleted between pages could alter the final export. The resulting file was not a consistent snapshot of the table.

With this change, the ExportTableToPointInTime full table scan will now be run inside a REPEATABLE READ READ ONLY transaction, guaranteeing every page reflects the table state at the time of the first read.

Closes #57
Copy link
Copy Markdown
Collaborator

@jcshepherd jcshepherd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My overarching concern with this is that on a large table (e.g. 1M or 1B items) with a lot of concurrent write activity, maintaining read isolation for a full table scan within a single transaction may lead to a large amount of storage space being consumed for item versions created after the transaction starts, which vacuum won't be able to reclaim space for until the transaction ends. This is potentially many GBs of additional storage consumed by a large, hot table that export is running on. It looks like the check for the item count happens outside the transaction (correct me if I'm wrong), so even with an export limit, it'll still do a full table scan. A partial mitigation would be to move the item count limit check inside the transaction and end the operation early if it's reached. Is that feasible?

@robinnsc
Copy link
Copy Markdown
Collaborator Author

robinnsc commented May 21, 2026

My overarching concern with this is that on a large table (e.g. 1M or 1B items) with a lot of concurrent write activity, maintaining read isolation for a full table scan within a single transaction may lead to a large amount of storage space being consumed for item versions created after the transaction starts, which vacuum won't be able to reclaim space for until the transaction ends. This is potentially many GBs of additional storage consumed by a large, hot table that export is running on. It looks like the check for the item count happens outside the transaction (correct me if I'm wrong), so even with an export limit, it'll still do a full table scan. A partial mitigation would be to move the item count limit check inside the transaction and end the operation early if it's reached. Is that feasible?

Yeah that's a good call. Made that change in the latest commit to move the item count check into the on_page callback. Now when the count limit is exceeded, the callback would return a validation error, which short circuits scan_full_table_snapshot_impl without committing, and releases the REPEATABLE READ snapshot so vacuum can resume reclaiming space

@robinnsc robinnsc requested a review from jcshepherd May 26, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] ExportTableToPointInTime is not a consistent snapshot

2 participants