-
Notifications
You must be signed in to change notification settings - Fork 304
fix: Use end_lsn for commit fragment offset to fix max_batch_size boundary bug #3764
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
When a transaction has exactly max_batch_size changes, the commit fragment was getting the same last_log_offset as the preceding data fragment. This caused ShapeLogCollector to drop it as a duplicate. Changes: - Use Commit.end_lsn (not Begin.final_lsn) for commit fragment offset - Remove last_log_offset from MessageConverter state (now derived) - Make max_batch_size configurable via replication_opts for testing
- Update expected offsets in MessageConverter tests - Add test for exact batch boundary case - Adjust router test to handle global_last_seen_lsn >= op lsn
Add end-to-end tests verifying that transactions with exactly max_batch_size (100) changes are fully received by clients. These tests exercise the full pipeline (ReplicationClient -> ShapeLogCollector -> HTTP API -> Client) and confirm the fix for the bug where commit-only fragments at exact batch boundaries were silently dropped due to duplicate offsets.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Comment |
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3764 +/- ##
=======================================
Coverage 87.36% 87.36%
=======================================
Files 23 23
Lines 2011 2011
Branches 531 531
=======================================
Hits 1757 1757
Misses 252 252
Partials 2 2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Found 2 test failures on Blacksmith runners: Failures
|
Summary
Fixes a bug where transactions with exactly
max_batch_size(default: 100) changes had their commit fragment silently dropped byShapeLogCollector.Root Cause
When a transaction has exactly
max_batch_sizechanges:maybe_flush/1, returning Fragment 1 withlast_log_offset = (final_lsn, 198)maybe_flushcreates a new empty fragment but did NOT resetstate.last_log_offsetlast_log_offset = (final_lsn, 198)ShapeLogCollector.handle_txn_fragment/2drops Fragment 2 as duplicate (offset <= last_processed_offset)Solution
Use
Commit.end_lsninstead ofBegin.final_lsnfor the commit fragment'slast_log_offset. In Postgres logical replication,end_lsn > final_lsn, guaranteeing the commit fragment's offset is always greater than preceding data fragments.Changes
message_converter.ex: Removelast_log_offsetfrom state; useend_lsnfor commit offsetreplication_client.ex: Makemax_batch_sizeconfigurable via NimbleOptions