Skip to content

Conversation

@alco
Copy link
Member

@alco alco commented Jan 23, 2026

Summary

Fixes a bug where transactions with exactly max_batch_size (default: 100) changes had their commit fragment silently dropped by ShapeLogCollector.

Root Cause

When a transaction has exactly max_batch_size changes:

  1. The 100th change triggers maybe_flush/1, returning Fragment 1 with last_log_offset = (final_lsn, 198)
  2. maybe_flush creates a new empty fragment but did NOT reset state.last_log_offset
  3. When the Commit arrives, Fragment 2 gets the same last_log_offset = (final_lsn, 198)
  4. ShapeLogCollector.handle_txn_fragment/2 drops Fragment 2 as duplicate (offset <= last_processed_offset)

Solution

Use Commit.end_lsn instead of Begin.final_lsn for the commit fragment's last_log_offset. In Postgres logical replication, end_lsn > final_lsn, guaranteeing the commit fragment's offset is always greater than preceding data fragments.

Changes

  • message_converter.ex: Remove last_log_offset from state; use end_lsn for commit offset
  • replication_client.ex: Make max_batch_size configurable via NimbleOptions
  • Unit tests: Update expected offsets and add boundary case test
  • Integration tests: New end-to-end tests for 99, 100, 101, and 200 row transactions

alco added 3 commits January 23, 2026 16:43
When a transaction has exactly max_batch_size changes, the commit
fragment was getting the same last_log_offset as the preceding data
fragment. This caused ShapeLogCollector to drop it as a duplicate.

Changes:
- Use Commit.end_lsn (not Begin.final_lsn) for commit fragment offset
- Remove last_log_offset from MessageConverter state (now derived)
- Make max_batch_size configurable via replication_opts for testing
- Update expected offsets in MessageConverter tests
- Add test for exact batch boundary case
- Adjust router test to handle global_last_seen_lsn >= op lsn
Add end-to-end tests verifying that transactions with exactly
max_batch_size (100) changes are fully received by clients.

These tests exercise the full pipeline (ReplicationClient ->
ShapeLogCollector -> HTTP API -> Client) and confirm the fix
for the bug where commit-only fragments at exact batch boundaries
were silently dropped due to duplicate offsets.
@coderabbitai
Copy link

coderabbitai bot commented Jan 23, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.


Comment @coderabbitai help to get the list of available commands and usage tips.

@netlify
Copy link

netlify bot commented Jan 23, 2026

Deploy Preview for electric-next ready!

Name Link
🔨 Latest commit 25e45e0
🔍 Latest deploy log https://app.netlify.com/projects/electric-next/deploys/69739a3c602982000819e4ac
😎 Deploy Preview https://deploy-preview-3764--electric-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov
Copy link

codecov bot commented Jan 23, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.36%. Comparing base (4c7a4b6) to head (25e45e0).
⚠️ Report is 1 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3764   +/-   ##
=======================================
  Coverage   87.36%   87.36%           
=======================================
  Files          23       23           
  Lines        2011     2011           
  Branches      531      531           
=======================================
  Hits         1757     1757           
  Misses        252      252           
  Partials        2        2           
Flag Coverage Δ
packages/experimental 87.73% <ø> (ø)
packages/react-hooks 86.48% <ø> (ø)
packages/start 82.83% <ø> (ø)
packages/typescript-client 93.47% <ø> (ø)
packages/y-electric 56.05% <ø> (ø)
typescript 87.36% <ø> (ø)
unit-tests 87.36% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blacksmith-sh
Copy link
Contributor

blacksmith-sh bot commented Jan 23, 2026

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
Elixir.Electric.ShapeCacheTest/test get_or_create_shape_handle/
2 against real db crashes when initial snapshot query fails to return data quickly enou
gh
View Logs
Elixir.Electric.Shapes.ConsumerTest/
test transaction handling with real storage should terminate after :hibernate_after ms
View Logs

Fix in Cursor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants