persist blocks and FullCommitQCs in data layer via WAL#3126
Open
wen-coding wants to merge 4 commits intomainfrom
Open
persist blocks and FullCommitQCs in data layer via WAL#3126wen-coding wants to merge 4 commits intomainfrom
wen-coding wants to merge 4 commits intomainfrom
Conversation
Add GlobalBlockPersister (using indexedWAL) to persist data.inner.blocks across restarts. Each WAL entry embeds its GlobalBlockNumber since Block doesn't carry it. data.NewState now accepts a GlobalBlockWAL interface and pulls preloaded blocks from it via LoadedBlocks(). Fix a bug in PushQC where updateNextBlock was skipped when blocks were already preloaded but the QC arrived without new blocks. Includes integration test for the reload path and unit tests for the persister. Made-with: Cursor
Same pattern as the commitqcs bug: in no-op mode, truncateBefore returned early without advancing s.next. Also improve the no-op test to exercise truncate-then-persist. Made-with: Cursor
Add GlobalCommitQCPersister alongside GlobalBlockPersister so the data layer can recover both blocks and QCs after a crash. Without this, a restart could leave blocks without their verifying QCs (or vice-versa) because avail may prune QCs on a different schedule. Key changes: - Add ReadAt to indexedWAL for random-access reads - Add GlobalCommitQCPersister (one WAL entry per FullCommitQC) - Introduce DataWAL struct grouping both persisters with TruncateBefore and Close helpers - NewState loads QCs from WAL and calls updateNextBlock during construction so nextBlock is correct immediately on recovery - NewDataWAL constructor; no-op persisters when stateDir is None - Replace TestNewStateReloadsPreloadedBlocks with end-to-end TestStateRecoveryFromWAL that pushes through PushQC and restarts Made-with: Cursor
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3126 +/- ##
==========================================
+ Coverage 58.62% 58.65% +0.03%
==========================================
Files 2099 2101 +2
Lines 173751 174085 +334
==========================================
+ Hits 101867 102117 +250
- Misses 62816 62870 +54
- Partials 9068 9098 +30
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
- Fix gofmt: indentation in runPruning, extra blank line after NewDataWAL - Fix staticcheck S1016: use type conversion in loadAllGlobalBlocks - Use errors.Join in DataWAL.Close() so both WALs are always closed - Remove redundant LoadedGlobalCommitQC struct; LoadedQCs() now returns []*types.FullCommitQC directly since First is derivable from the QC - Inline nextBlock advancement in NewState recovery to avoid recording stale receive-latency metrics on restart Made-with: Cursor
wen-coding
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GlobalBlockPersisterandGlobalCommitQCPersisterbacked byindexedWAL, so the data layer can recover both blocks and QCs after a crashDataWALstruct withNewDataWAL,Close, andTruncateBeforehelpers; integrate intodata.StateNewStateloads blocks and QCs from WAL on startup and reconciles cursors sonextBlockis correct immediately on recoveryReadAttoindexedWALfor random-access reads needed by variable-size QC truncationTest plan
globalblocks_test.go: persist & reload, truncate & reload, truncate all, no-op, duplicate ignored, gap error, continue after reloadglobalcommitqcs_test.go: persist & reload, truncate & reload, truncate all, no-op, duplicate ignored, gap error, mid-range truncation, continue after reloadstate_test.go:TestStateRecoveryFromWAL— pushes QCs viaPushQC, closes WALs, reopens from same dir, verifiesNextBlock()and all blocks/QCs recovereddata,avail,consensus,p2p/giga)