Draft
Conversation
Implement temporal sharding for the CCF ledger, allowing completed ledger ranges to be sealed into immutable shards. Sealed shards are archived to read_only_directories via host ring buffer messaging. Key changes: - New KV tables for shard info and shard policy (src/service/tables/shards.h) - Shard lifecycle management in node_state.h (seal, auto-seal, initialize) - Governance actions: seal_current_shard, set_shard_policy - JS bindings: ccf.node.sealShard(), ccf.node.setShardPolicy() - Host-side ledger chunk archival to read_only_directories/shards/<id>/ - Ring buffer message type: ledger_shard_sealed - Unit tests (src/node/test/sharding.cpp) and e2e test (tests/sharding.py) - Documentation (doc/operations/sharding.rst)
The seal flow was previously synchronous — seal_shard() immediately transitioned the shard to Sealed and sent the ring buffer message before the snapshot was actually committed. Now seal_shard() keeps the shard in Sealing state and shard_seal_in_progress = true. When the snapshotter commits the shard-seal snapshot, it fires on_shard_seal_committed which: - Updates the shard to Sealed in the KV - Sends the ledger_shard_sealed ring buffer message to the host - Clears shard_seal_in_progress This ensures ledger chunks are only archived after the boundary snapshot is durable, and a crash during sealing leaves the shard in a recoverable Sealing state.
- Add sharding object to cchost_config.json schema (required for node startup validation) - Fix -Wshadow: rename last_idx to chunk_last_idx in ledger.h - Fix -Wshadow: rename to_host to shard_writer in node_state.h - Fix deadlock: seal callback now sets atomic flag, complete_shard_seal() deferred to tick()
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement temporal sharding for the CCF ledger, allowing completed ledger ranges to be sealed into immutable shards. Sealed shards are archived to read_only_directories via host ring buffer messaging.
Key changes: