Configurable attestations saving + validator handler reliability/observability#121
Open
kirill-simakov-p2p wants to merge 5 commits into
Open
Configurable attestations saving + validator handler reliability/observability#121kirill-simakov-p2p wants to merge 5 commits into
kirill-simakov-p2p wants to merge 5 commits into
Conversation
Same pattern as blocks.blobs.enable (commit fdc70b8). When set to false, updateAttestationsForBlock and updateElectraAttestationsForBlock return early without writing to t_attestations. Useful for backfill scenarios where only deposit_requests and sync_committees are needed - avoids writing ~144 GB of attestation data.
In go-eth2-client v0.27.1, head event handlers receive the SSE connection context. When the SSE stream disconnects (Cloudflare timeout, network issues), the context gets canceled and kills in-flight beacon state downloads for validator balances (~325 MB, takes 8+ seconds). This was not an issue with go-eth2-client v0.24.0 where handlers did not receive the SSE context. The old DWH ChainD (v0.10.1-dbb, go-eth2-client v0.24.0) works fine with the same beacon nodes for this reason. Fix: use context.WithoutCancel() to detach from SSE context, restoring the v0.24.0 behavior.
Log elapsed time for each step of epoch processing: - validators fetch (state="head" download + DB compare) - balances fetch + write per epoch - total handler duration Needed to diagnose why balance gap grows overnight despite zero errors from CF/beacon node.
andrey-kozel
approved these changes
May 11, 2026
mikhail-kiselev-lambda
approved these changes
May 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three independent improvements we have been running on a fork for several weeks against mainnet and Hoodi nodes. Upstreaming now ahead of the in-flight #120 because two of the three touch files that overlap with that branch and we'd rather rebase you than ourselves.
1.
blocks.attestations.enableconfiguration flag (default:true)Mirrors the existing
blocks.blobs.enablepattern added in #119. Lets operators skip writing attestations contained in blocks when they aren't needed downstream (typical case: backfilling old slots where attestations dominate write volume).blocks.attestations.enablekey--blocks.attestations.enableCHAIND_BLOCKS_ATTESTATIONS_ENABLEattestationsSavingparameter on theblocksservice;updateAttestationsForBlockandupdateElectraAttestationsForBlockreturn early when disabled2. Detach validator handler from SSE event-stream context
The validator handler currently inherits the context of the head/finality SSE event that triggered it. When the event stream reconnects, that context is cancelled mid-write and partial writes are rolled back at the database boundary. We saw this manifest as repeated rollbacks under transient beacon-node disconnects.
Fix: derive a fresh
context.Background()for the downstream balance write so transactions are not at the mercy of the SSE socket.3. Fetch/write duration logs in validator balances handler
Adds
fetchandwritedurations to the existingBalance epoch processedinfo log. We use these to diagnose backpressure between beacon RPC and Postgres, and to attribute slowdowns to the right side.Tested
attestations.enable=falsecuts insert volume ont_attestationsto zero as expectedFiles
CHANGELOG.mdREADME.mdblocks.attestations.enablechaind.config.docker-compose.ymlmain.goservices/blocks/standard/{handler,parameters,service}.goservices/validators/standard/handler.goNotes for review
ReleaseVersion; happy to add a commit if you want this on0.10.3.