cmd/utils/app: import all RLP files in a single process#21514
Closed
yperbasis wants to merge 1 commit into
Closed
Conversation
The `import` command documented multi-file support but only imported cliCtx.Args().First(). The hive ethereum/consensus simulator feeds one RLP block per file (/blocks/NNNN.rlp), so its erigon entrypoint invoked `erigon import` once per file — a full node startup (DB, P2P, private RPC, MCP) per block. Reorg-stress tests with hundreds of blocks (ForkStressTest 303, walletReorganizeOwners 259) thus exceeded hive's 180s readiness limit and were killed. importChain now imports every argument in one node process, tolerating per-file failures when several files are given (fatal for a single file), matching the command's documented contract. The MCP server is also force-disabled for the one-shot import. Paired with batching the import loop in hive's clients/erigon/erigon.sh, this turns ~200s of per-block startups into one startup (<1s for 303 blocks). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
Author
|
Closing as a duplicate of #21513, which is the canonical PR for this fix (ethereum/hive#1519 references #21513 by number as its dependency). #21513 is functionally equivalent. The only delta here was also force-disabling the MCP server during one-shot import — happy to fold that into #21513 separately if wanted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The nightly
ethpandaops/hive-testslegacy(ethereum/consensus) suite fails 9 tests against erigonmain— all variants ofForkStressTest(×5) andwalletReorganizeOwners(×4). They are timeouts, not consensus failures: each ran ~180.5s (hive's--client.checktimelimit=180s) and was killed mid-import with no errors.Root cause: the consensus simulator feeds one RLP block per file (
/blocks/NNNN.rlp, nochain.rlp), and erigon's hive entrypoint invokederigon importonce per file. Each invocation is a full node startup (DB open, P2P networking, private gRPC RPC, MCP server) costing ~0.66s before a single block is read. Reorg-stress tests have hundreds of blocks (ForkStressTest = 303, walletReorganizeOwners = 259), so 303 × 0.66s ≈ 200s > 180s → the client never becomes RPC-ready in time. The other 32 607 tests have few blocks and pass.Fix
importChaindocumented multi-file support (<filename> (<filename 2> ... <filename N>)) but only importedcliCtx.Args().First(). It now imports every file in one node process viaimportFiles, tolerating per-file failures when several files are given (fatal for a single file) — matching the documented contract and preserving the InvalidBlocks-test behavior the old per-process loop provided. The MCP server is also force-disabled for the one-shot import, alongside the existing NAT/downloader/external-consensus disables.This is the binary half of the fix. The load-bearing companion batches the per-file loop in hive's
clients/erigon/erigon.shinto oneerigon importinvocation (separate PR toethereum/hive); both are needed to clear the suite.Validation
Reconstructed the exact hive scenario from the local legacy-tests fixtures + client logs and ran the rebuilt binary:
0xc14ab05e…✅ =lastblockhash0xda82854c…✅ =lastblockhash0xc14ab05e…✅The in-process side-chain/reorg logic yields the correct canonical head ~200× faster. This mirrors the native
testutil.BlockTestharness, which already loopsInsertChainper block in a single process for these same tests.make lint→ 0 issuesmake erigon integrationbuild cleanimport_cmd_test.go(TDD, Red→Green): all-files-imported + single-file-fatal + multi-file-tolerant🤖 Generated with Claude Code