Skip to content

cmd/utils/app: import all RLP files in a single process#21514

Closed
yperbasis wants to merge 1 commit into
mainfrom
yperbasis/import-multifile
Closed

cmd/utils/app: import all RLP files in a single process#21514
yperbasis wants to merge 1 commit into
mainfrom
yperbasis/import-multifile

Conversation

@yperbasis
Copy link
Copy Markdown
Member

Problem

The nightly ethpandaops/hive-tests legacy (ethereum/consensus) suite fails 9 tests against erigon main — all variants of ForkStressTest (×5) and walletReorganizeOwners (×4). They are timeouts, not consensus failures: each ran ~180.5s (hive's --client.checktimelimit=180s) and was killed mid-import with no errors.

Root cause: the consensus simulator feeds one RLP block per file (/blocks/NNNN.rlp, no chain.rlp), and erigon's hive entrypoint invoked erigon import once per file. Each invocation is a full node startup (DB open, P2P networking, private gRPC RPC, MCP server) costing ~0.66s before a single block is read. Reorg-stress tests have hundreds of blocks (ForkStressTest = 303, walletReorganizeOwners = 259), so 303 × 0.66s ≈ 200s > 180s → the client never becomes RPC-ready in time. The other 32 607 tests have few blocks and pass.

Fix

importChain documented multi-file support (<filename> (<filename 2> ... <filename N>)) but only imported cliCtx.Args().First(). It now imports every file in one node process via importFiles, tolerating per-file failures when several files are given (fatal for a single file) — matching the documented contract and preserving the InvalidBlocks-test behavior the old per-process loop provided. The MCP server is also force-disabled for the one-shot import, alongside the existing NAT/downloader/external-consensus disables.

This is the binary half of the fix. The load-bearing companion batches the per-file loop in hive's clients/erigon/erigon.sh into one erigon import invocation (separate PR to ethereum/hive); both are needed to clear the suite.

Validation

Reconstructed the exact hive scenario from the local legacy-tests fixtures + client logs and ran the rebuilt binary:

Test Blocks Processes Time Resulting head
ForkStressTest_Frontier (12-fork reorg) 303 1 <1s 0xc14ab05e… ✅ = lastblockhash
walletReorganizeOwners_Homestead 259 1 <1s 0xda82854c… ✅ = lastblockhash
ForkStressTest + injected malformed files 1 bad files skipped (no panic), head still 0xc14ab05e…

The in-process side-chain/reorg logic yields the correct canonical head ~200× faster. This mirrors the native testutil.BlockTest harness, which already loops InsertChain per block in a single process for these same tests.

  • make lint → 0 issues
  • make erigon integration build clean
  • New unit test import_cmd_test.go (TDD, Red→Green): all-files-imported + single-file-fatal + multi-file-tolerant

🤖 Generated with Claude Code

The `import` command documented multi-file support but only imported
cliCtx.Args().First(). The hive ethereum/consensus simulator feeds one
RLP block per file (/blocks/NNNN.rlp), so its erigon entrypoint invoked
`erigon import` once per file — a full node startup (DB, P2P, private RPC,
MCP) per block. Reorg-stress tests with hundreds of blocks (ForkStressTest
303, walletReorganizeOwners 259) thus exceeded hive's 180s readiness limit
and were killed.

importChain now imports every argument in one node process, tolerating
per-file failures when several files are given (fatal for a single file),
matching the command's documented contract. The MCP server is also
force-disabled for the one-shot import. Paired with batching the import
loop in hive's clients/erigon/erigon.sh, this turns ~200s of per-block
startups into one startup (<1s for 303 blocks).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@yperbasis yperbasis requested a review from AskAlexSharov as a code owner May 29, 2026 14:20
@yperbasis
Copy link
Copy Markdown
Member Author

Closing as a duplicate of #21513, which is the canonical PR for this fix (ethereum/hive#1519 references #21513 by number as its dependency). #21513 is functionally equivalent. The only delta here was also force-disabling the MCP server during one-shot import — happy to fold that into #21513 separately if wanted.

@yperbasis yperbasis closed this May 29, 2026
@yperbasis yperbasis deleted the yperbasis/import-multifile branch May 29, 2026 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant