feat(l1): add experimental REST/SSZ /new-payload-with-witness endpoint

[developeruche]

This PR introduces an experimental POST /new-payload-with-witness REST endpoint to Ethrex.

Motivation
Witness retrieval today requires two round-trips , engine_newPayload followed by debug_executionWitness, with the response serialized as JSON. For large witnesses, this encoding alone inflates payload size by roughly 2×. On Ethrex, this combined flow currently takes ~8 seconds for a 500 MB witness block.

By serving the endpoint over HTTP with SSZ serialization and merging both operations into one, we reduce that to ~1.2 seconds, a ~6.5× improvement.

Description

• Adds a REST-based POST /new-payload-with-witness route outside the JSON-RPC layer
• Uses SSZ for both inbound payload and outbound witness encoding
• Validates the payload and produces the execution witness in a single atomic operation
• Marked as experimental, consistent with how debug_executionWitness was introduced

It is also important to note that there are ongoing discussions around introducing SSZ + HTTP to the Engine API more broadly, replacing JSON-RPC entirely, including proposals to combine "new-payload" execution and witness retrieval into a single operation. Once these discussions mature, we can standardize around a unified approach. However, this is a significant architectural change and will likely take time to reach consensus.

Engine: add Rest-SSZ spec
ethereum/execution-apis#793
Add SSZ to Engine API
ethereum/execution-apis#764
Proposal: opt-in witness retrieval via flag on existing engine methods
ethereum/execution-apis#799

[greptile-apps]

Greptile Summary

This PR adds an experimental POST /new-payload-with-witness REST endpoint to the Engine API that accepts a JSON-encoded ExecutionPayload, executes the block, and returns an SSZ-encoded witness in a single round-trip — replacing the existing two-call flow and cutting reported witness retrieval time from ~8 s to ~1.2 s.

• A new witness_block_executor thread and add_block_pipeline_with_witness_inner handle block execution and background storage, with a store_lock mutex used to prevent the next witness call from reading stale trie state until the previous block's store completes.
• The nonce mismatch check in default_hook.rs is commented out with a TODO tag, disabling a core EVM transaction validation guard for the entire node.
• The v5 REST handler directly indexes params[0] before the later parse_params length check, and background store failures are silently swallowed after the caller has already received a Valid response.

Confidence Score: 1/5

Not safe to merge — the nonce validation removal in default_hook.rs breaks a fundamental EVM security invariant for the entire node, not just the new endpoint.

The commented-out nonce mismatch check in default_hook.rs means every transaction processed by this node now bypasses the guard that ensures a transaction's nonce matches the sender's current account nonce. This is a node-wide regression unrelated to the experimental endpoint's own correctness. Additionally, the witness path's background store can silently fail after returning Valid to the consensus client, and the store_lock coordination gap means the regular block executor can read stale trie state if both execution paths are ever active simultaneously.

crates/vm/levm/src/hooks/default_hook.rs (nonce check removed), crates/blockchain/blockchain.rs (background store error handling and store_lock scope), crates/networking/rpc/engine/rest.rs (params bounds check)

Security Review

• Transaction nonce validation disabled (crates/vm/levm/src/hooks/default_hook.rs): The nonce mismatch check is commented out entirely. All EVM transaction processing now accepts transactions regardless of whether the sender nonce matches the account nonce, enabling replay and out-of-order transaction attacks across the entire node.

Important Files Changed

Filename	Overview
crates/vm/levm/src/hooks/default_hook.rs	Nonce mismatch check is fully commented out with a TODO; this disables a critical EVM transaction validation guard for all transaction processing.
crates/blockchain/blockchain.rs	Adds add_block_pipeline_with_witness_inner that stores blocks in a background thread and returns the witness early; storage errors are silently swallowed, and the store_lock does not coordinate with the regular block executor path.
crates/networking/rpc/engine/rest.rs	New REST endpoint for SSZ-encoded witness responses; params[0] is accessed without a prior length check in the v5 handler, which panics on an empty JSON array body.
crates/networking/rpc/engine/payload.rs	Adds PayloadStatusWithWitness, witness-aware handler variants, and add_block_with_witness; logic mirrors the existing non-witness handlers closely.
crates/networking/rpc/rpc.rs	Wires up the new REST routes and spawns a dedicated witness_block_executor thread; the witness worker channel is always started, making the experimental endpoint unconditionally active.
crates/networking/rpc/engine/mod.rs	Advertises rest_engine_newPayloadWithWitness in exchangeCapabilities response unconditionally.

Sequence Diagram

sequenceDiagram
    participant CL as Consensus Client
    participant REST as REST Handler
    participant WE as witness_block_executor
    participant BC as Blockchain
    participant BG as Background Store Thread
    participant DB as Storage

    CL->>REST: POST /new-payload-with-witness (JWT + JSON)
    REST->>REST: authenticate + parse + validate
    REST->>WE: send(oneshot_tx, block, bal)
    WE->>BC: add_block_pipeline_with_witness(block, bal)
    BC->>BC: acquire+drop store_lock (wait for prev BG)
    BC->>BC: execute_block_pipeline
    BC->>BC: generate_witness
    BC->>BG: spawn(store_lock held by BG)
    BC-->>WE: Ok(Some(rpc_witness))
    WE-->>REST: Ok(Some(rpc_witness))
    REST-->>CL: 200 OK SSZ response
    Note over BG,DB: Async — errors only logged
    BG->>DB: store_witness
    BG->>DB: store_block_static
    BG->>BG: release store_lock

Loading

Comments Outside Diff (2)

1. crates/blockchain/blockchain.rs, line 222-259 (link)

   Store failure is silently swallowed; endpoint returns Valid on storage error

   store_block_static is called on a detached background thread, and its Err path is only logged — it is never surfaced back to the caller. The witness and Valid response have already been returned to the REST caller before the background thread even starts, so a storage error leaves the node in a state where it confirmed block acceptance to the CL but never persisted the block.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: crates/blockchain/blockchain.rs
   Line: 222-259
   
   Comment:
   **Store failure is silently swallowed; endpoint returns `Valid` on storage error**
   
   `store_block_static` is called on a detached background thread, and its `Err` path is only logged — it is never surfaced back to the caller. The witness and `Valid` response have already been returned to the REST caller before the background thread even starts, so a storage error leaves the node in a state where it confirmed block acceptance to the CL but never persisted the block.
   
   How can I resolve this? If you propose a fix, please make it concise.

2. crates/blockchain/blockchain.rs, line 136-142 (link)

   store_lock does not coordinate with the regular block executor path

   The store_lock is acquired and immediately dropped at the top of add_block_pipeline_with_witness_inner to wait for the previous iteration's background store to finish. However, the lock is never acquired by add_block_pipeline_inner (the regular block_executor thread). If both execution paths are active concurrently, the regular executor will read trie state that may not yet reflect the previous block's writes, silently producing a corrupt state root or an execution failure against stale state.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: crates/blockchain/blockchain.rs
   Line: 136-142
   
   Comment:
   **`store_lock` does not coordinate with the regular block executor path**
   
   The `store_lock` is acquired and immediately dropped at the top of `add_block_pipeline_with_witness_inner` to wait for the previous iteration's background store to finish. However, the lock is never acquired by `add_block_pipeline_inner` (the regular `block_executor` thread). If both execution paths are active concurrently, the regular executor will read trie state that may not yet reflect the previous block's writes, silently producing a corrupt state root or an execution failure against stale state.
   
   How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

Fix the following 4 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 4
crates/vm/levm/src/hooks/default_hook.rs:82-92
**Nonce validation silently removed**

The nonce mismatch check has been commented out entirely. This check is the core EVM guard that prevents accepting transactions whose nonce doesn't match the current account nonce — without it, replayed transactions, out-of-order transactions, and already-consumed nonces all pass validation. This affects every transaction processed by the EVM node, not just the new witness endpoint, and the `TODO:DEVELOPERUCHE` tag signals it was intentionally deferred rather than fixed.

### Issue 2 of 4
crates/networking/rpc/engine/rest.rs:285-298
**Panic on empty or non-array JSON body**

`params[0]` is indexed directly before any length check. If the request body is a valid JSON array with fewer than 1 element (e.g., `[]`), this panics with an index-out-of-bounds. The `parse_params` call that does validate `params.len() != 4` comes later — after this direct indexing. An authenticated request that sends `[]` as the body will panic the handler task.

```suggestion
    if params.is_empty() {
        return json_error_response(-32602, "Parse error: Expected 4 params, got 0");
    }
    let Ok(raw_bal_hash) = params[0]
        .get("blockAccessList")
```

### Issue 3 of 4
crates/blockchain/blockchain.rs:222-259
**Store failure is silently swallowed; endpoint returns `Valid` on storage error**

`store_block_static` is called on a detached background thread, and its `Err` path is only logged — it is never surfaced back to the caller. The witness and `Valid` response have already been returned to the REST caller before the background thread even starts, so a storage error leaves the node in a state where it confirmed block acceptance to the CL but never persisted the block.

### Issue 4 of 4
crates/blockchain/blockchain.rs:136-142
**`store_lock` does not coordinate with the regular block executor path**

The `store_lock` is acquired and immediately dropped at the top of `add_block_pipeline_with_witness_inner` to wait for the previous iteration's background store to finish. However, the lock is never acquired by `add_block_pipeline_inner` (the regular `block_executor` thread). If both execution paths are active concurrently, the regular executor will read trie state that may not yet reflect the previous block's writes, silently producing a corrupt state root or an execution failure against stale state.

_{Reviews (1): Last reviewed commit: "update default kurtosis yaml" | Re-trigger Greptile}