feat(tooling): detect and mark out-of-gas-by-design tests

[leolara]

Important

🛑 Stacked on #2906 — review and merge this PR only after #2906.

This branch is built on top of #2906 (leolara/mark-gas-checking-tests) and
reuses its scripts/mark_tests.py. Until #2906 merges into
forks/amsterdam, the diff below also contains #2906's commits. Once #2906
is merged and this branch is rebased onto the updated base, the diff collapses
to just the out-of-gas detection commit. Please do not merge before #2906.

🗒️ Description

Adds tooling to find tests that run out of gas by design — at the top-level
transaction or any sub-call depth — and mark them, complementing #2906's
gas-assertion detection.

scripts/detect_oog_by_design.py scans EELS EIP-3155 traces under an
--evm-dump-dir and flags every test whose execution ran out of gas. The
signal is exact equality on the trace error field == "OutOfGasError" (the
EELS exception class name written at
src/ethereum_spec_tools/evm_tools/t8n/evm_trace/eip3155.py), which is uniform
across every fork and distinct from all other halt classes. This gives zero
false positives against the other all-gas-consuming halts
(StackOverflowError, StackUnderflowError, InvalidOpcode, Revert). Fill
success is the "by design" filter: if a traced test filled successfully, any OOG
in its trace matched the author's declared post-state.

The output is the mark_tests.py wrapped form {marker, nodeids}, so the two
scripts chain directly. The lossy sanitized dump-dir names are reconstructed
into pytest node IDs and verified against the source AST (re-adding the
test_ prefix and locating the function, including any enclosing class) before
being emitted; names that cannot be verified are reported rather than guessed —
and mark_tests.py independently re-checks every node ID against the
filesystem, so a wrong ID can never silently mis-mark a test.

Also adds:

• a mark_oog_tests just recipe that runs detect + mark against
  pre-produced traces. The heavy traced fill is documented in the recipe
  header rather than run inline, so the fork and worker count stay in the
  reviewer's hands (traces are large and memory-hungry; a single fork suffices
  because OOG-by-design is fork-invariant in practice);
• scripts/fill_for_oog_detection.sh to produce the traces detached;
• oog_detector_validation_report.md recording the validation: 6 positive
  variants detected (top-level, sub-call depth 2–4, ooge/oogm, CREATE-init,
  multi-depth-in-one-tx), 5 non-OOG halts correctly ignored, a 1:1 match with
  ground truth read straight from the traces.

The detector is stdlib-only and does not import execution_testing, so it
is unaffected by the substring-match limitation in that package's
_is_out_of_gas_error helper (it tests for "out of gas", which never matches
the EELS class name OutOfGasError).

🔗 Related Issues or PRs

Depends on and must merge after #2906.

✅ Checklist

• All: Ran fast static checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  just static

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Considered updating the online docs in the ./docs/ directory.
• All: Set appropriate labels for the changes (only maintainers can apply labels).

Cute Animal Picture

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.49%. Comparing base (55d774b) to head (1ab1fd0).
⚠️ Report is 15 commits behind head on forks/amsterdam.

Additional details and impacted files

@@                 Coverage Diff                 @@
##           forks/amsterdam    #2941      +/-   ##
===================================================
+ Coverage            90.44%   90.49%   +0.05%     
===================================================
  Files                  535      535              
  Lines                32439    32430       -9     
  Branches              3012     3012              
===================================================
+ Hits                 29338    29349      +11     
+ Misses                2573     2563      -10     
+ Partials               528      518      -10     

Flag	Coverage Δ
unittests	90.49% <ø> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[leolara]

How this was validated

Filled on Osaka with --traces (one test per fill, single worker) and ran the detector over the resulting traces. Ground truth was read straight from each test's trace error field.

Positive — detected (the OOG variant each exercises):

Test (tests/ported_static/…)	Variant
stRevertTest/test_revert_prefound_oog	top-level OOG (memory expansion)
stTransactionTest/test_contract_store_clears_oog	top-level OOG (SSTORE)
stLogTests/test_log_in_oog_call	sub-call OOG, depth 2
stCallCodes/test_callcallcallcode_001_oogm_before	nested depth 3, out-of-gas-memory
stCallCodes/test_callcallcall_000_ooge	nested depth 4, out-of-gas-execution
stRevertTest/test_revert_depth_create_oog	CREATE-init; OOG at both depth 1 and 2 in one tx

Negative — correctly not flagged (each is a different all-gas-consuming halt or a non-OOG outcome):

Test (tests/ported_static/…)	Trace error
vmTests/test_block_info	(none — success)
stStackTests/test_stack_overflow	StackOverflowError
stSystemOperationsTest/test_create_with_invalid_opcode	StackUnderflowError
stRevertTest/test_revert_opcode_in_create_returns	Revert
stRevertTest/test_loop_calls_then_revert	(REVERT handled in-frame)

The detector flagged exactly the 6 tests whose traces contain OutOfGasError and none of the 5 others — a 1:1 match with ground truth.