Skip to content
Test Linux

test_runner: support expecting a test-case to fail #78726

Sign in to view logs

test_runner: support expecting a test-case to fail

test_runner: support expecting a test-case to fail #78726

Triggered via pull request December 7, 2025 19:26
@JakobJingleheimerJakobJingleheimer
ready_for_review #60669
JakobJingleheimer:test_runner/feat/should-fail
Status Failure
Total duration 2h 35m 40s
Artifacts –

test-linux.yml

on: pull_request
Matrix: test-linux
Fit to window
Zoom out
Zoom in

Annotations

4 errors and 2 notices
test-linux (ubuntu-24.04)
Process completed with exit code 2.
test-linux (ubuntu-24.04): test/test-runner/test-output-describe-it.mjs#L0
--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected ... Skipped lines 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync pass todo\n' + + 'ok 5 - sync pass todo # TODO\n' + - 'ok 1 - sync pass todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass todo with message\n' + + 'ok 6 - sync pass todo with message # TODO this is a passing todo\n' + - 'ok 2 - sync pass todo with message # TODO this is a passing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync todo\n' + + 'not ok 7 - sync todo # TODO\n' + - 'not ok 3 - sync todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):4'\n" + " failureType: 'testCodeFailure'\n" + ... ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 8 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync todo with message\n' + + 'not ok 9 - sync todo with message # TODO this is a failing todo\n' + - 'not ok 4 - sync todo with message # TODO this is a failing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: sync skip pass\n' + + 'ok 10 - sync skip pass # SKIP\n' + - 'ok 5 - sync skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 11 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + - 'ok 6 - sync skip pass with message # SKIP this is skipped\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + - 'ok 7 - sync pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + - 'not ok 8 - sync throw fail\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + - 'ok 9 - async skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + - 'ok 10 - async pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: mixing describe/it and test should work\n' + + 'ok 17 - mixing describe/it and test should work\n' + - 'ok 11 - mixing describe/it and test should work\n' + ' ---\n' + ' duration_ms: *\n' + "
test-linux (ubuntu-24.04-arm)
Process completed with exit code 2.
test-linux (ubuntu-24.04-arm): test/test-runner/test-output-describe-it.mjs#L0
--- stderr --- node:internal/modules/run_main:107 triggerUncaughtException( ^ AssertionError [ERR_ASSERTION]: Expected values to be strictly equal: + actual - expected ... Skipped lines 'TAP version 13\n' + + '# Subtest: sync expect fail (method)\n' + + 'ok 1 - sync expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: sync expect fail (options)\n' + + 'ok 2 - sync expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (method)\n' + + 'ok 3 - async expect fail (method) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + + '# Subtest: async expect fail (options)\n' + + 'ok 4 - async expect fail (options) # EXPECTED FAILURE\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync pass todo\n' + + 'ok 5 - sync pass todo # TODO\n' + - 'ok 1 - sync pass todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass todo with message\n' + + 'ok 6 - sync pass todo with message # TODO this is a passing todo\n' + - 'ok 2 - sync pass todo with message # TODO this is a passing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync todo\n' + + 'not ok 7 - sync todo # TODO\n' + - 'not ok 3 - sync todo # TODO\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):4'\n" + " failureType: 'testCodeFailure'\n" + ... ' ...\n' + + '# Subtest: sync todo with expect fail\n' + + 'ok 8 - sync todo with expect fail # TODO\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync todo with message\n' + + 'not ok 9 - sync todo with message # TODO this is a failing todo\n' + - 'not ok 4 - sync todo with message # TODO this is a failing todo\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: sync skip pass\n' + + 'ok 10 - sync skip pass # SKIP\n' + - 'ok 5 - sync skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + + '# Subtest: sync skip expect fail\n' + + 'ok 11 - sync skip expect fail # SKIP\n' + + ' ---\n' + + ' duration_ms: *\n' + + " type: 'test'\n" + + ' ...\n' + '# Subtest: sync skip pass with message\n' + + 'ok 12 - sync skip pass with message # SKIP this is skipped\n' + - 'ok 6 - sync skip pass with message # SKIP this is skipped\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync pass\n' + + 'ok 13 - sync pass\n' + - 'ok 7 - sync pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: sync throw fail\n' + + 'not ok 14 - sync throw fail\n' + - 'not ok 8 - sync throw fail\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + " location: '/test/fixtures/test-runner/output/describe_it.js:(LINE):1'\n" + " failureType: 'testCodeFailure'\n" + ... '# Subtest: async skip pass\n' + + 'ok 15 - async skip pass # SKIP\n' + - 'ok 9 - async skip pass # SKIP\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: async pass\n' + + 'ok 16 - async pass\n' + - 'ok 10 - async pass\n' + ' ---\n' + ' duration_ms: *\n' + " type: 'test'\n" + ' ...\n' + '# Subtest: mixing describe/it and test should work\n' + + 'ok 17 - mixing describe/it and test should work\n' + - 'ok 11 - mixing describe/it and test should work\n' + ' ---\n' + ' duration_ms: *\n' + "
sccache stats
0% - 0 hits, 0 misses, 0 errors
sccache stats
0% - 0 hits, 0 misses, 0 errors