Print worker crash output

[christiangnrd]

In Metal.jl, when working on wrappers to the underlying libraries, when invalid inputs are provided, the worker crashes and no error output is displayed. These crash messages usually have a useful error message, forcing me to rerun the test outside of the test environment.

[giordano]

I looked into this because I'm also interested in seeing the output of crashed workers, but this is looking complicated.

For successful jobs:

ParallelTestRunner.jl/src/ParallelTestRunner.jl

Lines 251 to 284 in 608f947

	data = @eval mod begin
	GC.gc(true)
	Random.seed!(1)
	pipe = Pipe()
	pipe_initialized = Channel{Nothing}(1)
	reader = @async begin
	take!(pipe_initialized)
	read(pipe, String)
	end
	io = IOContext(pipe, :color=>$(color))
	stats = redirect_stdio(; stdout=io, stderr=io) do
	put!(pipe_initialized, nothing)
	# @testset CustomTestRecord switches the all lower-level testset to our custom testset,
	# so we need to have two layers here such that the user-defined testsets are using `DefaultTestSet`.
	# This also guarantees our invariant about `WorkerTestSet` containing a single `DefaultTestSet`.
	@timed @testset WorkerTestSet "placeholder" begin
	@testset DefaultTestSet $name begin
	$f
	end
	end
	end
	close(pipe.in)
	output = fetch(reader)
	(; testset=stats.value, output, stats.time, stats.bytes, stats.gctime)
	end
	# process results
	rss = Sys.maxrss()
	record = TestRecord(data..., rss)
	GC.gc(true)
	return record

we capture the output, store the result in a TestRecord and return it, and this is then fetched by the orchestrator:

ParallelTestRunner.jl/src/ParallelTestRunner.jl

Lines 978 to 980 in 608f947

	result = try
	Malt.remote_eval_wait(Main, wrkr, :(import ParallelTestRunner))
	Malt.remote_call_fetch(invokelatest, wrkr, runtest,

But for crashed jobs we don't have a chance to record the output anywhere, because the worker itself died. Unless we can get the stdout/stderr from the Malt.Worker object in the orchestrator process?

[christiangnrd]

Didn’t mean to close my finger fell while I was reading

[giordano]

Opened upstream issue at JuliaPluto/Malt.jl#102.