ci: pin rust-toolchain.toml to nightly-2026-04-28 (bd-at72)#177
Merged
Conversation
The TS Test Suite workflow has been failing on every push to main since 2026-05-10 with SIGSEGV in rustc nightly during the wasm32-unknown-unknown release build (opt-level=3). Two attempts to widen RUST_MIN_STACK (16 MB, then 32 MB) only chased the crash to deeper crates -- rustc's suggested value escalated to 64 MB and the crash moved between quarto-pandoc-types, quarto-ast-reconcile, and tokio. Not a fixed-size stack deficit. Bisection via dated rustup nightlies, run locally on macOS aarch64: - nightly-2026-04-28 (commit 52b6e2c20, 2026-04-27) -- wasm32 build completes cleanly in 38s - nightly-2026-05-11 (commit 4b0c9d76a, 2026-05-10) -- reliable SIGSEGV in LLVM ThinLTO codegen, matches CI exactly CI was running nightly-2026-05-11 because rustup pulls latest nightly on every run; the previous green CI dates (last on 2026-05-08) pulled a build from before the regression landed. The bug appears to be in LLVM's ThinLTO/optimization passes (backtrace through FPPassManager::runOnModule and LLVMRustWriteOutputFile), not in rustc Rust-side code. Pin nightly to 2026-04-28 -- the date that matches the cached toolchain the team is already running locally -- until upstream stabilizes nightly again. Bump the date when the regression is resolved. Drops the earlier RUST_MIN_STACK env-var changes from the workflow files (they didn't help, and a single mechanism is cleaner than belt-and-suspenders).
cscheid
force-pushed
the
bugfix/ci-rust-min-stack
branch
from
3c35c1a to
368e2cc
Compare
cscheid
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
TS Test Suiteworkflow has been failing on every push tomainsince 2026-05-10 with SIGSEGV inrustcnightly during thewasm32-unknown-unknownrelease build. The first iteration of this PR tried twoRUST_MIN_STACKbumps (16 MB, then 32 MB) — both worked exactly long enough to expose a deeper crash, withrustc's own hint escalating to 64 MB and the SIGSEGV moving betweenquarto-pandoc-types,quarto-ast-reconcile, andtokio. That's not a fixed-size stack deficit; it's a regression in somerustc/ LLVM pass.To isolate the trigger, I installed two dated nightlies side-by-side (
nightly-2026-04-28andnightly-2026-05-11) and builtwasm-quarto-hub-clientagainst each.nightly-2026-04-28(rustc commit52b6e2c20, 2026-04-27) builds cleanly in 38 seconds.nightly-2026-05-11(commit4b0c9d76a, 2026-05-10) SIGSEGVs reliably with the same backtrace shape CI sees — throughFPPassManager::runOnModuleandLLVMRustWriteOutputFile, i.e. inside LLVM optimization codegen. CI was pulling the broken nightly becauserustupfetches the latest unfdated nightly on every clean run; local environments don't reproduce because their cached nightly predates 2026-05-10.This PR pins
rust-toolchain.tomltonightly-2026-04-28— the date matches what the team already has cached locally, so it should be byte-for-byte the samerustceveryone is currently running. The earlierRUST_MIN_STACKenv-var changes in the workflow files have been dropped; the pin replaces them entirely. We'll bump the pinned date once upstream resolves the regression (tracked in bd-at72).Test plan
nightly-2026-05-11— confirmed SIGSEGVcargo xtask verify(full) with the pinnednightly-2026-04-28— passesCloses bd-at72.