fix(sovereign-ci): chown sibling workspace back from root on container exit#28
Merged
Merged
Conversation
…r exit Root cause (Five Whys — paiml/infra#69): 1. Security job fails in 15s with rm: EACCES on aprender/**/.git and aprender-present-widgets/src/*.rs in sibling workspace. 2. Files are owned by root:root; the security job runs as the runner user (uid 1000) and can't rewrite them. 3. Root-owned files come from the test/lint/coverage/bench jobs — those run inside a container whose process uid is 0. 4. The container mounts $GITHUB_WORKSPACE (the runner's _work tree) and the sibling checkout writes to $GITHUB_WORKSPACE/.., which is also bind-mounted via the host filesystem. Files written by root inside the container land on the host as root-owned. 5. Subsequent non-container jobs (security) — and every *future* run of the container jobs themselves, which start by `rm -rf`'ing the stale clones — cannot reclaim the tree. Fix: every container job chowns $GITHUB_WORKSPACE/.. back to uid 1000 in an `if: always()` tail step, so root-owned files never escape the container. Security job gets a defense-in-depth `sudo chown` at the top to recover from any pre-fix residue. Manually chowned 83838 files across the 16 runners on intel to unblock existing PRs; this patch prevents the poison from re-accumulating. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
\$GITHUB_WORKSPACE/..back to uid 1000 on exit (if: always()), preventing root-owned files from leaking onto the runner host via bind mounts.sudo chownpre-step to recover from pre-fix residue.Root cause (Five Whys — paiml/infra#69)
rm: cannot remove 'aprender/.../*.rs': Permission denied.root:root; security runs as the runner user and can't rewrite them.cd \$GITHUB_WORKSPACE/..) writes into a bind-mounted host dir, so root ownership lands on the runner host, not just the container.rm -rfcleanup — can't reclaim the tree. Result: silent 15s failures that recur on every PR.Scope
Reusable workflow used by ~38 repos (course-studio, rmedia, aprender, bashrs, …). After merge, every job run gets the cleanup automatically; no per-repo change needed.
Immediate unblock
Before this patch lands, manually chowned 83,838 root-owned files across the 16 runners on
intelto unblock existing PRs. This patch prevents re-accumulation.Test plan
/home/noah/data/actions-runner*/_work/🤖 Generated with Claude Code