Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .github/workflows/sovereign-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
# Change once here → applies to all 38 repos instantly.
#
# Self-hosted jobs run inside the sovereign-ci container (built by forjar).
# Each job gets an isolated filesystem — no shared ~/.rustup/, no race conditions.

Check warning on line 7 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

7:81 [line-length] line too long (82 > 80 characters)
# Image: localhost:5000/sovereign-ci:stable (local registry on mac-server)
# Rebuild: cd infra && make -f machines/intel/Makefile ci-image
#
Expand Down Expand Up @@ -48,7 +48,7 @@
default: false
type: boolean
extra_pkgs:
description: 'Extra apt packages to install in container (e.g. python3-dev libclang-dev)'

Check warning on line 51 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

51:81 [line-length] line too long (97 > 80 characters)
required: false
default: ''
type: string
Expand Down Expand Up @@ -78,7 +78,7 @@
jobs:
test:
name: test
runs-on: [self-hosted, clean-room]

Check failure on line 81 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

label "clean-room" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2025-vs2026", "windows-2022", "windows-11-arm", "ubuntu-slim", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "macos-latest", "macos-latest-xlarge", "macos-latest-large", "macos-26-intel", "macos-26-xlarge", "macos-26-large", "macos-26", "macos-15-intel", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xlarge", "macos-14-large", "macos-14", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
container:
image: localhost:5000/sovereign-ci:stable@sha256:10486da5daa3786f3264aa0e19fdde007e7ba1eca1d47ba587947946e42bd871
# Phase 3 §5.3 — sccache rustc cache + /var/log/ci-metrics for F9 stats.
Expand All @@ -94,7 +94,7 @@
persist-credentials: false
- name: Install extra packages
if: ${{ inputs.extra_pkgs != '' }}
run: |

Check failure on line 97 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2015:info:2:25: Note that A && B || C is not if-then-else. C may run when A is true
apt-get update -qq && apt-get install -y -qq ${{ inputs.extra_pkgs }} 2>/dev/null || \
sudo apt-get update -qq && sudo apt-get install -y -qq ${{ inputs.extra_pkgs }} 2>/dev/null || true
- name: Checkout sibling repos (path deps)
Expand Down Expand Up @@ -142,7 +142,7 @@
# Note: generated contract macros may have unused variables (provable-contracts#64).
# This is handled by adding -A unused-variables to the clippy step.
- name: Generate contract assertions (pv codegen)
run: |

Check failure on line 145 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2012:info:9:3: Use find instead of ls to better handle non-alphanumeric filenames
# pv is baked into sovereign-ci:stable at /usr/local/cargo/bin/pv
PV=""
for candidate in /usr/local/cargo/bin/pv /usr/local/bin/pv; do
Expand Down Expand Up @@ -192,7 +192,7 @@
RUSTC_WRAPPER: ${{ inputs.enable_sccache && 'rustc-sccache' || '' }}
SCCACHE_DIR: ${{ inputs.enable_sccache && '/sccache' || '' }}
USE_NEXTEST: ${{ inputs.use_nextest }}
run: |

Check failure on line 195 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:9:22: Double quote to prevent globbing and word splitting

Check failure on line 195 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:7:43: Double quote to prevent globbing and word splitting

Check failure on line 195 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:6:27: Double quote to prevent globbing and word splitting

Check failure on line 195 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:14:36: Double quote to prevent globbing and word splitting

Check failure on line 195 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:13:20: Double quote to prevent globbing and word splitting

Check failure on line 195 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

shellcheck reported issue in this script: SC2086:info:10:38: Double quote to prevent globbing and word splitting
# Mark workspace as safe for git operations inside tests (dubious ownership in containers)
git config --global --add safe.directory "$GITHUB_WORKSPACE"
# Phase 2 §4.3 — nextest drops ~35% off test-job wall-clock on large suites.
Expand All @@ -216,9 +216,23 @@
"/var/log/ci-metrics/sccache-${{ github.run_id }}-${{ inputs.repo }}-test.json" \
2>/dev/null || echo "::warning::sccache stats unavailable"

# FIVE-WHYS ROOT CAUSE (2026-04-20, paiml/infra#69):
# Container runs as root; sibling-clone dir $GITHUB_WORKSPACE/.. is
# bind-mounted from the runner host, so root-owned files leak out of
# the container onto the runner's _work tree. Subsequent non-container
# jobs (e.g. `security`) run as the runner user and can't `rm -rf`
# the stale clones → silent 15s failures.
# Fix: chown back to the runner uid/gid before the container exits.
# `always()` so the chown runs even if tests fail, preventing poison
# from surviving red builds.
- name: Restore runner ownership of sibling workspace
if: always()
run: |
chown -R 1000:1000 "$GITHUB_WORKSPACE/.." 2>/dev/null || true

lint:
name: lint
runs-on: [self-hosted, clean-room]

Check failure on line 235 in .github/workflows/sovereign-ci.yml

View workflow job for this annotation

GitHub Actions / validate

label "clean-room" is unknown. available labels are "windows-latest", "windows-latest-8-cores", "windows-2025", "windows-2025-vs2026", "windows-2022", "windows-11-arm", "ubuntu-slim", "ubuntu-latest", "ubuntu-latest-4-cores", "ubuntu-latest-8-cores", "ubuntu-latest-16-cores", "ubuntu-24.04", "ubuntu-24.04-arm", "ubuntu-22.04", "ubuntu-22.04-arm", "macos-latest", "macos-latest-xlarge", "macos-latest-large", "macos-26-intel", "macos-26-xlarge", "macos-26-large", "macos-26", "macos-15-intel", "macos-15-xlarge", "macos-15-large", "macos-15", "macos-14-xlarge", "macos-14-large", "macos-14", "self-hosted", "x64", "arm", "arm64", "linux", "macos", "windows". if it is a custom label for self-hosted runner, set list of labels in actionlint.yaml config file
container:
image: localhost:5000/sovereign-ci:stable@sha256:10486da5daa3786f3264aa0e19fdde007e7ba1eca1d47ba587947946e42bd871
# Phase 3 §5.3 — sccache rustc cache + /var/log/ci-metrics for F9 stats.
Expand Down Expand Up @@ -345,6 +359,10 @@
else
echo "::warning::No deny.toml — skipping supply chain audit"
fi
- name: Restore runner ownership of sibling workspace
if: always()
run: |
chown -R 1000:1000 "$GITHUB_WORKSPACE/.." 2>/dev/null || true

coverage:
name: coverage
Expand Down Expand Up @@ -471,6 +489,19 @@
with:
files: lcov.info
continue-on-error: true
# FIVE-WHYS ROOT CAUSE (2026-04-20, paiml/infra#69):
# Container runs as root; sibling-clone dir $GITHUB_WORKSPACE/.. is
# bind-mounted from the runner host, so root-owned files leak out of
# the container onto the runner's _work tree. Subsequent non-container
# jobs (e.g. `security`) run as the runner user and can't `rm -rf`
# the stale clones → silent 15s failures.
# Fix: chown back to the runner uid/gid before the container exits.
# `always()` so the chown runs even if tests fail, preventing poison
# from surviving red builds.
- name: Restore runner ownership of sibling workspace
if: always()
run: |
chown -R 1000:1000 "$GITHUB_WORKSPACE/.." 2>/dev/null || true

bench:
name: bench
Expand Down Expand Up @@ -601,6 +632,19 @@
path: bench-results.txt
retention-days: 90
continue-on-error: true
# FIVE-WHYS ROOT CAUSE (2026-04-20, paiml/infra#69):
# Container runs as root; sibling-clone dir $GITHUB_WORKSPACE/.. is
# bind-mounted from the runner host, so root-owned files leak out of
# the container onto the runner's _work tree. Subsequent non-container
# jobs (e.g. `security`) run as the runner user and can't `rm -rf`
# the stale clones → silent 15s failures.
# Fix: chown back to the runner uid/gid before the container exits.
# `always()` so the chown runs even if tests fail, preventing poison
# from surviving red builds.
- name: Restore runner ownership of sibling workspace
if: always()
run: |
chown -R 1000:1000 "$GITHUB_WORKSPACE/.." 2>/dev/null || true

security:
name: security
Expand All @@ -619,6 +663,16 @@
run: |
apt-get update -qq && apt-get install -y -qq ${{ inputs.extra_pkgs }} 2>/dev/null || \
sudo apt-get update -qq && sudo apt-get install -y -qq ${{ inputs.extra_pkgs }} 2>/dev/null || true
# FIVE-WHYS ROOT CAUSE RECOVERY (2026-04-20, paiml/infra#69):
# Container jobs upstream may have left root-owned files in the
# sibling workspace (bind-mount leak). Every container job now
# chowns back on exit, but for defense in depth — and to recover
# from runs that predated the fix — reclaim ownership before we
# touch the sibling tree. Without this, `rm -rf` on stale clones
# fails with EACCES and the job dies in 15s.
- name: Reclaim sibling workspace ownership (defense in depth)
run: |
sudo chown -R "$(id -u):$(id -g)" "$GITHUB_WORKSPACE/.." 2>/dev/null || true
- name: Checkout sibling repos (path deps)
run: |
cd "$GITHUB_WORKSPACE/.."
Expand Down
Loading