sysbox v0.7.0 doesn't work on Ubuntu 24.04 + kernel 6.8 + containerd v2 — install + mount failures

[kkroo]

Summary

Tried to install sysbox v0.7.0 on a fresh k8s node — Ubuntu 24.04 (noble), kernel 6.8.0-111-generic, containerd 2.2.1, kubelet 1.35.3. Two independent gaps prevent it from working end-to-end:

1. sysbox-deploy-k8s installer writes the containerd runtime stanza under the v1 plugin namespace, which containerd 2.x ignores.
2. sysbox-runc fails to mount sysfs in the pod sandbox at sandbox creation time with mount through procfd: operation not permitted, even after manually fixing Ct doc #1.

Both look like the same general theme: sysbox v0.7.0 was tested against pre-2026 stacks (containerd 1.x + kernel ≤ 6.2) and hasn't been bumped to track newer host environments.

Environment

Layer	Version
OS	Ubuntu 24.04 LTS (noble)
Kernel	6.8.0-111-generic (amd64)
containerd	2.2.1
kubelet	1.35.3
sysbox	v0.7.0-0 (registry.nestybox.com/nestybox/sysbox-deploy-k8s:v0.7.0-0)
AppArmor	enabled (kernel.apparmor_restrict_unprivileged_userns=1 is the Ubuntu 24.04 default)
shiftfs	not available on this kernel (above shiftfs_max_kernel_ver=6.2) — should fall back to idmapped mounts
CONFIG_USER_NS	y

Gap #1: containerd v2 plugin namespace mismatch

/opt/sysbox/scripts/sysbox-deploy-k8s.sh (function config_containerd_for_sysbox) writes the sysbox-runc runtime stanza using dasel put under TOML path:

plugins.io\.containerd\.grpc\.v1\.cri.containerd.runtimes.sysbox-runc.*

This is the containerd v1 plugin namespace. Containerd 2.x uses a different namespace:

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.sysbox-runc]
  runtime_type = 'io.containerd.runc.v2'
  [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.sysbox-runc.options]
    BinaryName = '/usr/bin/sysbox-runc'
    SystemdCgroup = true

Our cluster's /etc/containerd/config.toml only has v2-namespace entries — no v1 backward-compat. So after the installer runs:

• systemctl is-active containerd → active
• crictl info shows sysbox-runc is not registered
• Pods with runtimeClassName: sysbox-runc fail with runtime not registered

Workaround: hand-edit the config to write the stanza under the v2 namespace. With that, crictl info | jq '.config.containerd.runtimes."sysbox-runc"' returns the correct binary and the runtime is registered.

Fix suggested: detect containerd version at install time (containerd --version or check existing config namespace) and write under the v1 OR v2 path accordingly. Both namespaces are valid in different containerd versions.

Gap #2: sandbox creation fails with mount through procfd: operation not permitted

After applying the workaround for #1, a minimal sysbox pod still fails at sandbox creation:

apiVersion: v1
kind: Pod
spec:
  runtimeClassName: sysbox-runc
  containers:
  - name: x
    image: alpine:3.20
    command: [sh, -c, "sleep 60"]

containerd error:

RunPodSandbox failed:
OCI runtime create failed: container_linux.go:439: starting container process caused:
process_linux.go:608: container init caused:
rootfs_linux.go:76: setting up rootfs mounts caused:
rootfs_linux.go:1276: mounting "sysfs" to rootfs "/run/.../rootfs" at "sys" caused:
mount through procfd: operation not permitted

What I tried

1. AppArmor sysctls — set both kernel.apparmor_restrict_unprivileged_userns=0 and kernel.unprivileged_userns_apparmor_policy=0. No change.
2. AppArmor profile — patched /etc/apparmor.d/unprivileged_userns to add allow mount, and allow umount, rules, then apparmor_parser -r. No change.
3. Verified AppArmor isn't the cause — containerd-shim-runc-v2 runs unconfined, so its child runc processes inherit unconfined. They never transition into the unprivileged_userns profile. sudo dmesg shows zero AppArmor DENIED entries during the failure (only unrelated otelcol-contrib ptrace denials).

The EPERM is coming from the kernel mount syscall directly, not from AppArmor.

Likely root cause

sysbox-runc v0.7.0 reports OCI specs: 1.1.0+dev, meaning the runc fork is based on runc 1.1.x. There are known fixes in upstream runc 1.2.x for kernel 6.x's tightened procfd-based mount semantics (see opencontainers/runc#4435 and the linked kernel patches around move_mount / open_tree enforcement). The procfd path in sysbox-runc predates those fixes.

This is not fixable from the host config as far as I can tell.

Asks

1. Bump sysbox-runc's runc fork base to a runc release that handles kernel 6.x procfd mount semantics (runc 1.2+).
2. Update sysbox-deploy-k8s.sh to detect containerd v1 vs v2 and write the runtime stanza under the right plugin namespace.
3. Document the minimum supported runc / containerd / kernel matrix more explicitly. The current release notes don't make it obvious that v0.7.0 won't work on Ubuntu 24.04's kernel 6.8.

Happy to share the full sysbox-mgr/sysbox-fs/containerd logs from the failed install attempt if helpful.