fix(install): set DM_DISABLE_UDEV=1 to prevent dm semaphore deadlock in container IPC namespace

[andrewdunndev]

Summary

Defense-in-depth fix for the tpm2-luks install hang caused by libdevmapper udev cookie semaphore deadlock in the container's IPC namespace.

The primary fix is to run the install container with --ipc=host. This env var catches cases where IPC sharing is not configured.

Problem

bootc install to-disk --block-setup tpm2-luks hangs indefinitely at cryptsetup luksOpen. The root cause is a SysV semaphore deadlock between libdevmapper and udevd across the container's isolated IPC namespace.

libdevmapper creates SysV semaphores ("udev cookies") to synchronize device-mapper operations with udevd. The container's default IPC namespace is isolated from the host. udevd runs on the host and cannot see the container's semaphore. luksOpen blocks forever on semop().

Kernel stack trace of the hanging process:

[<0>] __do_semtimedop+0x3a8/0xd50
[<0>] do_semtimedop+0x15e/0x1a0
[<0>] do_syscall_64+0x7e/0x6b0

This affects luksOpen and luksClose inside bootc's container environment, regardless of TPM2 configuration.

Fix

Set DM_DISABLE_UDEV=1 in global_init() as defense-in-depth. The primary fix is --ipc=host on the container invocation, which shares the host IPC namespace and allows libdevmapper's udev cookie semaphores to reach udevd. bootc already propagates /run, so IPC sharing is a natural extension.

The env var is placed in global_init() alongside the existing HOME workaround, following the same pattern for process-wide environment setup early in main().

What this changes:

• dm udev cookie semaphores are not created, so no deadlock even without --ipc=host
• The kernel still creates /dev/mapper/root via devtmpfs
• udev rules for dm events don't fire inside the container, but bootc uses /dev/mapper/root directly (never udev symlinks)

What this does NOT change:

• Partition device nodes and udev_settle() are unaffected
• The installed system's boot-time behavior is unaffected
• BlockSetup::Direct path is unaffected

Testing

Three independent tests on Fedora 42 (systemd 257.11, cryptsetup 2.8.4, podman 5.8.0), plus a build verification of the exact patch:

Condition	Result
Stock bootc, default podman flags	HANG
Stock bootc + DM_DISABLE_UDEV=1	PASS
Stock bootc + --ipc=host	PASS
Patched bootc (this PR)	PASS

Both workarounds confirm the IPC namespace semaphore as root cause.

Fixes: #2089
Related: #421, #477

[gemini-code-assist]

Code Review

This pull request addresses a deadlock issue with cryptsetup in containerized environments by setting the DM_DISABLE_UDEV=1 environment variable. The fix is well-explained and seems correct. My main feedback is to suggest a safer, unsafe-free way to set the environment variable for the child processes, which would improve the robustness of the implementation.

[gemini-code-assist]

While using unsafe to set a process-wide environment variable is justified here with a good safety comment, a safer and more idiomatic approach would be to set the environment variable only for the specific child processes that need it. This avoids unsafe code entirely and more narrowly scopes the change.

You can use std::process::Command::env() to set an environment variable for a specific command. Since the Task helper exposes its Command struct as a public field cmd, you could modify the cryptsetup and systemd-cryptenroll calls to set DM_DISABLE_UDEV on their respective commands.

For example:

let mut task = Task::new("Initializing LUKS for root", "cryptsetup");
task.cmd.env("DM_DISABLE_UDEV", "1");
task.args(["luksFormat", ...]).run()?;

This would allow you to remove the disable_dm_udev_sync function and its unsafe block.

[andrewdunndev]

Considered this approach. The reason for the process-wide env var is that luksClose in install.rs (line ~2122, Task::new_and_run("Closing root LUKS device", "cryptsetup", ["close", luksdev])) also needs it, and that call is outside baseline.rs. The per-command approach would require changes in two files and refactoring the new_and_run convenience method to expose cmd.env(). Went with set_var to minimize the blast radius of the PR to one file. Happy to switch to per-command .env() if the maintainer prefers it.

[cgwalters]

I reopened and commented on #421 which is about the larger problem here - if we did that then there'd be no devmapper usage at all at installation time by bootc.

The container's default IPC namespace is isolated from the host.

I'd rather change that via changing our default install flow to use --ipc=host. We already propagate /run so this really makes sense.

As far as setenv - we already have global_init() that is the place for this.