Skip to content

ProcessWriter doesn't forward child stdout/stderr to container stdout #3

@sylvesterdamgaard

Description

@sylvesterdamgaard

Summary

Child process output (nginx access lines, php-fpm master messages, queue worker logs, scheduler runs) is visible in the cbox-init TUI's live log view but never appears in the container's own stdout/stderr — so docker logs / kubectl logs only ever show cbox-init's own JSON startup messages, even though the supervisor is correctly capturing the child streams.

Confirmed in v2.1.1 running inside ghcr.io/cboxdk/php-baseimages/php-fpm-nginx:8.5-bookworm with the default cbox-init.yaml from the baseimage.

What we see

  • TUI (cbox-init logs --process scheduler etc.): full log stream from every process, with level + stream + timestamp annotations.
  • kubectl logs <pod> after a fresh start with traffic + scheduled jobs: 57 lines total, every one of them cbox-init's own startup output ("Cbox Init starting" → "All processes started successfully"). Zero lines from nginx, php-fpm, or any child process.

Reproduction

# Pod's php-fpm-nginx baseimage already has nginx logs symlinked to
# /dev/stdout + /dev/stderr (verified via `ls -la /var/log/nginx/`).
# Same pod, write to PID 1 (cbox-init) directly:
$ kubectl exec <pod> -- sh -c 'echo HELLO-FROM-PID1 >>/proc/1/fd/1'
$ kubectl logs <pod> --since=10s
HELLO-FROM-PID1                          # ✓ reaches container stdout

# Now write to nginx master's fd 1 (which is the pipe whose read end
# is held by cbox-init):
$ kubectl exec <pod> -- bash -c '
    NGX=$(pgrep -f "nginx: master" | head -1);
    echo HELLO-FROM-NGX-PIPE >>/proc/$NGX/fd/1'
$ kubectl logs <pod> --since=10s     # silent

# But the same line shows up in the TUI:
$ kubectl exec -ti <pod> -- cbox-init logs nginx
[INFO] [stdout] [nginx/nginx-0] HELLO-FROM-NGX-PIPE

Diagnosis

The pipes are wired correctly. From inside the running container:

=== nginx master:
  fd 1 -> pipe:[3894742]    # nginx writes its access_log here (via /dev/stdout)
  fd 2 -> pipe:[3894743]    # nginx writes its error_log here (via /dev/stderr)

=== cbox-init (PID 1) read pipes:
  fd 9  -> pipe:[3894742]   # ✓ reads nginx stdout
  fd 12 -> pipe:[3894743]   # ✓ reads nginx stderr
  fd 3  -> pipe:[3892199]   # ✓ reads fpm master stderr

So Supervisor.startCommand does set cmd.Stdout = stdoutWriter (a *logger.ProcessWriter) and the OS pipe is established. ProcessWriter.processEntry is parsing the lines (TUI proves it) and storing them in the per-process logBuffer — that's the path the TUI consumes.

The line that's also supposed to forward the entry to cbox-init's slog handler — pw.Logger.<Level>(message, baseAttrs...) — apparently doesn't reach the root handler (the one with os.Stdout) for these entries. Either the logger that ProcessWriter is given is scoped to a handler that only feeds the TUI broadcaster/buffer, or the per-process / per-stream level routing happens to filter everything out by default.

Configurable behaviour, not necessarily a bug

There's an existing per-process / per-stream log-level filter (operators can pick which levels surface to container stdout vs stay in the TUI buffer). The current default config from the baseimage produces a 100%-silent kubectl logs view, which is the surprising part — operators don't get a hint that the levels need tuning.

So this issue is asking for one or both of:

  1. Better default: have at least INFO-and-above from each process forwarded to container stdout out of the box. The TUI is great for an operator already attached, but the container's own logs are what every other tool (kubectl, Loki, datadog agent) reads, and they shouldn't be empty.
  2. Documented escape hatch: make the knob that controls "TUI-only vs also-stdout" obvious in the example cbox-init.yaml, with a comment that explains why someone running this in k8s probably wants to flip it.

Bonus: surface a one-time WARN at startup if every process is configured to suppress its output from container stdout — would have caught this in seconds.

Environment

  • cbox-init: v2.1.1
  • Baseimage: ghcr.io/cboxdk/php-baseimages/php-fpm-nginx:8.5-bookworm (post fix(setup): respect PUID/PGID + look up www-data from /etc/passwd #2 — nginx logs symlinked correctly)
  • Runtime: k3s 1.35.4 on Hetzner Cloud (cx33), containerd
  • Default cbox-init.yaml from baseimage (php-fpm + nginx + queue-default + scheduler all with stdout: true, stderr: true)

Impact

Operators running cbox-init in k8s / Compose / any orchestrator that scrapes container stdout get an empty log stream by default. No visibility into traffic, framework errors, queue failures, or upstream timeouts. The TUI is a great operator tool but it's not a substitute for a log shipper picking up container stdout.

Workaround until defaults change

Configure nginx + php-fpm to write directly to /proc/1/fd/1 (cbox-init's own stdout pipe) instead of /dev/stdout. Bypasses the supervisor's per-process routing. Ugly, but it gets logs to kubectl logs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions