feat: application sidekicks = non-HTTP workers with shared state

[nicolas-grekas]

Add support for "sidekick" workers: long-running PHP scripts that run outside the HTTP request cycle, observe their environment, and publish configuration to HTTP workers in real time.

This enables patterns like Redis Sentinel discovery, secret rotation, feature flag streaming, and cache invalidation — without polling, TTLs, or redeployment.

New PHP functions

• frankenphp_sidekick_get_vars(string|array $name, float $timeout = 30.0): array
  Starts a sidekick and returns its published variables. The first call blocks until the sidekick calls set_vars() or the timeout expires. Subsequent calls return the latest snapshot immediately. When given an array of names, all sidekicks are started in parallel and vars are returned keyed by name. Works in both worker and non-worker mode.

• frankenphp_sidekick_set_vars(array $vars): void
  Publishes a snapshot of variables from inside a sidekick script. All keys and values must be strings. Each call replaces the entire snapshot atomically. Can only be called from a sidekick context.

• frankenphp_sidekick_should_stop(): bool
  Cooperative shutdown check. Sidekick scripts poll this in their event loop to exit gracefully when FrankenPHP shuts down. Can only be called from a sidekick context.

Caddyfile configuration

example.com {
    php_server {
        sidekick_entrypoint /app/bin/console
    }
}

How it works

// Sidekick entrypoint (e.g. bin/console)
$command = $_SERVER['argv'][1] ?? '';  // or $_SERVER['FRANKENPHP_SIDEKICK_NAME']

match ($command) {
    'redis-watcher' => runRedisWatcher(),
    default => throw new \RuntimeException("Unknown sidekick: $command"),
};

function runRedisWatcher(): void {
    frankenphp_sidekick_set_vars([
        'MASTER_HOST' => '10.0.0.1',
        'MASTER_PORT' => '6379',
    ]);
    while (!frankenphp_sidekick_should_stop()) {
        $master = discoverRedisMaster();
        frankenphp_sidekick_set_vars([
            'MASTER_HOST' => $master['host'],
            'MASTER_PORT' => (string) $master['port'],
        ]);
        usleep(100_000);
    }
}

// HTTP worker or regular php_server script
frankenphp_handle_request(function () {
    $redis = frankenphp_sidekick_get_vars('redis-watcher');
    $host = $redis['MASTER_HOST']; // always up to date
});

Design highlights

• No race condition on startup: get_vars blocks until the sidekick has published its initial state
• Atomic snapshots: set_vars replaces all vars at once — no partial state
• Explicit API: caller gets a plain array, no implicit $_SERVER injection
• Strict context enforcement: set_vars and should_stop throw if not called from a sidekick
• At-most-once start: safe to call get_vars from multiple HTTP workers — only one starts the sidekick
• Parallel start: get_vars(['a', 'b']) starts all sidekicks concurrently
• Per-php_server scoping: each php_server block has its own SidekickRegistry — different apps on the same Caddy instance are fully isolated
• Crash recovery: sidekicks are restarted automatically on crash (existing worker restart logic)
• Graceful degradation: function_exists('frankenphp_sidekick_get_vars') lets the same code work with or without FrankenPHP
• Works in both worker and non-worker mode: get_vars works from any PHP script served by php_server
• bin/console compatible: sidekick name is available as $_SERVER['argv'][1] and $_SERVER['FRANKENPHP_SIDEKICK_NAME']
• Binary safe: values can contain null bytes, UTF-8, etc.

Runtime behavior

• Sidekick threads skip HTTP request startup/shutdown
• SCRIPT_FILENAME is set correctly for non-.php entrypoints
• Execution timeout is automatically disabled
• Shebangs (#!/usr/bin/env php) are silently skipped

[AlliBalliBaba]

Interesting approach to parallelism, what would be a concrete use case for only letting information flow one way from the sidekick to the http workers?

Usually the flow would be inverted, where a http worker offloads work to a pool of 'sidekick' workers and can optionally wait for a task to complete.

[henderkes]

Thank you for the contribution. Interesting idea, but I'm thinking we should merge the approach with #1883. The kind of worker is the same, how they are started is but a detail.

@nicolas-grekas the Caddyfile setting should likely be per php_server, not a global setting.

[nicolas-grekas]

@AlliBalliBaba The use case isn't task offloading (HTTP->worker), but out-of-band reconfigurability (environment->worker->HTTP). Sidekicks observe external systems (Redis Sentinel failover, secret rotation, feature flag changes, etc.) and publish updated configuration that HTTP workers pick up on their next request; with per-request consistency guaranteed via $_SERVER injection. No polling, no TTLs, no redeployment.

Task offloading (what you describe) is a valid and complementary pattern, but it solves a different problem. The non-HTTP worker foundation here could support both.

@henderkes Agreed that the underlying non-HTTP worker type overlaps with #1883. The foundation (skip HTTP startup/shutdown, immediate readiness, cooperative shutdown) is the same. The difference is the API layer and the DX goals:

• Minimal FrankenPHP config: a single sidekick_entrypoint in php_server(thanks for the idea). No need to declare individual workers in the Caddyfile. The PHP app controls which sidekicks to start via frankenphp_sidekick_start(), keeping the infrastructure config simple.

• Graceful degradability: apps should work correctly with or without FrankenPHP. The same codebase should work on FrankenPHP (with real-time reconfiguration) and on traditional setups (with static or always refreshed config).

• Nice framework integration: the sidekick_entrypoint pointing to e.g. bin/console means sidekicks are regular framework commands, making them easy to develop.

Happy to follow up with your proposals now that this is hopefully clarified.
I'm going to continue on my own a bit also :)

[dunglas]

Great PR!

Couldn't we create a single API that covers both use case?

We try to keep the number of public symbols and config option as small as possible!

[henderkes]

@henderkes Agreed that the underlying non-HTTP worker type overlaps with #1883. The foundation (skip HTTP startup/shutdown, immediate readiness, cooperative shutdown) is the same. The difference is the API layer and the DX goals:

Yes, that's why I'd like to unify the two API's and background implementations into one. Unfortunately the first task worker attempt didn't make it into main, but perhaps @AlliBalliBaba can use his experience with the previous PR to influence this one. I'd be more in favour of a general API, than a specific sidecar one.

[nicolas-grekas]

The PHP-side API has been significantly reworked since the initial iteration: I replaced $_SERVER injection with explicit get_vars/set_vars protocol.

The old design used frankenphp_set_server_var() to inject values into $_SERVER implicitly. The new design uses an explicit request/response model:

• frankenphp_sidekick_set_vars(array $vars): called from the sidekick to publish a complete snapshot atomically
• frankenphp_sidekick_get_vars(string|array $name, float $timeout = 30.0): array: called from HTTP workers to read the latest vars

Key improvements:

• No race condition on startup: get_vars blocks until the sidekick has called set_vars. The old design had a race where HTTP requests could arrive before the sidekick had published its values.
• Strict context enforcement: set_vars and should_stop throw RuntimeException if called from a non-sidekick context.
• Atomic snapshots: set_vars replaces all vars at once. No partial state possible
• Parallel start: get_vars(['redis-watcher', 'feature-flags']) starts all sidekicks concurrently, waits for all, returns vars keyed by name.
• Works in both worker and non-worker mode: get_vars works from any PHP script served by php_server, not just from frankenphp_handle_request() workers.

Other changes:

• sidekick_entrypoint moved from global frankenphp block to per-php_server (as @henderkes suggested)
• Removed the $argv parameter: the sidekick name is the command, passed as $_SERVER['argv'][1]
• set_vars is restricted to sidekick context only (throws if called from HTTP workers)
• get_vars accepts string|array: when given an array, all sidekicks start in parallel
• Atomic snapshots: set_vars replaces all vars at once, no partial state
• Binary-safe values (null bytes, UTF-8)

[nicolas-grekas]

Thanks @dunglas and @henderkes for the feedback. I share the goal of keeping the API surface minimal.

Thinking about it more, the current API is actually quite small and already general:

• 1 Caddyfile setting: sidekick_entrypoint (per php_server)
• 3 PHP functions: get_vars, set_vars, should_stop

The name "sidekick" works as a generic concept: a helper running alongside. The current set_vars/get_vars protocol covers the config-publishing use case. For task offloading (HTTP->worker) later, the same sidekick infrastructure could support:

• frankenphp_sidekick_send_task(string $name, mixed $payload): mixed
• frankenphp_sidekick_receive_task(): mixed

Same worker type, same sidekick_entrypoint, same should_stop(). Just a different communication pattern added on top. No new config, no new worker type.

So the path would be:

1. This PR: sidekicks with set_vars/get_vars (config publishing)
2. Future PR: add send_task/receive_task (task offloading), reusing the same non-HTTP worker foundation

The foundation (non-HTTP threads, cooperative shutdown, crash recovery, per-php_server scoping) is shared. Only the communication primitives differ.

WDYT?

[nicolas-grekas]

I think the failures are unrelated - a cache reset would be needed. Any help on this topic?

[alexandre-daubois]

Hmm, it seems they are on some versions, for example here: https://github.com/php/frankenphp/actions/runs/23192689128/job/67392820942?pr=2287#step:10:3614

For the cache, I'm not aware of a Github feature that allow to clear everything unfortunately 🙁

[henderkes]

The name "sidekick" works as a generic concept: a helper running alongside. The current set_vars/get_vars protocol covers the config-publishing use case. For task offloading (HTTP->worker) later, the same sidekick infrastructure could support:

My only worry with this is that "sidekick" implies that there's a "main" character related to it. That's the case here, but wouldn't necessarily be the case for task- or extension workers.
Think of something like
task_worker bin/console messenger:consume scheduler_default async -vv

Other than the naming, I don't object the api.

[AlliBalliBaba]

If we want to unify these concepts, the sidekick workers should probably be configured like regular workers and started always from the Caddy config (very messy to let http workers start and stop them). Just a worker instead of sidekick_entrypoint and mark the worker as non_http somehow.

The frankenphp_sidekick_set_vars and frankenphp_sidekick_get_vars are essentially just a process-wide store, you could do the same with something like apcu. I think having global variables for ZTS could be very useful, but it should be well thought through since there many potential use cases like caches or locks. You probably also need ways to flush and debug the store, like apcu offers. Might make sense to have something like a FrankenPHP\Store or so and support more than just strings.

[henderkes]

If we want to unify these concepts, the sidekick workers should probably be configured like regular workers and started always from the Caddy config (very messy to let http workers start and stop them). Just a worker instead of sidekick_entrypoint and mark the worker as non_http somehow.

Absolute agree, though I believe it would be good to be able to start them from http code and establish communication through channels of sort.

The frankenphp_sidekick_set_vars and frankenphp_sidekick_get_vars are essentially just a process-wide store, you could do the same with something like apcu. I think having global variables for ZTS could be very useful, but it should be well thought through since there many potential use cases like caches or locks. You probably also need ways to flush and debug the store, like apcu offers. Might make sense to have something like a FrankenPHP\Store or so and support more than just strings.

Funny you say that, I created one over the last few days, but I quickly figured that the Cgo overhead and Go generally being much slower than optimised C made this a bit of a futile attempt. Ristretto backend was ~4x slower than apcu/direct C copying. If we want to implement a FrankenPHP\Store it will have to be well-thought out and implemented in pure C.

[nicolas-grekas]

All green, ready on my side!🎳

@henderkes Thanks for validating the CGo overhead concern. A proper FrankenPHP\Store would need pure C and is a separate, much larger project. The sidekick API is intentionally narrow: lifecycle management + config publishing.

Glad we agree on starting from PHP! That's the current design: get_vars implicitly starts the sidekick on first call.

@AlliBalliBaba About APCu: I want sidekicks to be a core feature of FrankenPHP that people can reliably build on, not something depending on an optional third-party extension with its own bugs and serialization overhead. The sidekick API solves a different problem than a shared store: lifecycle management (start, wait-for-ready, crash recovery, shutdown), blocking first call (no polling), and per-sidekick scoping. String-only is by design; config updates don't need complex types, and explicit serialization is always available.

"very messy to let http workers start and stop them"

It's actually clean. Every HTTP worker has a bootstrap phase before entering the frankenphp_handle_request loop. That's where get_vars should be called. At-most-once semantics make it safe: all workers call get_vars('redis-watcher') during boot, only the first starts the sidekick, all block until it's ready. Sidekicks are never stopped from PHP; they live for the process lifetime and shut down cooperatively via should_stop().

About naming: my preference goes to "sidekick" precisely because these workers ARE secondary to the main app. A Messenger consumer is a different pattern (standalone queue processor). Sidekicks observe the environment and support the main app. The name is also memorable and distinct from the existing worker concept.

Caddy-config-based workers could be added later as a complementary approach for ops-level control, although I'm not sure that'd provide the best DX for app developers. Those will prefer not touching the Caddyfile when adding a new sidekick (eg when installing a third-party-provided one).

[AlliBalliBaba]

About APCu: I want sidekicks to be a core feature of FrankenPHP that people can reliably build on, not something depending on an optional third-party extension with its own bugs and serialization overhead

Not saying it must be apcu and it would also be fine to start off with string-only support. Just saying that you can't change the API afterwards without BC break, so the functions/classes added to core should be well-named and well thought out so there's room for extension and optinization.

"very messy to let http workers start and stop them"

It is messy since the http-workers can start these background threads at runtime without any mechanism to oversee or stop them. It can become hard to reason about which sidekick workers are currently running.
If you define them via config, you can be sure: "there is exactly one redis discovery worker active in this process at all times" by just looking at the config. You can even have a pool of worker threads and different entry-scripts.

The original concept of 'task workers' had workers call frankenphp_handle_task() or frankenphp_handle_request() to mark them as ready (requests are not received until all workers are ready). Other threads then could send a task to the worker. In your case, the worker could just be pinged periodically from the go side with a task to discover the redis endpoint.