Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Note that in Temporal, **Worker Deployment** is sometimes referred to as **Deplo
- Deletion of resources associated with drained Worker Deployment Versions
- `Manual`, `AllAtOnce`, and `Progressive` rollouts of new versions
- Ability to specify a "gate" Workflow that must succeed on the new version before routing real traffic to that version
- [Autoscaling](/develop/worker-performance#recommended-approach) of versioned Deployments
- Autoscaling of versioned Deployments

Refer to the [Temporal Worker Controller repo](https://github.com/temporalio/temporal-worker-controller/) for usage details.

Expand Down Expand Up @@ -90,3 +90,161 @@ helm install temporal-worker-controller ./helm/temporal-worker-controller \
```

Refer to [GitHub](https://github.com/temporalio/temporal-worker-controller/tree/main/helm/temporal-worker-controller/templates) for other Worker Controller deployment templates.

## Per-version autoscaling

You can perform autoscaling by attaching Kubernetes resources like HPAs, PodDisruptionBudgets, or other custom scalers to each versioned Deployment via the `WorkerResourceTemplate`.

The Worker Controller creates one Kubernetes Deployment per Worker version or Build ID. If you attach an HPA directly to a single Deployment, for example, it will break as the versions roll over. For example, the old HPA may still target the old Deployment, the new Deployment has no HPA, and you have to manage clean up.

`WorkerResourceTemplate` solves this by treating the Kubernetes resource as a template. The controller renders one instance per Worker version with running Workers, injects the correct versioned Deployment name, and cleans up automatically when the versioned Deployment is deleted.

:::tip

This is also the recommended mechanism for metric-based or backlog-based autoscaling: attach a standard HPA with custom metrics to your Workers and the controller keeps one per running Worker version, each pointing to the correct Deployment.

:::

### How it works

1. You create a `WorkerResourceTemplate` that references a `TemporalWorkerDeployment` and contains the resource spec in `spec.template`.
2. The validating webhook checks that you have permission to manage that resource type yourself and that the resource kind is on the allowed list.
3. On each reconcile loop, the controller renders one copy of `spec.template` per Worker version with a running Deployment, injects fields, and applies it via Server-Side Apply.
4. Each copy is owned by the corresponding versioned Deployment, so it's garbage-collected automatically when that Deployment is deleted.
5. `WorkerResourceTemplate.status.versions` is updated with the applied or failed status for each version.

### Auto-injection

The controller auto-injects two fields when you set them to `null` in `spec.template`. Setting them to `null` is the explicit signal that you want injection. If you omit the field entirely, nothing is injected. If you set a non-null value, the webhook rejects the `WorkerResourceTemplate` because the controller owns these fields.

| Field | Injected value |
|-------|---------------|
| `spec.scaleTargetRef` (any resource with this field) | `{apiVersion: apps/v1, kind: Deployment, name: <versioned-deployment-name>}` |
| `spec.selector.matchLabels` (any resource with this field) | `{temporal.io/build-id: <buildID>, temporal.io/deployment-name: <twdName>}` |

The `scaleTargetRef` injection applies to any resource type that has a `scaleTargetRef` field, not just HPAs. Other autoscaler custom resource definitions (CRDs) use the same field and benefit from the same injection.

### Resource naming

Each per-Build-ID copy is given a unique, DNS-safe name derived from the `(twdName, wrtName, buildID)` triple. Names are capped at 47 characters to be safe for all Kubernetes resource types, including Deployment which has pod-naming constraints that effectively limit Deployment names to ~47 characters. The name always ends with an 8-character hash of the full triple, so uniqueness is guaranteed even when the human-readable prefix is truncated.

Use `kubectl get <kind>` after a reconcile to see the created resources and their names.

### Allowed resource kinds and RBAC

The `workerResourceTemplate.allowedResources` value in Helm serves two purposes: it defines which resource kinds the webhook will accept and it drives the RBAC rules granted to the controller's `ClusterRole`. Only kinds listed here can be embedded in a `WorkerResourceTemplate`.

The default allows HPAs:

```yaml
workerResourceTemplate:
allowedResources:
- kinds: ["HorizontalPodAutoscaler"]
apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
```

To also allow `PodDisruptionBudgets`, add an entry:

```yaml
workerResourceTemplate:
allowedResources:
- kinds: ["HorizontalPodAutoscaler"]
apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
- kinds: ["PodDisruptionBudget"]
apiGroups: ["policy"]
resources: ["poddisruptionbudgets"]
```

Each entry has three fields:
- `kinds` — kind names the webhook accepts (case-insensitive)
- `apiGroups` — API groups used to generate the controller's RBAC rules
- `resources` — resource names used to generate the controller's RBAC rules

#### What the webhook checks

When you create or update a `WorkerResourceTemplate`, the webhook performs `SubjectAccessReviews` to verify:

1. **You** (the requesting user) can create and update the embedded resource type in that Namespace.
2. **The controller's service account** can create and update the embedded resource type in that Namespace.

If either check fails, the request is rejected. This prevents privilege escalation so you can't use `WorkerResourceTemplate` to create resources you don't already have permission to create yourself.

Users who create `WorkerResourceTemplates` need RBAC permission to manage the embedded resource type directly. For example, to let a team create `WorkerResourceTemplates` that embed HPAs, they need the standard `autoscaling` permissions in their Namespace.

### Webhook TLS

The `WorkerResourceTemplate` validating webhook requires TLS. The Helm chart uses cert-manager to provision the certificate (`certmanager.enabled: true` is the default).

If cert-manager is not already installed in your cluster, you can either install it separately ([cert-manager installation docs](https://cert-manager.io/docs/installation/)) or let the Helm chart install it as a subchart by setting `certmanager.install: true`.

### Example: HPA per worker version

```yaml
apiVersion: temporal.io/v1alpha1
kind: WorkerResourceTemplate
metadata:
name: my-worker-hpa
namespace: my-namespace
spec:
# Reference the TemporalWorkerDeployment to attach to.
temporalWorkerDeploymentRef:
name: my-worker

# The resource template. The controller creates one copy per worker version
# with a running Deployment.
template:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
# {} tells the controller to auto-inject the versioned Deployment reference.
# Do not set this to a real value — the webhook will reject it.
scaleTargetRef: {}
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
```

See the [temporal-worker-controller wrt-hpa.yaml](https://github.com/temporalio/temporal-worker-controller/blob/e63f8c5d0dfad8d4c9112de48c91af2fe2637552/examples/wrt-hpa.yaml) for an example pre-configured for the helloworld demo.

### Example: PodDisruptionBudget per worker version

```yaml
apiVersion: temporal.io/v1alpha1
kind: WorkerResourceTemplate
metadata:
name: my-worker-pdb
namespace: my-namespace
spec:
temporalWorkerDeploymentRef:
name: my-worker
template:
apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
minAvailable: 1
# {} tells the controller to auto-inject {temporal.io/build-id, temporal.io/deployment-name}.
selector:
matchLabels: {}
```

### Checking status

```bash
# See all WorkerResourceTemplates and which TWD they reference
kubectl get WorkerResourceTemplate -n my-namespace

# See per-Build-ID apply status
kubectl get WorkerResourceTemplate my-worker-hpa -n my-namespace \
-o jsonpath='{.status.versions}' | jq .

# See the created HPAs
kubectl get hpa -n my-namespace
```
Loading