Skip to content

Commit 32da496

Browse files
claudeJoibel
claude
authored and
Joibel
committed
feat(controller): auto-restart pods that failed before starting
Signed-off-by: Alan Clucas <alan@clucas.org>
1 parent 999d04c commit 32da496

33 files changed

+1771
-744
lines changed

‎.features/pending/pod-restart.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
Description: Restart pods that fail before starting
2+
Authors: [Alan Clucas](https://github.com/Joibel)
3+
Component: General
4+
Issues: 12572
5+
6+
Automatically restart pods that fail before starting for reasons like node eviction.
7+
This is safe to do even for non-idempotent workloads.
8+
You need to configure this in your workflow controller configmap for it to take effect.

‎api/jsonschema/schema.json‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/openapi-spec/swagger.json‎

Lines changed: 4 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎config/config.go‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,39 @@ type Config struct {
123123

124124
// ArtifactDrivers lists artifact driver plugins we can use
125125
ArtifactDrivers []ArtifactDriver `json:"artifactDrivers,omitempty"`
126+
127+
// FailedPodRestart configures automatic restart of pods that fail before entering Running state
128+
// (e.g., due to Eviction, DiskPressure, Preemption). This allows recovery from transient
129+
// infrastructure issues without requiring a retryStrategy on templates.
130+
FailedPodRestart *FailedPodRestartConfig `json:"failedPodRestart,omitempty"`
131+
}
132+
133+
// FailedPodRestartConfig configures automatic restart of pods that fail before entering Running state.
134+
// This is useful for recovering from transient infrastructure issues like node eviction due to
135+
// DiskPressure or MemoryPressure without requiring a retryStrategy on every template.
136+
type FailedPodRestartConfig struct {
137+
// Enabled enables automatic restart of pods that fail before entering Running state.
138+
// When enabled, pods that fail due to infrastructure issues (like eviction) without ever
139+
// running their main container will be automatically recreated.
140+
// Default is false.
141+
Enabled bool `json:"enabled,omitempty"`
142+
143+
// MaxRestarts is the maximum number of automatic restarts per node before giving up.
144+
// This prevents infinite restart loops. Default is 3.
145+
MaxRestarts *int32 `json:"maxRestarts,omitempty"`
146+
}
147+
148+
// GetMaxRestarts returns the configured max restarts or the default value of 3.
149+
func (c *FailedPodRestartConfig) GetMaxRestarts() int32 {
150+
if c == nil || c.MaxRestarts == nil {
151+
return 3
152+
}
153+
return *c.MaxRestarts
154+
}
155+
156+
// IsEnabled returns true if the feature is enabled.
157+
func (c *FailedPodRestartConfig) IsEnabled() bool {
158+
return c != nil && c.Enabled
126159
}
127160

128161
// ArtifactDriver is a plugin for an artifact driver

‎docs/fields.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1991,6 +1991,7 @@ NodeStatus contains status information about an individual node in the workflow
19911991
|`daemoned`|`boolean`|Daemoned tracks whether or not this node was daemoned and need to be terminated|
19921992
|`displayName`|`string`|DisplayName is a human readable representation of the node. Unique within a template boundary|
19931993
|`estimatedDuration`|`integer`|EstimatedDuration in seconds.|
1994+
|`failedPodRestarts`|`integer`|FailedPodRestarts tracks the number of times the pod for this node was restarted due to infrastructure failures before the main container started.|
19941995
|`finishedAt`|[`Time`](#time)|Time at which this node completed|
19951996
|`hostNodeName`|`string`|HostNodeName name of the Kubernetes node on which the Pod is running, if applicable|
19961997
|`id`|`string`|ID is a unique identifier of a node within the worklow It is implemented as a hash of the node name, which makes the ID deterministic|

‎docs/metrics.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,6 +381,28 @@ Total number of pods that started pending by reason.
381381
| `reason` | Summary of the kubernetes Reason for pending |
382382
| `namespace` | The namespace that the pod is in |
383383

384+
#### `pod_restarts_total`
385+
386+
Total number of pods automatically restarted due to infrastructure failures before the main container started.
387+
This counter tracks pods that were automatically restarted by the [failed pod restart](pod-restarts.md) feature.
388+
These are infrastructure-level failures (like node eviction) that occur before the main container enters the Running state.
389+
390+
| attribute | explanation |
391+
|-------------|-------------------------------------------------------------------------------------------------------------|
392+
| `reason` | The infrastructure failure reason: `Evicted`, `NodeShutdown`, `NodeAffinity`, or `UnexpectedAdmissionError` |
393+
| `condition` | The node condition that caused the pod restart, e.g., `DiskPressure`, `MemoryPressure` |
394+
| `namespace` | The namespace that the pod is in |
395+
396+
`reason` will be one of:
397+
398+
- `Evicted`: Node pressure eviction (`DiskPressure`, `MemoryPressure`, etc.)
399+
- `NodeShutdown`: Graceful node shutdown
400+
- `NodeAffinity`: Node affinity/selector no longer matches
401+
- `UnexpectedAdmissionError`: Unexpected error during pod admission
402+
403+
`condition` is extracted from the pod status message when available (e.g., `DiskPressure`, `MemoryPressure`).
404+
It will be empty if the condition cannot be determined.
405+
384406
#### `pods_gauge`
385407

386408
A gauge of the number of workflow created pods currently in the cluster in each phase.

‎docs/pod-restarts.md‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
# Automatic Pod Restarts
2+
3+
Argo Workflows can automatically restart pods that fail due to infrastructure issues before the main container starts.
4+
This feature handles transient failures like node evictions, disk pressure, or unexpected admission errors without requiring a `retryStrategy` on your templates.
5+
6+
## How It Works
7+
8+
When a pod fails before its main container enters the Running state, the workflow controller checks if the failure reason indicates an infrastructure issue.
9+
If so, the pod is automatically deleted and recreated, allowing the workflow to continue.
10+
For safety this mechanism only works on pods we know never started, for pods that might have started `retryStrategy` is the solution.
11+
12+
This is different from [retryStrategy](retries.md), which handles application-level failures after the container has run.
13+
These are complementary mechanisms, in that both can occur.
14+
Automatic pod restarts handle infrastructure-level failures that occur before your code even starts.
15+
16+
### Restartable Failure Reasons
17+
18+
The following pod failure reasons trigger automatic restarts:
19+
20+
| Reason | Description |
21+
|--------|-------------|
22+
| `Evicted` | Node pressure eviction (`DiskPressure`, `MemoryPressure`, etc.) |
23+
| `NodeShutdown` | Graceful node shutdown |
24+
| `NodeAffinity` | Node affinity/selector no longer matches |
25+
| `UnexpectedAdmissionError` | Unexpected error during pod admission |
26+
27+
### Conditions for Restart
28+
29+
A pod qualifies for automatic restart when ALL of the following are true:
30+
31+
1. The pod phase is `Failed`
32+
2. The main container never entered the `Running` state
33+
3. The failure reason is one of the restartable reasons listed above
34+
4. The restart count for this pod hasn't exceeded the configured maximum
35+
36+
## Configuration
37+
38+
Enable automatic pod restarts in the workflow controller ConfigMap:
39+
40+
```yaml
41+
apiVersion: v1
42+
kind: ConfigMap
43+
metadata:
44+
name: workflow-controller-configmap
45+
data:
46+
failedPodRestart: |
47+
enabled: true
48+
maxRestarts: 3
49+
```
50+
51+
### Configuration Options
52+
53+
| Option | Type | Default | Description |
54+
|--------|------|---------|-------------|
55+
| `enabled` | `bool` | `false` | Enable automatic pod restarts |
56+
| `maxRestarts` | `int` | `3` | Maximum restart attempts per node before giving up |
57+
58+
## Monitoring
59+
60+
When a pod is automatically restarted, the node status is updated with:
61+
62+
- `FailedPodRestarts`: Counter tracking how many times the pod was restarted
63+
- `Message`: Updated to indicate the restart, e.g., `Pod auto-restarting due to Evicted: The node had condition: [DiskPressure]`
64+
65+
You can view restart counts in the workflow status:
66+
67+
```bash
68+
kubectl get wf my-workflow -o jsonpath='{.status.nodes[*].failedPodRestarts}'
69+
```
70+
71+
The [`pod_restarts_total`](metrics.md#pod_restarts_total) metric tracks restarts by reason, condition, and namespace.
72+
73+
## Comparison with `retryStrategy`
74+
75+
| Feature | Automatic Pod Restarts | retryStrategy |
76+
|---------|----------------------|---------------|
77+
| **Trigger** | Infrastructure failures before container starts | Application failures after container runs |
78+
| **Configuration** | Global (controller ConfigMap) | Per-template |
79+
| **Use case** | Node evictions, disk pressure, admission errors | Application errors, transient failures |
80+
| **Counter** | `failedPodRestarts` in node status | `retries` in node status |
81+
82+
Both features can work together.
83+
If a pod is evicted before starting, automatic restart handles it.
84+
If the container runs and fails, `retryStrategy` handles it.
85+
Some pods may not be idempotent, and so a `retryStrategy` would not be suitable, but restarting the pod is safe.
86+
87+
## Example
88+
89+
A workflow running on a node that experiences disk pressure:
90+
91+
1. Pod is scheduled and init containers start
92+
2. Node experiences `DiskPressure`, evicting the pod before main container starts
93+
3. Controller detects the eviction and `FailedPodRestarts` condition
94+
4. Pod is deleted, and in workflow the node is marked as Pending to recreate the pod
95+
5. New pod is created on a healthy node
96+
6. Workflow continues normally
97+
98+
The workflow succeeds without any template-level retry configuration needed.

‎docs/retries.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,10 @@
22

33
Argo Workflows offers a range of options for retrying failed steps.
44

5+
!!! Note "restarts"
6+
For infrastructure-level failures that occur before your container starts (like node evictions or disk pressure), see [Automatic Pod Restarts](pod-restarts.md).
7+
This page covers application-level retries using `retryStrategy`.
8+
59
## Configuring `retryStrategy` in `WorkflowSpec`
610

711
```yaml

‎docs/workflow-controller-configmap.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ Config contains the root of the configuration settings for the workflow controll
9797
| `SSO` | [`SSOConfig`](#ssoconfig) | SSO in settings for single-sign on |
9898
| `Synchronization` | [`SyncConfig`](#syncconfig) | Synchronization via databases config |
9999
| `ArtifactDrivers` | `Array<`[`ArtifactDriver`](#artifactdriver)`>` | ArtifactDrivers lists artifact driver plugins we can use |
100+
| `FailedPodRestart` | [`FailedPodRestartConfig`](#failedpodrestartconfig) | FailedPodRestart configures automatic restart of pods that fail before entering Running state (e.g., due to Eviction, DiskPressure, Preemption). This allows recovery from transient infrastructure issues without requiring a retryStrategy on templates. |
100101

101102
## NodeEvents
102103

@@ -344,3 +345,14 @@ ArtifactDriver is a plugin for an artifact driver
344345
| `Name` | `wfv1.ArtifactPluginName` (string (name of an artifact plugin)) | Name is the name of the artifact driver plugin |
345346
| `Image` | `string` | Image is the docker image of the artifact driver |
346347
| `ConnectionTimeoutSeconds` | `int32` | ConnectionTimeoutSeconds is the timeout for the artifact driver connection, 5 seconds if not set |
348+
349+
## FailedPodRestartConfig
350+
351+
FailedPodRestartConfig configures automatic restart of pods that fail before entering Running state. This is useful for recovering from transient infrastructure issues like node eviction due to DiskPressure or MemoryPressure without requiring a retryStrategy on every template.
352+
353+
### Fields
354+
355+
| Field Name | Field Type | Description |
356+
|---------------|------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
357+
| `Enabled` | `bool` | Enabled enables automatic restart of pods that fail before entering Running state. When enabled, pods that fail due to infrastructure issues (like eviction) without ever running their main container will be automatically recreated. Default is false. |
358+
| `MaxRestarts` | `int32` | MaxRestarts is the maximum number of automatic restarts per node before giving up. This prevents infinite restart loops. Default is 3. |

‎manifests/base/crds/full/argoproj.io_workflows.yaml‎

Lines changed: 3 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)