windmill-labs · tristantr · Dec 11, 2025 · Dec 11, 2025 · Dec 11, 2025 · Dec 11, 2025
diff --git a/changelog/2025-12-15-trigger-suspended-mode/index.md b/changelog/2025-12-15-trigger-suspended-mode/index.md
@@ -0,0 +1,17 @@
+---
+slug: trigger-suspended-mode
+version: v1.590.0
+title: Trigger suspended mode
+tags: ['Triggers']
+description: Triggers can now be placed in suspended mode to queue incoming jobs without running them automatically, enabling safer debugging and testing of trigger configurations.
+image: ./trigger-suspended-mode.png
+features:
+  [
+    'Suspend triggers to queue jobs without automatic execution',
+    'Review and manage all suspended jobs in a dedicated table',
+    'Resume or discard jobs individually or in bulk',
+    'Updated trigger configurations automatically apply to resumed jobs',
+    'Useful for debugging runnable or trigger logic without disabling the trigger'
+  ]
+docs: /docs/getting_started/triggers#suspended-mode
+---
diff --git a/changelog/2025-12-15-trigger-suspended-mode/trigger-suspended-mode.png b/changelog/2025-12-15-trigger-suspended-mode/trigger-suspended-mode.png
diff --git a/docs/advanced/21_scaling/index.mdx b/docs/advanced/21_scaling/index.mdx
@@ -0,0 +1,260 @@
+import DocCard from '@site/src/components/DocCard';
+import WorkerQueueSimulator from '@site/src/components/WorkerQueueSimulator';
+
+# Scaling workers
+
+Windmill uses a worker queue architecture where workers pull jobs from a shared queue and execute them one at a time. Understanding this pattern is essential for properly sizing your worker pool to meet your business requirements.
+
+## How workers process jobs
+
+Workers are autonomous processes that pull jobs from a queue in order of their scheduled time. Each worker:
+
+- Executes one job at a time using full CPU and memory
+- Pulls the next job as soon as the current one completes
+- Can run up to 26 million jobs per month (at 100ms per job)
+
+This architecture is horizontally scalable: add more workers to increase throughput, remove workers to reduce costs. There is no coordination overhead between workers.
+
+<div className="grid grid-cols-2 gap-6 mb-4">
+	<DocCard
+		title="Workers and worker groups"
+		description="Learn about Windmill's worker architecture and how to configure worker groups."
+		href="/docs/core_concepts/worker_groups"
+	/>
+	<DocCard
+		title="Architecture"
+		description="Overview of Windmill's technical architecture."
+		href="/docs/misc/architecture"
+	/>
+</div>
+
+## Interactive simulator
+
+Use this simulator to visualize how jobs flow through the queue and understand the relationship between job arrival rate, job duration, and worker count.
+
+<WorkerQueueSimulator />
+
+### Simulator modes
+
+- **Batch**: All jobs are submitted at once, simulating scheduled bulk operations
+- **Continuous**: Jobs arrive at a steady rate, simulating regular workloads
+- **Random**: Jobs arrive at varying intervals, simulating unpredictable traffic
+
+### Key metrics
+
+- **Elapsed time**: Total time from first job to last completion
+- **Jobs/sec**: Actual throughput achieved
+- **Worker occupancy**: Percentage of time each worker spent processing (vs idle)
+
+## Sizing your worker pool
+
+The right number of workers depends on your specific requirements. Consider these factors:
+
+### Job duration and arrival rate
+
+The fundamental relationship is:
+
+```
+Required workers ≥ Job arrival rate × Average job duration
+```
+
+For example, if jobs arrive at 10/second and each takes 2 seconds:
+- Minimum workers needed: 10 × 2 = 20 workers
+
+With fewer workers, jobs will queue up. With more workers, some will be idle.
+
+### Maximum acceptable queue time
+
+If jobs must not wait more than X seconds before starting:
+
+```
+Required workers = (Peak arrival rate × Job duration) + (Peak arrival rate × Max queue time)
+```
+
+**Example**: Peak rate 5 jobs/sec, duration 3s, max wait 2s:
+- Workers needed: (5 × 3) + (5 × 2) = 15 + 10 = 25 workers
+
+This ensures even during peak load, no job waits more than 2 seconds.
+
+### Handling traffic peaks
+
+If your workload has predictable peaks (weekends, end of month, etc.):
+
+1. **Fixed capacity**: Size for peak load, accept idle workers during off-peak
+2. **Autoscaling**: Configure min/max workers to automatically adjust
+
+<div className="grid grid-cols-2 gap-6 mb-4">
+	<DocCard
+		title="Autoscaling"
+		description="Automatically scale workers based on queue depth and occupancy."
+		href="/docs/core_concepts/autoscaling"
+	/>
+</div>
+
+## Practical examples
+
+### Scenario 1: Batch ETL processing
+
+**Requirement**: Process 1,000 daily reports, each taking 30 seconds, complete within 2 hours
+
+- Total processing time: 1,000 × 30s = 30,000 seconds
+- Available time: 2 hours = 7,200 seconds
+- Minimum workers: 30,000 / 7,200 = 4.2 → **5 workers**
+
+With 5 workers, all jobs complete in approximately 100 minutes.
+
+### Scenario 2: Real-time webhook processing
+
+**Requirement**: Handle 100 webhooks/minute during business hours, each taking 5 seconds, max latency 10 seconds
+
+- Arrival rate: 100/60 = 1.67 jobs/second
+- Minimum workers: 1.67 × 5 = 8.3 workers
+- For 10s max latency headroom: **10 workers**
+
+### Scenario 3: Weekend traffic spikes
+
+**Requirement**: Normal load 2 jobs/sec, weekend peaks at 8 jobs/sec, jobs take 1 second each
+
+- Normal load: 2 × 1 = 2 workers minimum
+- Peak load: 8 × 1 = 8 workers minimum
+- **Recommended**: Use autoscaling with min=3, max=10
+
+Configure autoscaling to scale up when queue depth increases and scale down when occupancy drops below 25%.
+
+## Priority queues with worker groups
+
+For mixed workloads where some jobs are more time-sensitive:
+
+1. Create separate [worker groups](../../core_concepts/9_worker_groups/index.mdx) with different tags
+2. Assign high-priority jobs to dedicated workers
+3. Let lower-priority jobs share remaining capacity
+
+**Example configuration**:
+
+- `high-priority` worker group: 5 dedicated workers, handles critical customer-facing operations
+- `default` worker group: 10 workers, handles everything else
+- `low-priority` worker group: 3 workers, handles background analytics
+
+This ensures critical jobs are never blocked by bulk operations.
+
+<div className="grid grid-cols-2 gap-6 mb-4">
+	<DocCard
+		title="Worker groups"
+		description="Configure worker groups with different tags for job routing."
+		href="/docs/core_concepts/worker_groups"
+	/>
+	<DocCard
+		title="High priority jobs"
+		description="Set job priorities within a queue."
+		href="/docs/core_concepts/jobs#high-priority-jobs"
+	/>
+</div>
+
+## Monitoring and alerting
+
+Track worker performance to identify scaling needs:
+
+- **Queue metrics**: Monitor delayed jobs per tag and queue wait times
+- **Occupancy rates**: High sustained occupancy (>75%) suggests adding workers
+- **Worker alerts**: Get notified when workers go offline
+
+<div className="grid grid-cols-2 gap-6 mb-4">
+	<DocCard
+		title="Queue metrics"
+		description="Visualize queue depth and delays across worker groups."
+		href="/docs/core_concepts/worker_groups#queue-metrics"
+	/>
+	<DocCard
+		title="Critical alerts"
+		description="Configure alerts for worker failures and queue buildup."
+		href="/docs/core_concepts/critical_alerts"
+	/>
+</div>
+
+## Autoscaling configuration
+
+For dynamic workloads, configure autoscaling to automatically adjust worker count:
+
+| Parameter | Recommended starting value |
+| --- | --- |
+| Min workers | Expected base load / job duration |
+| Max workers | Peak load / job duration × 1.5 |
+| Scale-out trigger | 75% occupancy or jobs waiting > min_workers |
+| Scale-in trigger | Less than 25% occupancy for 5+ minutes |
+| Cooldown | 60-120 seconds between scaling events |
+
+The autoscaling algorithm checks every 30 seconds and considers:
+- Number of jobs waiting in queue
+- Worker occupancy rates over 15s, 5m, and 30m intervals
+- Cooldown periods to prevent thrashing
+
+<div className="grid grid-cols-2 gap-6 mb-4">
+	<DocCard
+		title="Autoscaling"
+		description="Configure automatic worker scaling based on demand."
+		href="/docs/core_concepts/autoscaling"
+	/>
+</div>
+
+## Worker memory sizing
+
+Workers come in different sizes based on memory limits. The right size depends on your job requirements:
+
+| Worker size | Memory | Compute units |
+| --- | --- | --- |
+| Small | 1GB | 0.5 CU |
+| Standard | 2GB | 1 CU |
+| Large | >2GB | 2 CU (self-hosted capped at 2 CU regardless of actual memory) |
+
+### Choosing the right memory limit
+
+Set worker memory based on the **maximum memory any individual job will need**, plus some headroom:
+
+- **Simple API calls, webhooks, light scripts**: 1-2GB is typically sufficient
+- **Data processing, ETL jobs**: May need 4GB+ depending on data volume processed in memory
+- **Large file processing, ML inference**: Consider 8GB+ for memory-intensive operations
+
+If a job exceeds the worker's memory limit, it will be killed by the operating system. Monitor job memory usage and increase worker memory if you see OOM (out of memory) errors.
+
+### Memory vs worker count trade-off
+
+For the same compute budget, you can choose between:
+
+- **More small workers**: Better parallelism for many short jobs
+- **Fewer large workers**: Better for memory-intensive jobs that can't be parallelized
+
+**Example**: 4 CUs can be configured as:
+- 8 small workers (1GB each) - good for high-volume, light jobs
+- 4 standard workers (2GB each) - balanced configuration
+- 2 large workers (4GB each) - good for memory-intensive ETL
+
+<div className="grid grid-cols-2 gap-6 mb-4">
+	<DocCard
+		title="Plans and pricing"
+		description="Understand compute units and how worker memory affects billing."
+		href="/docs/misc/plans_details"
+	/>
+</div>
+
+## Cost optimization
+
+Worker billing is based on usage time with minute granularity:
+
+- 10 workers for 1/10th of the month costs the same as 1 worker for the full month
+- Use autoscaling to minimize idle workers
+- Consider [dedicated workers](../../core_concepts/25_dedicated_workers/index.mdx) for high-throughput single-script scenarios
+
+Mark development and staging instances as "Non-prod" in [instance settings](../18_instance_settings/index.mdx) so they don't count toward your compute limits.
+
+<div className="grid grid-cols-2 gap-6 mb-4">
+	<DocCard
+		title="Plans and pricing"
+		description="Understand compute units and pricing."
+		href="/docs/misc/plans_details"
+	/>
+	<DocCard
+		title="Dedicated workers"
+		description="High-throughput execution for single scripts."
+		href="/docs/core_concepts/dedicated_workers"
+	/>
+</div>
diff --git a/docs/core_concepts/11_persistent_storage/data_tables.mdx b/docs/core_concepts/11_persistent_storage/data_tables.mdx
@@ -74,7 +74,16 @@ SELECT * FROM friend WHERE id = $user_id;
 </TabItem>
 </Tabs>
 
----
+We recommend to only have one or a few data tables per workspace, and to use schemas to organize your data.
+
+![Data Table schemas](./datatable_images/datatable_schemas.png 'Data Table schemas')
+
+You can reference schemas normally with the `schema.table` syntax, or set the default search path with this syntax (Python / Typescript) :
+
+```ts
+sql = wmill.datatable(':myschema'); // or 'named_datatable:myschema'
+sql`SELECT * FROM mytable`; // refers to myschema.mytable
+```
 
 ### Assets integration
 

diff --git a/docs/core_concepts/11_persistent_storage/datatable_images/datatable_schemas.png b/docs/core_concepts/11_persistent_storage/datatable_images/datatable_schemas.png
diff --git a/docs/core_concepts/9_worker_groups/index.mdx b/docs/core_concepts/9_worker_groups/index.mdx
@@ -338,6 +338,11 @@ Autoscaling is available in the [Enterprise plan](/pricing).
 		description="Autoscaling automatically adjusts the number of workers based on your workload demands."
 		href="/docs/core_concepts/autoscaling"
 	/>
+	<DocCard
+		title="Scaling workers"
+		description="Interactive simulator and guide for sizing your worker pool."
+		href="/docs/advanced/scaling"
+	/>
 </div>
 
 ## Worker scripts

diff --git a/docs/getting_started/8_triggers/enable_suspended_mode.png b/docs/getting_started/8_triggers/enable_suspended_mode.png
diff --git a/docs/getting_started/8_triggers/index.mdx b/docs/getting_started/8_triggers/index.mdx
@@ -390,3 +390,37 @@ You can test your triggers in test mode:
 	allowFullScreen
 	className="border-2 rounded-lg object-cover w-full dark:border-gray-800"
 ></iframe>
+
+## Suspended mode
+
+When a trigger is in suspended mode, it continues to accept payloads and queue jobs, but those jobs won't run automatically. This is useful for debugging your runnable or trigger logic without disabling the trigger entirely.
+
+To enable suspended mode, toggle the "Suspend job execution" option in the trigger settings:
+
+![Enable suspended mode](./enable_suspended_mode.png)
+
+### Managing suspended jobs
+
+You can review all suspended jobs by clicking the "See suspended jobs" button:
+
+![Open suspended jobs](./open_suspended_jobs_button.png)
+
+This opens a table showing all queued jobs for the trigger:
+
+![Suspended jobs table](./suspended_jobs_table.png)
+
+From this table, you can:
+- Resume individual jobs to execute them
+- Discard jobs that are no longer needed
+- Resume all jobs at once
+- Discard all jobs at once
+
+### Updating trigger configuration
+
+If you modify the trigger's configuration (such as changing the runnable, retry settings, or error handler) and save, resumed jobs will run using the updated configuration:
+
+![Reassigned suspended jobs](./reassigned_suspended_jobs.png)
+
+:::warning
+If your old runnable had a preprocessor, the new one should have one too (and vice versa), as the arguments format differs based on whether a preprocessor is present.
+:::
diff --git a/docs/getting_started/8_triggers/open_suspended_jobs_button.png b/docs/getting_started/8_triggers/open_suspended_jobs_button.png
diff --git a/docs/getting_started/8_triggers/reassigned_suspended_jobs.png b/docs/getting_started/8_triggers/reassigned_suspended_jobs.png
diff --git a/docs/getting_started/8_triggers/suspended_jobs_table.png b/docs/getting_started/8_triggers/suspended_jobs_table.png
diff --git a/docs/integrations/git_repository.mdx b/docs/integrations/git_repository.mdx
@@ -34,10 +34,16 @@ Instead of using a long lived personal access token to authenticate with GitHub
 
 GitHub App is available under [Windmill Enterprise](/pricing).
 
-As a [Windmill workspace admin](https://www.windmill.dev/docs/core_concepts/roles_and_permissions#admin), you can install the GitHub app to multiple organizations and link them to your Windmill workspaces. Once an app has been installed to a workspace, you can install it to other workspace where you have the admin role.
-
 ![GitHub App](/integrations/visual_elements/github_app_installation.png.webp)
 
+### Network requirements
+
+The GitHub App feature requires your Windmill instance to communicate with `https://stats.windmill.dev` to obtain GitHub installation tokens. This is the same endpoint used for [telemetry](../advanced/18_instance_settings/index.mdx#telemetry).
+
+If your GitHub organization uses [IP allow lists](https://docs.github.com/en/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/managing-allowed-ip-addresses-for-your-organization), you will need to whitelist the IP address of `stats.windmill.dev` to allow it to request installation tokens from GitHub on behalf of your Windmill instance. Contact support@windmill.dev to get the current IP address.
+
+As a [Windmill workspace admin](https://www.windmill.dev/docs/core_concepts/roles_and_permissions#admin), you can install the GitHub app to multiple organizations and link them to your Windmill workspaces. Once an app has been installed to a workspace, you can install it to other workspace where you have the admin role.
+
 :::warning
 You will only be able to use the installation token for [Git sync](../advanced/11_git_sync/index.mdx).
 :::
@@ -49,3 +55,9 @@ A GitHub app can only be installed to a GitHub organization once. Hence to assoc
 :::warning
 The JWT token associated to your GitHub app installation is sensitive and has the rights to request a short lived installation token. To revoke the JWT, you need to uninstall the GitHub app from your organization and re-install it to re-associate it with a windmill instance.
 :::
+
+<video
+	className="border-2 rounded-lg object-cover w-full h-full dark:border-gray-800"
+	controls
+	src="/videos/github_app_installation.mp4"
+/>
diff --git a/docs/misc/19_enterprise_onboarding/index.mdx b/docs/misc/19_enterprise_onboarding/index.mdx
@@ -187,6 +187,11 @@ Make sure your worker setup [complies with your subscription limits](../7_plans_
 		description="Configure and manage worker pools for job execution."
 		href="/docs/core_concepts/worker_groups"
 	/>
+	<DocCard
+		title="Scaling workers"
+		description="Interactive simulator and guide for sizing your worker pool."
+		href="/docs/advanced/scaling"
+	/>
 	<DocCard
 		title="Worker group management UI"
 		description="Manage worker groups, tags, and configurations from the UI."

diff --git a/sidebars.js b/sidebars.js
@@ -1028,6 +1028,7 @@ const sidebars = {
 				'advanced/clients/python_client',
 				'advanced/clients/rust_client',
 				'advanced/explicit_progress/index',
+				'advanced/scaling/index',
 				'misc/share_on_hub/index'
 			]
 		},