Skip to content

docs: update blob size limit error troubleshooting page #4379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lennessyy wants to merge 3 commits into
base: main
Choose a base branch
from
Open

docs: update blob size limit error troubleshooting page #4379

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6c303ec
docs: update blob size limit error page with external storage guidance
lennessyy Apr 2, 2026
c792b7b
docs: revert prettier line-width reformats
lennessyy Apr 2, 2026
6a13895
docs: restructure blob size limit page with accuracy fixes
lennessyy Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 51 additions & 30 deletions docs/troubleshooting/blob-size-limit-error.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
id: blob-size-limit-error
title: Troubleshoot the blob size limit error
sidebar_label: Blob size limit error
description: The BlobSizeLimitError occurs when a Workflow's payload exceeds the 2 MB request limit or the 4 MB Event History transaction limit set by Temporal. Reduce blob size via compression or batching.
title: Troubleshoot payload and gRPC message size limit errors
sidebar_label: Payload and gRPC message size limit errors
description: Temporal enforces size limits on data passed between Workers and the Temporal Service. Learn about the 2 MB payload limit and 4 MB gRPC message limit, their error messages, and how to resolve them.
toc_max_heading_level: 4
keywords:
- error
Expand All @@ -12,48 +12,69 @@ tags:
- Failures
---

The `BlobSizeLimitError` is an error that occurs when the size of a blob (payloads including Workflow context and each Workflow and Activity argument and return value) exceeds the set limit in Temporal.
Temporal enforces size limits on the data that passes between the Temporal Client, Workers, and the Temporal Service.
There are two distinct limits, each producing different error messages and behaviors, and they require different solutions:

- The max payload for a single request is 2 MB.
- The max size limit for any given [Event History](/workflow-execution/event#event-history) transaction is 4 MB.
- [Payload size limit](#payload-size-limit)
- [gRPC message size limit](#grpc-message-size-limit)

## Why does this error occur?
## Payload size limit

This error occurs when the size of the blob exceeds the maximum size allowed by Temporal.
The Temporal Service enforces a size limit on individual payloads. This limit is 2 MB on Temporal Cloud,
but is configurable on self-hosted deployments with a default of 2 MB.
A [payload](/dataconversion#payload) represents the serialized binary data for the input and output of Workflows and Activities.
When a single payload exceeds this limit, the Temporal Service rejects the request and terminates the Workflow.

This limit helps ensure that the Temporal Service prevents excessive resource usage and potential performance issues when handling large payloads.
### Error messages

## How do I resolve this error?
The error message depends on which operation carried the oversized payload. Examples include:
- `BadScheduleActivityAttributes: ScheduleActivityTaskCommandAttributes.Input exceeds size limit`
- `WORKFLOW_TASK_FAILED_CAUSE_BAD_UPDATE_WORKFLOW_EXECUTION_MESSAGE`

To resolve this error, reduce the size of the blob so that it is within the 4 MB limit.
### How to resolve

There are multiple strategies you can use to avoid this error:
1. Offload large payloads to an object store to reduce the risk of exceeding payload size limits:

1. Use compression with a [custom payload codec](/payload-codec) for large payloads.
1. Pass references to the stored payloads within the Workflow instead of the actual data.
1. Retrieve the payloads from the object store when needed during execution.

- This addresses the immediate issue of the blob size limit; however, if blob sizes continue to grow this problem can arise again.
This is called the [claim check pattern](https://dataengineering.wiki/Concepts/Software+Engineering/Claim+Check+Pattern).
You can implement your own claim check pattern by using a custom [Payload Codec](/payload-codec),
or use [External Storage](/external-storage).

2. Break larger batches of commands into smaller batch sizes:
This is the most reliable way to avoid hitting payload size limits. Consider implementing the claim check pattern
for Workflows and Activities that have the potential to receive or return large payloads, even if they are currently within the limit.

- Workflow-level batching:
1. Modify the Workflow to process Activities or Child Workflows into smaller batches.
2. Iterate through each batch, waiting for completion before moving to the next.
- Workflow Task-level batching:
1. Execute Activities in smaller batches within a single Workflow Task.
2. Introduce brief pauses or sleeps (for example, 1ms) between batches.
:::tip Support, stability, and dependency info

3. Consider offloading large payloads to an object store to reduce the risk of exceeding blob size limits:
1. Pass references to the stored payloads within the Workflow instead of the actual data.
2. Retrieve the payloads from the object store when needed during execution.
External Storage is currently in [Pre-release](/evaluate/development-production-features/release-stages#pre-release). All APIs
are experimental and may be subject to backwards-incompatible changes. Join the
[#large-payloads Slack channel](https://temporalio.slack.com/archives/C09VA2DE15Y) to provide feedback or ask for help.

## Workflow termination due to oversized response
:::

When a Workflow Task response exceeds the 4 MB gRPC message size limit, Temporal automatically terminates the Workflow Execution. This is a non-recoverable error. The Workflow can't progress if it generates a response that's too large, so retrying won't help.
1. Use compression with a [custom Payload Codec](/payload-codec) for large payloads. This addresses the immediate issue, but if payload sizes continue to grow, the problem can arise again.

This typically happens when a Workflow schedules too many Activities, Child Workflows, or other commands in a single Workflow Task. The total size of all commands generated by the Workflow Task must fit within the 4 MB limit.
## gRPC message size limit

If your Workflow was terminated for this reason, you'll see a `WorkflowExecutionTerminated` event in the Event History with the cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`.
All communication between the Temporal Client, Workers, and the Temporal Service uses gRPC, which enforces a 4 MB limit on each request. This limit applies to the full request, including all payload data and command metadata. For example, when a Workflow schedules multiple Activities in a single Workflow Task, the Worker sends one request containing all those commands to schedule the Activities and their inputs. If the combined size exceeds 4 MB, the gRPC layer rejects the request.

To prevent this, use the batching strategies described above to split work across multiple Workflow Tasks instead of scheduling everything at once.
A Workflow can hit this limit even when every individual payload is under 2 MB. Scheduling several Activities with moderate-sized inputs, or hundreds of Activities with tiny inputs in the same Workflow Task can push the combined request past 4 MB. Activity results are also subject to this limit.

See the [gRPC Message Too Large error reference](/references/errors#grpc-message-too-large) for more details.
### Error messages

- `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`: When a Workflow Worker completes a Workflow Task, it sends all the commands the Workflow produced (such as Activity schedules and their inputs) back to the Temporal Service. If that response exceeds 4 MB, the SDK catches the gRPC error and sends a failed Workflow Task response with this cause. Because replay produces the same oversized response, the Workflow gets stuck in a retry loop that isn't visible in the Event History.
Copy link
Copy Markdown
Contributor

@jmaeagle99 jmaeagle99 Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a new error message that is analogous to this one called WORKFLOW_TASK_FAILED_CAUSE_PAYLOADS_TOO_LARGE.

The SDK will get the payload limits from the server and intentionally fail workflow tasks if it contains payloads that are over the payload size limit. This enables the workflow to be retried instead of failing it so that other solutions (such as external storage) may be applied to alleviate the problem and allow the workflow to continue.

Copy link
Copy Markdown
Contributor Author

@lennessyy lennessyy Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, I'm a little confused by this. What makes this error different from the other payload errors (in the above section)? Why does this allow retries while other payload errors terminate the workflow per Vlad's testing?

Or do you mean this is a new error message we are adding together with the release of external storage?

Copy link
Copy Markdown
Contributor

@jmaeagle99 jmaeagle99 Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a new error code. Prior to adding this code and the new behavior to the SDK, if the SDK submitted payloads that were greater than 2 MB but less than 4 MB (the gRPC limit), the server would fail the workflow (not just the workflow task, but the entire workflow). The SDK does the best effort size check on the worker and intentionally fails the workflow task instead of uploading the large payload that would fail the workflow.


- `StartToCloseTimeout`: This error message can be misleading. When an Activity Worker completes an Activity Task, it sends the Activity result back to the Temporal Service. If that result exceeds 4 MB, the gRPC call fails and the Worker can't deliver it. The server never receives the completion and eventually times out the Activity. The actual `ResourceExhausted` gRPC error only appears in Worker logs.

### How to resolve

1. Break larger batches of commands into smaller batch sizes:
- Workflow-level batching:
1. Modify the Workflow to process Activities or Child Workflows in smaller batches.
2. Iterate through each batch, waiting for completion before moving to the next.
- Workflow Task-level batching:
1. Execute Activities in smaller batches within a single Workflow Task.
2. Introduce brief pauses or sleeps between batches.

2. If the request is large because of payload sizes rather than the number of commands, refer to the [Payload size limit](#payload-size-limit) section for solutions.
Loading