Regression after upgrading Microsoft.Azure.DurableTask.Core 3.2.0 → 3.6.0 (TaskHub initialization errors + unexpected 404s against Azure Table Storage)

[ohadtayl]

Hi team,
After upgrading Microsoft.Azure.DurableTask.Core and DurableTask.AzureStorage from 3.2.0 to 3.6.0, we started seeing a large spike in storage-related failures coming from the Azure Storage provider. These errors begin immediately after deployment and disappear completely when rolling back to 3.2.0.

Right after the upgrade, orchestration workers start throwing repeated errors while trying to initialize or process work for the TaskHub.
Here is an example of one of the failures:

An unexpected failure occurred while processing instance 'mgmt-cme2e1204233657cluster': DurableTask.AzureStorage.Storage.DurableTaskStorageException: An error occurred while communicating with Azure Storage ---> Azure.RequestFailedException: The specified resource does not exist.
RequestId:670a9fd6-a002-0052-4bac-65cb63000000
Time:2025-12-05T05:58:29.0299754Z
Status: 404 (Not Found)
ErrorCode: ResourceNotFound

Content:
{"odata.error":{"code":"ResourceNotFound","message":{"lang":"en-US","value":"The specified resource does not exist.\nRequestId:670a9fd6-a002-0052-4bac-65cb63000000\nTime:2025-12-05T05:58:29.0299754Z"}}}

Headers:
Transfer-Encoding: chunked
x-ms-request-id: 670a9fd6-a002-0052-4bac-65cb63000000
x-ms-client-request-id: 87682197-128c-412b-91d5-045f83cfc17d
x-ms-version: REDACTED
X-Content-Type-Options: REDACTED
Cache-Control: no-cache
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
Date: Fri, 05 Dec 2025 05:58:28 GMT

at Azure.Data.Tables.TableRestClient.d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Azure.Data.Tables.TableClient.d__511.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at DurableTask.AzureStorage.Storage.ClientResponseExtensions.<DecorateFailure>d__1.MoveNext() in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 30 --- End of inner exception stack trace --- at DurableTask.AzureStorage.Storage.ClientResponseExtensions.<DecorateFailure>d__1.MoveNext() in /_/src/DurableTask.AzureStorage/Storage/ClientResponseExtensions.cs:line 34 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at DurableTask.AzureStorage.Storage.Table.<MergeEntityAsync>d__161.MoveNext() in //src/DurableTask.AzureStorage/Storage/Table.cs:line 97
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DurableTask.AzureStorage.Tracking.AzureTableTrackingStore.d__52.MoveNext() in //src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs:line 815
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DurableTask.AzureStorage.AzureStorageOrchestrationService.d__89.MoveNext() in //src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 1083
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DurableTask.AzureStorage.AzureStorageOrchestrationService.d__83.MoveNext() in //src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs:line 796

Impact-
TaskHub workers get stuck retrying.
Queue length (work items) grows quickly.
Latency spikes.
Some orchestrations never get picked up.
Rolling back to 3.2.0 immediately resolves the issue.

Were there any breaking changes introduced in 3.6.0 that affect the Azure Storage provider?
Did the table schema or required entity format change?
Are there new configuration flags or capabilities we need to enable?
Is this a known issue or regression?

[sophiatev]

Hi @ohadtayl , based on the line numbers that appear in the stack trace, it seems like the code path for terminating a pending orchestration is getting triggered, but there is no entry for the orchestration in the instance table, so when we attempt to update the instance table with a Merge call, the 404 is thrown.

Can you provide the name of your app so we can look up the errors in Kusto?

Is it possible you are trying to terminate purged orchestrations?

[ohadtayl]

Hey @sophiatev, thanks for getting back to me so quickly!
I wanted to clarify a few things about our setup:

We're using Durable Task Framework as NuGet libraries within our service, not Azure Functions. Our logs go to our own tracer, and we currently only capture Warning and Error levels, so we don’t have the Information-level traces you mentioned.

Is there anything specific I can provide to help you look into these errors?
We're also seeing another error type: "cmgmt-kustoppescus: Abandoning [TimerFired#0] message back to storagemonitoringhub-control-03 and setting a visibility delay of 2ms"

Why do we see more errors after upgrading the package, but significantly fewer when we roll back? Also, how is this related to the increase in control queue length and latency?

[sophiatev]

@ohadtayl It's difficult to ascertain what's going on without access to your logs on our end.

I can tell you that the path that's getting triggered is the one for terminating a pending orchestration. When attempting to update the instance status to "terminated", it says it cannot find the entry in the instance table and so a 404 gets thrown. As I mentioned in my earlier comment, one way this could happen is if you purge the pending orchestration before the terminate message gets dequeued.

[ohadtayl]

One thing we’re still trying to understand is why this behavior only surfaced after upgrading to 3.6.0, and why rolling back to 3.2.0 immediately eliminates all symptoms. Nothing in our deployment changed except the package versions, so we’re trying to pinpoint what in 3.6.0 could cause the termination‑pending path to run more frequently, or under conditions where it didn’t trigger before.
A few questions that might help us narrow this down:

What changed between 3.2.0 → 3.6.0 that might lead to the terminate‑pending logic being executed in cases where it previously wasn’t?
Why does the missing instance‑table entry in 3.6.0 translate into significantly increased latency, growing control‑queue length, and more retries, while 3.2.0 remains completely stable under identical load?
Is the regression you referenced in #1288 the likely cause of this pattern (404s during Merge + queue buildup + stuck workers)?

If there’s any specific logging, instance patterns, or message types that would help your investigation, we can surface those.
And if it would be more effective, we’re happy to set up a short debugging session so you can look at the behavior directly.
Let us know what would help you diagnose this further — we appreciate the guidance.

[sophiatev]

@ohadtayl Sorry, I should have been more clear. Terminating pending orchestrations was only introduced in 3.6.0, so that explains why this path could only be triggered after upgrading.

The missing instance-table entry in this case causes uncaught exceptions to surface, which could contribute to all of the problems you're describing, or there could be some other underlying issue as well. It's hard to determine without information-level logs what is going on.

Are you purging orchestrations? I can't think of what else could cause the instance entity to not exist in this path. I ultimately do not plan to merge that PR since really the issue is that we allow purging orchestrations in a non-terminal state. So I plan to work on another PR to address that (unless, somehow, there is something else that is causing the instance entity to not exist).

[ohadtayl]

We’re not purging orchestrations on our side. The only scenario that could occur from our end is terminating an orchestration that’s still in a Pending state. If this behavior was introduced in 3.6.0 and is expected, we can reproduce it in a dev environment with Information-level traces to help confirm. If that would be useful, let me know and I’ll set this up.

Also, is there any recommended long-term approach for migrating to 3.6.0 without hitting these issues? Is there something we can do on our side to make the transition smoother? Most of our orchestrations are eternal, so breaking and re-adding them would be problematic for us.

[sophiatev]

@ohadtayl The terminating pending orchestrations behavior was introduced in 3.6.0, but this "not found" exception is definitely not expected, especially if you aren't purging orchestrations. I wonder if somehow this is related to the use of continue-as-new. But either way, yes, please do try and reproduce it with information-level logging enabled for both DT.Core and Azure Storage.

In terms of migrating to 3.6.0 without hitting these issues, based on what you've said so far, you really shouldn't be hitting issues. I now worry that I did introduce a regression somehow with the new terminating pending orchestrations logic. So having information-level logs will be extremely helpful on our end. Hopefully we can determine the cause and work on a fix.