Exception: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The specified blob does not exist

[yell0wfl4sh]

We are noticing several error in our logs with the following trace:

An error occurred while processing messages on taskhubsimulation-control-00: DurableTask.AzureStorage.Storage.DurableTaskStorageException: The specified blob does not exist.
 ---> Microsoft.WindowsAzure.Storage.StorageException: The specified blob does not exist.
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteAsyncInternal[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadRangeToStreamAsync(Stream target, Nullable`1 offset, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, IProgress`1 progressHandler, CancellationToken cancellationToken)
   at DurableTask.AzureStorage.Storage.AzureStorageClient.WrapFunctionWithReturnType(Func`3 storageRequest, OperationContext context, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 158
   at DurableTask.AzureStorage.TimeoutHandler.ExecuteWithTimeout[T](String operationName, String account, AzureStorageOrchestrationServiceSettings settings, Func`3 operation, AzureStorageOrchestrationServiceStats stats, String clientRequestId)
   at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 149
Request Information
RequestID:3e63939b-101e-0074-2621-94552e000000
RequestDate:Sun, 10 Jul 2022 05:50:48 GMT
StatusMessage:The specified blob does not exist.
ErrorCode:BlobNotFound
ErrorMessage:The specified blob does not exist.
RequestId:3e63939b-101e-0074-2621-94552e000000
Time:2022-07-10T05:50:48.5116295Z

   --- End of inner exception stack trace ---
   at DurableTask.AzureStorage.Storage.AzureStorageClient.MakeStorageRequest[T](Func`3 storageRequest, String accountName, String operationName, String clientRequestId, Boolean force) in /_/src/DurableTask.AzureStorage/Storage/AzureStorageClient.cs:line 149
   at DurableTask.AzureStorage.Storage.Blob.DownloadToStreamAsync(MemoryStream target) in /_/src/DurableTask.AzureStorage/Storage/Blob.cs:line 101
   at DurableTask.AzureStorage.MessageManager.DownloadAndDecompressAsBytesAsync(Blob blob) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 211
   at DurableTask.AzureStorage.MessageManager.DownloadAndDecompressAsBytesAsync(String blobName) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 193
   at DurableTask.AzureStorage.MessageManager.DeserializeQueueMessageAsync(QueueMessage queueMessage, String queueName) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 148
   at DurableTask.AzureStorage.Messaging.ControlQueue.<>c__DisplayClass11_0.<<GetMessagesAsync>b__0>d.MoveNext() in /_/src/DurableTask.AzureStorage/Messaging/ControlQueue.cs:line 150

We are trying to understand what this might be related to and if this is something that we should be concerned about as we are seeing any issues/failure per se.

Thanks,
Yash

[cgillum]

Looking at the stack trace, this appears to be a case where a queue message references some large payload that has been stored in blob storage. Message contents are uploaded to blob storage automatically when a task input or output exceeds 45KB. However, it seems the particular blob referenced by this queue message no longer exists. Is there some process or configuration on your storage account that may have deleted some blobs that it shouldn't have?

This error likely will result in two issues:

1. You may continuously receive this message, creating noise in your logs
2. If that message was required by a running orchestration instance, then that instance will get stuck. However, depending on what kind of message this is, that instance might already be completed/gone.

[yell0wfl4sh]

Thanks for reply,

We have been getting this error for long time without any actual issue related to this, so this should not be a major concern.
At the same time, we would like to get rid of this noise as we get more than 20k error logs for this on daily basis. One way to block this is to use regex on our side while logging and prevent this log from getting in Kusto. I am interested in knowing if there is any other elegant way provided by DTF to mute specific noisy logs?

[cgillum]

If you want to permanently get rid of this, then one thing you could try is draining all (valid) messages from your queues until only these invalid messages are left. You could then manually purge them from the queue. It's a manual process for sure, but it would be the proper way to fix the issue rather than hiding logs.

I'm not aware of what options are available in terms of this kind of specific log filtering. It partly depends on whether you're using ETW logging or the ILoggerFactory support. If the latter, then you'd need to consult the documentation on whether/how logs can be filtered when using ILogger and ILoggerFactory.

We have this issue too.
We also experienced "frozen" orchestrators in the instances table from time to time. Sounds like this could be the reason for that.

Could this happen due to scaling scenarios? So one instance of the function app did already delete the blob when the other instance tried to read it?

@cgillum can you please explain or provide a link with instruction. I don't understand what you mean with "draining all messages". And how (and where) could I identify invalid messages?

[davidmrdavid]

Hi @gha-zund, let me try to fill in the gaps here.

We also experienced "frozen" orchestrators in the instances table from time to time. Sounds like this could be the reason for that.
Perhaps, though to be sure we'd need concrete instanceIDs and timeranges to investigate. If this happens again, you're welcome to open a GitHub issue with us for an investigation.

Could this happen due to scaling scenarios? So one instance of the function app did already delete the blob when the other instance tried to read it?

I'm not sure why scaling would have to do with this. However, if a blob got deleted while there was still an orchestrator that depends on it on a future invocation, I think that could cause this exception. Again, if you believe this is happening in your app, a separate GitHub issue with the appropriate details would allow us to get to the bottom of it

[yell0wfl4sh]

@cgillum: can you please add how to exactly perform these steps? We want to keep our service up and not cause any issue with running orchestrations.

[roycornelissen]

I know this is already an old issue, but we ran into exactly the same situation this week. This is a message that the runtime has been trying to dequeue thousands of times, but it fails with this exception. I am unable to find this message in any of the storage queus, tables etc. that Functions uses though... Were you able to resolve this @yell0wfl4sh ?

[etortec]

We also experienced the missing blob problem. In our case the problem appeared specifically for a durable function in which we use context.ContinueAsNew, and it seems only after we updated to Microsoft.Azure.WebJobs.Extensions.DurableTask 2.9.5. This new dependency version seems to contain a change related to how blobs are handled, and we suspect this was related to our problem.

We solved our problem by replacing our calls to context.ContinueAsNew by context.StartNewOrchestration, after which no new failing instances were reported.

Because we log to application insights we were able to determine the existing affected instances by running the query:

customEvents
| where name contains 'OrchestrationProcessingFailure'
| where customDimensions.Details startswith_cs "DurableTask.AzureStorage.Storage.DurableTaskStorageException: The specified blob does not exist."
| project
    instanceId = tostring(customDimensions.InstanceId),
    ExecutionId = tostring(customDimensions.ExecutionId)
| summarize count() by instanceId

After which we manually fixed these instances in table storage to allow them to complete and thus stop throwing the exception.

[cgillum]

I believe this issue has been fixed as part of #1189, and the fix will go out with the next version of the Durable extension.