Out of Memory Exception when Activity Trigger Function is called.

[fahadash]

Hello,
I am running into an issue where my Azure Durable Function run under Azure Resource with In-Process Model. My application uses the following versions of the packages

Microsoft.Azure.WebJobs.Extensions.DurableTask 2.10.0
Microsoft.Azure.DurableTask.AzureStorage 1.14.0
Microsoft.Azure.DurableTask.Core 2.13.0
Microsoft.Azure.WebJobs.Extensions.DurableTask.Analyzers 0.5.0
Microsoft.DurableTask.Sidecar.Protobuf 1.0.0

My Orchestrator calls an activity which calls another activity with a huge array in the parameter, and that last activity function never runs and I see an OutOfMemoryException in the Application Insights logs (stack trace at the end).

I have looked at the relevant code in this repo and it looks like the storage queue has that array saved as JSON and the activity function fails to even get triggered because the upstream logic that puts the array back together from JSON fails. Please correct me if this is wrong since I am not very familiar with the business logic of Durable Task Framework.

My questions for this team are

• Will upgrading to the latest version of Durable Task Framework packages solve my issue?
• I am planning on saving the JSON into a blob and passing the activity trigger function with a SAS URL of the blob, and do my own deserialization inside the activity function, how does this idea sound?
• How to best diagnose this issue, any guide with the setup instructions for the dev environment would be nice

An error occurred while processing messages on functionsappname-workitems: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at Newtonsoft.Json.JsonTextReader.PrepareBufferForReadData(Boolean append, Int32 charsRequired)
   at Newtonsoft.Json.JsonTextReader.ReadData(Boolean append, Int32 charsRequired)
   at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char quote)
   at Newtonsoft.Json.JsonTextReader.ParseString(Char quote, ReadType readType)
   at Newtonsoft.Json.JsonTextReader.ReadStringValue(ReadType readType)
   at Newtonsoft.Json.JsonTextReader.ReadAsString()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at DurableTask.AzureStorage.Utils.DeserializeFromJson[T](JsonSerializer serializer, String jsonString) in /_/src/DurableTask.AzureStorage/Utils.cs:line 200
   at DurableTask.AzureStorage.MessageManager.DeserializeQueueMessageAsync(QueueMessage queueMessage, String queueName) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 148
   at DurableTask.AzureStorage.Messaging.WorkItemQueue.GetMessageAsync(CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Messaging/WorkItemQueue.cs:line 47

[cgillum]

Hi @fahadash. Your analysis sounds accurate. Here are some responses to your questions:

Will upgrading to the latest version of Durable Task Framework packages solve my issue?

Not likely. The problem seems specific to your activity return payload, and any updates we've made to the library likely won't help prevent the out-of-memory exceptions you're seeing.

I am planning on saving the JSON into a blob and passing the activity trigger function with a SAS URL of the blob, and do my own deserialization inside the activity function, how does this idea sound?

Yes, this idea sounds reasonable and is a best practice when dealing with large payloads. The main question in my mind is whether the activity function can successfully deserialize the payload without running into an out-of-memory exception. Perhaps you will have better luck using a different set of JSON deserialization settings.

How to best diagnose this issue, any guide with the setup instructions for the dev environment would be nice

It seems to me that you've already diagnosed the issue. I'm not an expert on this, but another way you can diagnose internal failures like these (besides logging) is to reproduce the problem locally in your function app (using Visual Studio) with a debugger attached, configuring your debugger to break on all exceptions, and ensuring that "Just My Code" is disabled so that you can break on exceptions thrown from outside your function app code, like in DurableTask.AzureStorage. This package should have embedded source and symbols already, so you'll be able to see the exact line of code that's failing and even see the string payload that is attempting to be deserialized.