Add gRPC connection resiliency

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: Bernd Verst

CHANGELOG.md

@@ -12,16 +12,15 @@ ADDED
 - Added `GrpcChannelOptions` and `GrpcRetryPolicyOptions` for configuring
   gRPC transport behavior, including message-size limits, keepalive settings,
   and channel-level retry policy service configuration.
-- Added `GrpcWorkerResiliencyOptions` and `GrpcClientResiliencyOptions` for
-  configuring public gRPC reconnect, hello timeout, and channel recreation
-  thresholds.
 - Added optional `channel` and `channel_options` parameters to
   `TaskHubGrpcClient`, `AsyncTaskHubGrpcClient`, and `TaskHubGrpcWorker` to
   support pre-configured channel passthrough and low-level gRPC channel
   customization.
-- Added optional `resiliency_options` parameters to `TaskHubGrpcClient`,
-  `AsyncTaskHubGrpcClient`, and `TaskHubGrpcWorker` so applications can pass
-  gRPC resiliency settings through constructor APIs.
+- Added `GrpcWorkerResiliencyOptions` and `GrpcClientResiliencyOptions`, plus
+  `resiliency_options` constructor parameters on `TaskHubGrpcClient`,
+  `AsyncTaskHubGrpcClient`, and `TaskHubGrpcWorker`, to configure hello
+  deadlines, silent-disconnect detection, reconnect backoff, and channel
+  recreation thresholds for SDK-managed gRPC connections.
 - Added `get_orchestration_history()` and `list_instance_ids()` to the sync
   and async gRPC clients.
 - Added in-memory backend support for `StreamInstanceHistory` and
@@ -30,18 +29,13 @@ ADDED
 
 FIXED
 
-- Hardened `TaskHubGrpcWorker` reconnect handling so configured hello timeouts
-  apply on fresh connections, received work items reset failure tracking,
-  SDK-owned channels are cleaned up on shutdown and full resets, and
-  caller-owned channels are never recreated or closed during worker reconnects.
-- Fixed sync `TaskHubGrpcClient` transport resiliency so SDK-owned channels are
-  recreated after repeated transport failures while long-poll timeout
-  deadlines, successful replies, and application-level RPC errors reset the
-  failure tracker.
-- Fixed async `AsyncTaskHubGrpcClient` transport resiliency so SDK-owned
-  channels are recreated after repeated transport failures while long-poll
-  timeout deadlines, successful replies, and application-level RPC errors
-  reset the failure tracker.
+- Improved `TaskHubGrpcWorker` recovery from stale or disconnected gRPC streams
+  so configured hello timeouts apply on fresh connections, received work resets
+  failure tracking, SDK-owned channels are refreshed and cleaned up safely, and
+  caller-owned channels are never recreated or closed during reconnects.
+- Improved sync and async gRPC clients so repeated transport failures recreate
+  SDK-owned channels, while long-poll deadlines, successful replies, and
+  application-level RPC errors do not trigger unnecessary channel replacement.
 
 ## v1.4.0
 

docs/superpowers/specs/2026-04-23-grpc-resiliency-design.md

@@ -135,35 +135,39 @@ The monitor reports one of these outcomes:
 The outer worker loop uses those outcomes as follows:
 
 - `message_received`: reset health counters
-- `graceful_close_before_first_message`: count as channel poison
-- `graceful_close_after_message`: reconnect immediately without poisoning the
-  channel
+- `graceful_close_before_first_message`: immediately reset the current stream
+  and force a fresh SDK-owned channel on the next connect attempt
+- `graceful_close_after_message`: immediately reset the current stream and
+  reconnect without incrementing the transport-failure counter
 - `silent_disconnect`: count as channel poison
 - `shutdown`: exit cleanly
 
-This keeps rolling upgrades and normal peer-driven reconnects from being
-treated the same as a stale half-open stream.
+This keeps rolling upgrades and normal peer-driven reconnects from inflating
+the failure threshold while still forcing SDK-owned workers to establish a
+fresh channel after graceful stream closures.
 
 #### Failure counting and recreation
 
 The worker increments the consecutive-failure counter only for
 transport-shaped failures:
 
 - `UNAVAILABLE`
-- `Hello` `DEADLINE_EXCEEDED`
+- `DEADLINE_EXCEEDED`
 - explicit silent-disconnect timeout
-- graceful stream close before the first message
 
 It does not increment the counter for errors that channel recreation is
 unlikely to fix, such as:
 
 - `UNAUTHENTICATED`
 - `NOT_FOUND`
 - orchestration or activity execution failures
+- graceful stream closures before or after work items
 
 When the threshold is reached and the worker owns the channel, it recreates the
-channel and stub. When the worker does not own the channel, it keeps retrying
-the existing transport and logs that the channel could not be recreated.
+channel and stub. Graceful stream closures also force an immediate fresh
+SDK-owned channel even though they do not increment the threshold. When the
+worker does not own the channel, it keeps retrying the existing transport and
+logs that the channel could not be recreated.
 
 ### Client behavior
 
@@ -284,9 +288,9 @@ Add focused unit tests for the new behavior.
 
 - hello deadline failure counts toward recreation
 - silent-disconnect timeout is detected and classified
-- graceful close before the first message poisons the channel
-- graceful close after a message triggers reconnect without poisoning
-- user-supplied channels are not recreated
+- graceful stream closes force a fresh SDK-owned connection without increasing
+  the failure counter
+- user-supplied channels are not recreated or closed
 
 ### Client tests
 

durabletask-azuremanaged/CHANGELOG.md

@@ -11,10 +11,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `DurableTaskSchedulerClient`, `AsyncDurableTaskSchedulerClient`, and
   `DurableTaskSchedulerWorker` to allow combining custom gRPC interceptors with
   DTS defaults and to support pre-configured/customized gRPC channels.
-- Added optional `resiliency_options` parameters to
+- Added pass-through `resiliency_options` support on
   `DurableTaskSchedulerClient`, `AsyncDurableTaskSchedulerClient`, and
-  `DurableTaskSchedulerWorker` so applications can pass gRPC resiliency
-  settings through their constructors.
+  `DurableTaskSchedulerWorker` so Azure Managed applications can use the core
+  SDK's gRPC resiliency option types through their constructors.
 - Added `workerid` gRPC metadata on Durable Task Scheduler worker calls for
   improved worker identity and observability.
 - Improved sync access token refresh concurrency handling to avoid duplicate

pyproject.toml

@@ -51,6 +51,7 @@ include = ["durabletask", "durabletask.*"]
 minversion = "6.0"
 testpaths = ["tests"]
 asyncio_mode = "auto"
+addopts = "--import-mode=importlib"
 markers = [
     "azurite: tests that require Azurite (local Azure Storage emulator)",
 ]