Enhanced worker crash handling with integrated crash telemetry #5412

raujaiswal · 2025-11-28T10:04:07Z

Context

Design Doc: https://microsoftapc.sharepoint.com/:w:/r/teams/ADOTasksandAgents/_layouts/15/Doc.aspx?sourcedoc=%7BDFED2317-D226-4EC9-A6AB-1E3E53D91934%7D&file=worker%20crash%20handling.docx&action=default&mobileredirect=true

#AB2333253

Description

Provide a concise summary of the changes introduced in this PR.

Risk Assessment (Low / Medium / High)

Assess the risk level and justify your assessment. For example: code path sensitivity, usage scope, or backward compatibility concerns.

Unit Tests Added or Updated (Yes / No)

Indicate whether unit tests were added or modified to reflect the changes.

Additional Testing Performed

List manual or automated tests performed beyond unit tests (e.g., integration, scenario, regression).

Change Behind Feature Flag (Yes / No)

Can this change be behine feature flag, if not why?

Tech Design / Approach

Design has been written and reviewed.
Any architectural decisions, trade-offs, and alternatives are captured.

Documentation Changes Required (Yes/No)

Indicate whether related documentation needs to be updated.

User guides, API specs, system diagrams, or runbooks are updated.

Logging Added/Updated (Yes/No)

Appropriate log statements are added with meaningful messages.
Logging does not expose sensitive data.
Log levels are used correctly (e.g., info, warn, error).

Telemetry Added/Updated (Yes/No)

Custom telemetry (e.g., counters, timers, error tracking) is added as needed.
Events are tagged with proper metadata for filtering and analysis.
Telemetry is validated in staging or test environments.

Rollback Scenario and Process (Yes/No)

Rollback plan is documented.

Dependency Impact Assessed and Regression Tested (Yes/No)

All impacted internal modules, APIs, services, and third-party libraries are analyzed.
Results are reviewed and confirmed to not break existing functionality.

- Added enhanced crash handling logic with feature flag control - Implemented dual-mode operation for Plan v7 vs Plan v8+ scenarios - Added forced completion for Plan v8+ worker crashes - Enhanced logging for crash detection and completion analysis - Added notifyServerOfWorkerCrash variable for clear intent - Maintained backward compatibility with original logic - Added comprehensive trace logging for debugging

…crosoft/azure-pipelines-agent into users/raujaiswal/worker-crash

raujaiswal · 2025-12-05T11:01:15Z

/azp run

azure-pipelines · 2025-12-05T11:01:30Z

Azure Pipelines successfully started running 1 pipeline(s).

raujaiswal · 2025-12-08T04:03:21Z

/azp run

azure-pipelines · 2025-12-08T04:03:35Z

Azure Pipelines successfully started running 1 pipeline(s).

dassayantan24 · 2025-12-08T10:18:50Z

src/Agent.Listener/JobDispatcher.cs

+                                        {
+                                            // Direct plan event reporting for Plan v8+ worker crashes
+                                            Trace.Warning($"Plan event reporting for Plan v8+ worker crash [JobId:{message.JobId}, PlanVersion:{message.Plan.Version}, ExitCode:{returnCode}, Result:{result}]");
+                                            await ReportJobCompletionEventAsync(message, result);


Why are we not calling CompleteJobRequestAsync inside?
Are we completely changing the agent’s behavior to not call CompleteJobRequestAsync for V8?

@copilot answer this question

Why not call CompleteJobRequestAsync?
Because in Plan v8+ the server-side lifecycle is event-driven. The correct, supported path is RaisePlanEventAsync(JobCompletedPlanEvent). Using CompleteJobRequestAsync (legacy PATCH) in v8 bypasses the orchestration engine

Are we completely changing behavior for v8?
No. Normal v8 flow remains unchanged (Worker sends the event). We only add a crash-aware fallback in the Listener. The preferred fallback is plan-event (RaisePlanEventAsync).

dassayantan24 · 2025-12-08T10:20:31Z

src/Agent.Listener/JobDispatcher.cs

+
+                                        Trace.Info($"Enhanced crash handling enabled - Normal completion crash analysis [JobId:{message.JobId}, PlanVersion:{message.Plan.Version}, IsPlanV8Plus:{isPlanV8Plus}, IsWorkerCrash:{isWorkerCrash}, ExitCode:{returnCode}, NeedsForcedCompletion:{isPlanV8Plus && isWorkerCrash}]");
+
+                                        if (isPlanV8Plus && isWorkerCrash)


What if isPlanV8Plus = true and isWorkerCrash = false? did we test this behaviour?

Yes, we tested this behaviour, in that case worker itself has already reported to server and on the listener side it will call CompleteJobRequestAsync func where it checks, if plan v8 is enabled then it will skip reporting.

dassayantan24 · 2025-12-08T10:28:46Z

src/Agent.Sdk/Knob/AgentKnobs.cs

+            nameof(EnhancedWorkerCrashHandling),
+            "If true, enables enhanced worker crash handling with forced completion for Plan v8+ scenarios where worker crashes cannot send completion events",
+            new EnvironmentKnobSource("ENHANCED_WORKER_CRASH_HANDLING"),
+            new BuiltInDefaultKnobSource("false"));


Is this intended, not to have RuntimeKnobSource?

On the Listener side, enabling RuntimeKnobSource is not possible once the listener has started. Instead, I will replace this mechanism with a server API call, removing the dependency on the Agent knob.

updated with runtime control in Agent.cs file

sanjuyadav24 · 2025-12-08T10:12:46Z

src/Agent.Listener/JobDispatcher.cs

+                                    if (enhancedworkercrashhandlingenabled)
+                                    {
+                                        bool isPlanV8Plus = PlanUtil.GetFeatures(message.Plan).HasFlag(PlanFeatures.JobCompletedPlanEvent);
+                                        bool isWorkerCrash = !TaskResultUtil.IsValidReturnCode(returnCode);


should we rename this to worker failed to send status to server?
as in that case this can be extended to any events in future

src/Agent.Listener/JobDispatcher.cs

sanjuyadav24 · 2025-12-08T10:22:16Z

src/Agent.Listener/JobDispatcher.cs

+                Uri jobServerUrl = systemConnection.Url;
+
+                // Make sure SystemConnection Url match Config Url base for OnPremises server
+                if (!message.Variables.ContainsKey(Constants.Variables.System.ServerType) ||


this logic is similar to the logic in method LogWorkerProcessUnhandledException, could we please check if we can refactor this

Thank you for suggestion, refactored the part.

sanjuyadav24 · 2025-12-08T10:22:53Z

src/Agent.Sdk/Knob/AgentKnobs.cs

+        public static readonly Knob EnhancedWorkerCrashHandling = new Knob(
+            nameof(EnhancedWorkerCrashHandling),
+            "If true, enables enhanced worker crash handling with forced completion for Plan v8+ scenarios where worker crashes cannot send completion events",
+            new EnvironmentKnobSource("ENHANCED_WORKER_CRASH_HANDLING"),


please convert this to runtime knob

sanjuyadav24 · 2025-12-08T10:34:14Z

src/Agent.Listener/JobDispatcher.cs

                                        detailInfo = string.Join(Environment.NewLine, workerOutput);
                                        Trace.Info($"Return code {returnCode} indicate worker encounter an unhandled exception or app crash, attach worker stdout/stderr to JobRequest result.");
                                        await LogWorkerProcessUnhandledException(message, detailInfo, agentCertManager.SkipServerCertificateValidation);
+


shouldn't we have sendEvent to server method here as well, as here only we are logging if worker is terminated due to unhandled exception

Copilot · 2025-12-09T08:04:50Z

@raujaiswal I've opened a new pull request, #5423, to work on those changes. Once the pull request is ready, I'll request review from you.

…crosoft/azure-pipelines-agent into users/raujaiswal/worker-crash

raujaiswal · 2025-12-10T08:48:04Z

/azp run

azure-pipelines · 2025-12-10T08:48:17Z

Azure Pipelines successfully started running 1 pipeline(s).

raujaiswal · 2025-12-10T10:21:31Z

/azp run

azure-pipelines · 2025-12-10T10:21:46Z

Azure Pipelines successfully started running 1 pipeline(s).

sanjuyadav24 · 2025-12-10T10:29:43Z

src/Agent.Listener/JobDispatcher.cs

-                            //server URI (which was set at config time)
-                            jobServerUrl = result;
-                        }
+                        await jobServer.RaisePlanEventAsync(message.Plan.ScopeIdentifier, message.Plan.PlanType, message.Plan.PlanId, jobCompletedEvent, CancellationToken.None);


should we add a retry here? I do see we have a retry in Jobrunner for RaisePlanEventAsync

azure-pipelines-bot and others added 3 commits November 28, 2025 15:33

agentknob changes for worker crash

7a5bb52

Merge branch 'master' into users/raujaiswal/worker-crash

c05c057

raujaiswal marked this pull request as ready for review December 2, 2025 06:47

raujaiswal requested review from a team as code owners December 2, 2025 06:47

azure-pipelines-bot and others added 3 commits December 3, 2025 14:45

enhanced worker crash reporting for listner and added telemetry

f2f4138

Merge branch 'users/raujaiswal/worker-crash' of https://github.com/mi…

95aedec

…crosoft/azure-pipelines-agent into users/raujaiswal/worker-crash

Merge branch 'master' into users/raujaiswal/worker-crash

f0a3af4

raujaiswal marked this pull request as draft December 3, 2025 09:52

azure-pipelines-bot and others added 4 commits December 4, 2025 12:05

removed feature flag telemetry

e18c663

Merge branch 'users/raujaiswal/worker-crash' of https://github.com/mi…

425e29b

…crosoft/azure-pipelines-agent into users/raujaiswal/worker-crash

cleanup the code

af10a40

Merge branch 'master' into users/raujaiswal/worker-crash

1d3c339

raujaiswal marked this pull request as ready for review December 5, 2025 09:39

raujaiswal changed the title ~~Implement enhanced worker crash handling in JobDispatcher~~ Enhanced worker crash handling in JobDispatcher with added crash telemetry Dec 5, 2025

raujaiswal changed the title ~~Enhanced worker crash handling in JobDispatcher with added crash telemetry~~ Enhanced worker crash handling with added crash telemetry Dec 5, 2025

raujaiswal changed the title ~~Enhanced worker crash handling with added crash telemetry~~ Enhanced worker crash handling with integrated crash telemetry Dec 5, 2025

raujaiswal added the internal label Dec 5, 2025

raujaiswal added the misc Miscellaneous Changes label Dec 8, 2025

dassayantan24 reviewed Dec 8, 2025

View reviewed changes

sanjuyadav24 reviewed Dec 8, 2025

View reviewed changes

Copilot AI mentioned this pull request Dec 9, 2025

Clarify Plan v8+ worker crash completion behavior in response to review feedback #5423

Closed

azure-pipelines-bot added 4 commits December 10, 2025 14:02

refactored the chnages

1944d29

Merge branch 'users/raujaiswal/worker-crash' of https://github.com/mi…

eedd2e4

…crosoft/azure-pipelines-agent into users/raujaiswal/worker-crash

refactored the changes

ea914c3

refactored the changes

50fe7d7

azure-pipelines-bot added 3 commits December 10, 2025 14:38

refactored the changes

8e617cf

refactored the changes

f2699ce

refactored the changes

e67f638

sanjuyadav24 reviewed Dec 10, 2025

View reviewed changes


		Trace.Info($"Enhanced crash handling enabled - Normal completion crash analysis [JobId:{message.JobId}, PlanVersion:{message.Plan.Version}, IsPlanV8Plus:{isPlanV8Plus}, IsWorkerCrash:{isWorkerCrash}, ExitCode:{returnCode}, NeedsForcedCompletion:{isPlanV8Plus && isWorkerCrash}]");

		if (isPlanV8Plus && isWorkerCrash)

Enhanced worker crash handling with integrated crash telemetry #5412

Are you sure you want to change the base?

Enhanced worker crash handling with integrated crash telemetry #5412

Conversation

raujaiswal commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Context

Description

Risk Assessment (Low / Medium / High)

Unit Tests Added or Updated (Yes / No)

Additional Testing Performed

Change Behind Feature Flag (Yes / No)

Tech Design / Approach

Documentation Changes Required (Yes/No)

Logging Added/Updated (Yes/No)

Telemetry Added/Updated (Yes/No)

Rollback Scenario and Process (Yes/No)

Dependency Impact Assessed and Regression Tested (Yes/No)

Uh oh!

raujaiswal commented Dec 5, 2025

Uh oh!

azure-pipelines bot commented Dec 5, 2025

Uh oh!

raujaiswal commented Dec 8, 2025

Uh oh!

azure-pipelines bot commented Dec 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raujaiswal Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Dec 9, 2025

Uh oh!

raujaiswal commented Dec 10, 2025

Uh oh!

azure-pipelines bot commented Dec 10, 2025

Uh oh!

raujaiswal commented Dec 10, 2025

Uh oh!

azure-pipelines bot commented Dec 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

raujaiswal commented Nov 28, 2025 •

edited

Loading

raujaiswal Dec 10, 2025 •

edited

Loading