Skip to content

[WIP]fix: preserve soft-delete flags in workflow task ACK writes.#4620

Open
leozhang2018 wants to merge 1 commit into
koderover:mainfrom
leozhang2018:workflow-deletion-ack
Open

[WIP]fix: preserve soft-delete flags in workflow task ACK writes.#4620
leozhang2018 wants to merge 1 commit into
koderover:mainfrom
leozhang2018:workflow-deletion-ack

Conversation

@leozhang2018
Copy link
Copy Markdown
Collaborator

@leozhang2018 leozhang2018 commented Apr 10, 2026

What this PR does / Why we need it:

The workflow controller loads a WorkflowTask document into memory at startup and holds a reference (c.workflowTask) for the entire lifetime of the run. When a workflow is deleted while a task is still executing, DeleteWorkflowV4 soft-deletes all associated task documents by setting is_deleted: true and is_archived: true in MongoDB.

However, updateWorkflowTask (the ACK callback invoked after every job/stage transition) writes the full in-memory task object back to MongoDB using $set. Because the in-memory object was loaded before the workflow was deleted, its IsDeleted and IsArchived fields are both false. Every ACK call therefore silently undoes the soft-delete, resetting is_deleted to false and making the task reappear in the UI as "running" — even after the workflow has been removed.

This also meant that a soft-deleted task could never be cleared by InitQueue on service restart (which queries InCompletedTasks filtered by is_deleted: false), nor by any automatic cleanup path, leaving a permanent "zombie" running indicator for users.

What is changed and how it works?

Before writing the in-memory task back to MongoDB, sync IsDeleted and IsArchived from the freshly-read database copy (taskInColl) that is already fetched at the top of updateWorkflowTask. This ensures that a soft-delete performed concurrently (e.g., by DeleteWorkflowV4) is never overwritten by a stale ACK.

Does this PR introduce a user-facing change?

  • API change
  • database schema change
  • upgrade assistant change
  • change in non-functional attributes such as efficiency or availability
  • fix of a previous issue

This change is Reviewable

Signed-off-by: leozhang2018 <leozhang2018@gmail.com>
@leozhang2018 leozhang2018 added the bug Something isn't working label Apr 10, 2026
@landylee007 landylee007 changed the title fix: preserve soft-delete flags in workflow task ACK writes. [WIP]fix: preserve soft-delete flags in workflow task ACK writes. May 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working service/aslan

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants