Problem
The current DML two-stage ack implementation in storage sink has three correctness/semantics gaps:
-
PostEnqueue is triggered too early in cloud storage sink.
- It is currently fired in
dmlWriters.AddDMLEvent right after pushing into msgCh.
- This means "enqueue" only guarantees entering the first unbounded internal queue, not acceptance into the downstream write pipeline.
-
DMLEvent.PostEnqueue is not concurrency-safe.
PostEnqueue can be called both from enqueue path and via PostFlush fallback.
- The guard is a plain boolean, so concurrent calls can cause a data race and potentially double invocation.
-
FilterDMLEvent may lose enqueue callbacks.
- When a filtered event is rebuilt as a new
DMLEvent, PostTxnEnqueued is not migrated.
- This can silently drop enqueue-stage callbacks on filtered paths.
Expected behavior
- Enqueue-stage callback should represent acceptance into sink's write pipeline (after encoding/defragment dispatch), not just source queue insertion.
PostEnqueue should be exactly-once and race-free under concurrent calls.
- Filtering/rebuilding should preserve callback semantics (
PostTxnEnqueued and related state).
Related PR
Problem
The current DML two-stage ack implementation in storage sink has three correctness/semantics gaps:
PostEnqueueis triggered too early in cloud storage sink.dmlWriters.AddDMLEventright after pushing intomsgCh.DMLEvent.PostEnqueueis not concurrency-safe.PostEnqueuecan be called both from enqueue path and viaPostFlushfallback.FilterDMLEventmay lose enqueue callbacks.DMLEvent,PostTxnEnqueuedis not migrated.Expected behavior
PostEnqueueshould be exactly-once and race-free under concurrent calls.PostTxnEnqueuedand related state).Related PR