enhancement(topology): add per-sink authoritative acknowledgement control

[connoryy]

Summary

Implements per-sink "authoritative" acknowledgement control, as proposed in RFC 6517 but never built.

When a source has end-to-end acknowledgements enabled, it currently waits for all connected sinks to finalize events before acknowledging — including sinks that don't need delivery guarantees. This PR adds an authoritative field to the sink acknowledgements config. When at least one sink sets authoritative: true, only authoritative sinks participate in the ack chain. Non-authoritative sinks have their event finalizers stripped at the fan-out level, so the source's BatchNotifier resolves as soon as all authoritative sinks have processed the events.

When no sink sets authoritative: true, behavior is identical to current Vector — zero breaking changes.

Motivation

This addresses a long-standing design limitation where a slow, failing, or unreachable non-critical sink blocks acknowledgements for critical sinks:

• Sources should only wait to ack for sinks with acks enabled vectordotdev/vector#7369: sources should only wait for sinks with acks enabled (open 5+ years)
• sink failure dependency vectordotdev/vector#14828: Elasticsearch down blocks S3 delivery
• [Question] Backpressure propagation poses risk for all components (anti-pattern - cascading failure: single component failures leads to all components failure) vectordotdev/vector#19938: 10% packet loss on one sink throttles all sinks
• 2 sinks with buffer.when_full drop_newest/block doesn't work as expected.  vectordotdev/vector#20764: non-ack sink unreachable blocks ack-enabled sink
• Log duplication caused by acknowledgements vectordotdev/vector#22384: GCS failure causes log duplication in Datadog due to blocked acks

vectordotdev#10983 and vectordotdev#11346 moved ack config from sources to sinks but explicitly did not change the wait-for-all runtime behavior. This PR completes that work.

Approach

Config

A single Option<bool> field added to AcknowledgementsConfig:

pub struct AcknowledgementsConfig {
    enabled: Option<bool>,
    authoritative: Option<bool>,  // NEW — defaults to false per RFC 6517
}

All 63 sink configs pick this up automatically via the shared AcknowledgementsConfig struct. No individual sink changes needed.

Runtime: fan-out level stripping

Non-authoritative sinks have their EventFinalizer references stripped at the fan-out point — where events are distributed to downstream components — rather than at the sink input stream. This is critical because stateful transforms (like aggregate, reduce) between the fan-out and the sink can hold events for extended periods. Sink-level stripping would be too late — the transform would hold finalizer references, blocking the BatchNotifier.

The stripping calls events.take_finalizers() in Sender::flush() for non-authoritative senders. The dropped finalizers call update_batch(EventStatus::Dropped), which is a no-op on BatchStatus.

Design decision: stripping vs RFC's "immediate status update"

RFC 6517 proposed having authoritative sinks immediately set EventStatus::Recorded on finalization. This PR takes a different approach: strip finalizers from non-authoritative paths entirely. The semantic guarantee is identical (source only waits for authoritative sinks), but stripping is simpler:

• No changes to the finalization state machine
• No new status variants
• Works with any sink type without per-sink code changes
• Naturally handles stateful transforms (finalizers never enter the non-authoritative path)

Backwards compatibility

The authoritative field defaults to false. The feature activates only when at least one sink explicitly sets authoritative: true. When inactive:

• compute_authoritative_components() returns None
• No strip flags are set on any fan-out sender
• The if self.strip_finalizers branch in Sender::flush() is never taken
• Behavior is identical to current Vector

This is enforced by the any_sink_explicitly_authoritative guard in the topology builder and verified by two dedicated backwards-compatibility tests.

Performance

When inactive (no authoritative sinks): zero per-event overhead. The strip_finalizers bool check in Sender::flush() is always false and branch-predicted away. The compute_authoritative_components() BFS runs once at topology build time.

When active: the stripping work (take_finalizers() + Arc::make_mut + ref count operations) is the same work that would happen at the sink regardless — it's just done earlier. Non-authoritative transforms and buffers no longer carry unnecessary Arc<EventFinalizer> references, slightly reducing memory pressure.

Vector configuration

sinks:
  s3_archive:
    type: aws_s3
    inputs: ["transform"]
    acknowledgements:
      enabled: true
      authoritative: true
      # Source waits for this sink before acknowledging

  console_debug:
    type: console
    inputs: ["transform"]
    acknowledgements:
      enabled: true
      # No authoritative field — defaults to false
      # Source does NOT wait for this sink

  loki_monitoring:
    type: loki
    inputs: ["transform"]
    # No acknowledgements block at all
    # Source does NOT wait for this sink

How did you test this PR?

28 tests across 4 categories:

Config defaults and merging (2 tests):

• authoritative defaults to false for all combinations of enabled/authoritative/unset
• Local authoritative overrides global; global falls through when local is None

Graph walk / BFS (9 tests):

• Returns None when no sink is explicitly authoritative (backwards compat)
• Returns None for empty config
• Linear pipeline: all components on auth path included
• Fan-out: shared upstream marked authoritative; dedicated non-auth branch excluded
• Multiple authoritative sinks: both paths included
• Chain of transforms: entire chain included
• Diamond topology: all components included
• Multi-component realistic config with no authoritative: None

Fan-out stripping mechanics (2 new + 8 existing):

• Non-auth sender stripped, auth sender keeps finalizers, BatchNotifier resolves only from auth path
• Strip flag survives pause → replace cycle
• All 8 existing fanout tests pass unchanged

End-to-end topology (2 tests):

• authoritative_sink_controls_ack_chain: source acks when auth sink drains, even though non-auth sink holds finalizers forever
• authoritative_sink_blocks_ack_when_not_drained: source does NOT ack when auth sink holds finalizers, even though non-auth sink drains normally

All tests: cargo check clean, cargo test -p vector-core --lib fanout::tests (10 pass), cargo test -p vector-core config::test (7 pass), cargo test -p vector config::authoritative_tests (9 pass), cargo test -p vector topology::test::authoritative_acks (2 pass).

Change Type

• Bug fix
• New feature
• Dependencies
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

• Closes: Sources should only wait to ack for sinks with acks enabled vectordotdev/vector#7369
• Related: Move end-to-end acknowledgement setting to sinks vectordotdev/vector#10983
• Related: enhancement(sinks): Add acknowledgement configuration to sinks vectordotdev/vector#11346
• Related: sink failure dependency vectordotdev/vector#14828
• Related: [Question] Backpressure propagation poses risk for all components (anti-pattern - cascading failure: single component failures leads to all components failure) vectordotdev/vector#19938
• Related: 2 sinks with buffer.when_full drop_newest/block doesn't work as expected.  vectordotdev/vector#20764
• Related: Log duplication caused by acknowledgements vectordotdev/vector#22384