feat: update alert & implement alert.destination.disabled

[alexluong]

Changes

Alert Payload Schema

• Restructure consecutive_failures into a nested object with current, max, threshold
• Remove will_disable field (threshold == 100 implies disable)
• Remove custom MarshalJSON methods in favor of standard json.Marshal
• Add tenant_id to top-level of alert data
• Expand AlertDestination with filter, metadata, updated_at
• Simplify DeliveryAttempt struct: pass Event, Destination, Attempt directly

New alert.destination.disabled Callback

• Sent when destination is auto-disabled after consecutive failures
• Invariant check ensures DisabledAt is set on returned destination
• Uses a reason field (e.g., "consecutive_failure") instead of embedding trigger-specific data, keeping the payload extensible for future disable mechanisms (e.g., error rate)
• When the threshold is reached, both alert.destination.disabled and alert.destination.consecutive_failure (with threshold: 100) are sent. The disabled alert is sent first, but ordering is not guaranteed.

Consecutive Failure Alert at Threshold 100

• The destination in the consecutive failure alert reflects the post-disable state (includes disabled_at)

Error Handling

• Notifications are best-effort (logged, not propagated)
• DestinationDisabler returns disabled destination for timestamp consistency

Notification Delivery

Alert notifications are sent synchronously via HTTP within HandleAttempt, which itself is called asynchronously from the delivery pipeline (go h.handleAlertAttempt(...)). This means notifications don't block event delivery, but a slow notification (e.g., the disabled alert) will delay subsequent notifications (e.g., the consecutive failure alert) within the same handler call.

Future consideration: If notification latency becomes an issue, we could decouple the two notifications — either by sending them concurrently or by introducing a notification queue. For now, the synchronous approach keeps the implementation simple and the best-effort error handling makes the current behavior acceptable.

---

Alert Payloads

event is a full models.Event and attempt is a full models.Attempt.

alert.destination.consecutive_failure

{
  "topic": "alert.destination.consecutive_failure",
  "timestamp": "2025-01-15T10:30:00Z",
  "data": {
    "tenant_id": "tenant_123",
    "event": { ... },
    "attempt": { ... },
    "consecutive_failures": {
      "current": 10,
      "max": 20,
      "threshold": 50
    },
    "destination": {
      "id": "dest_xyz",
      "tenant_id": "tenant_123",
      "type": "webhook",
      "topics": ["*"],
      "filter": {},
      "config": {},
      "metadata": {},
      "created_at": "2025-01-01T00:00:00Z",
      "updated_at": "2025-01-01T00:00:00Z",
      "disabled_at": null
    }
  }
}

Note: At threshold: 100, destination.disabled_at will be set (reflects post-disable state).

alert.destination.disabled

{
  "topic": "alert.destination.disabled",
  "timestamp": "2025-01-15T10:30:00Z",
  "data": {
    "tenant_id": "tenant_123",
    "disabled_at": "2025-01-15T10:30:00Z",
    "reason": "consecutive_failure",
    "event": { ... },
    "attempt": { ... },
    "destination": {
      "id": "dest_xyz",
      "tenant_id": "tenant_123",
      "type": "webhook",
      "topics": ["*"],
      "filter": {},
      "config": {},
      "metadata": {},
      "created_at": "2025-01-01T00:00:00Z",
      "updated_at": "2025-01-15T10:30:00Z",
      "disabled_at": "2025-01-15T10:30:00Z"
    }
  }
}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
outpost-docs	Ready	Preview, Comment	Feb 12, 2026 2:49pm
outpost-website	Ready	Preview, Comment	Feb 12, 2026 2:49pm

[alexbouchardd]

I think we should rename alert.consecutive_failure to something like alert.destination.consecutive_failure or alert.destination.failure.

One thing to consider here is what we'll do once we have alerts based on failure rate and the associated event type.

1. Should we include attempt instead of event?

Yes, your proposition makes more sense

2

No strong opinion, but the current payload does lack progress, which is the threshold that was triggered

[alexluong]

One thing to consider here is what we'll do once we have alerts based on failure rate and the associated event type.

yes, let's do alert.destination.consecutive_failure in case we want to expand in the future?

but the current payload does lack progress

that's just the computed value of current / max tho, right? Or you want the threshold itself, so 50/70/90/100?

also do you know current can be higher than max? In that case, would progress be fixed at 100 (or 1) or would we continue adding it up?

[alexbouchardd]

I would represent the threshold itself which may not line up with the current / max exactly.

[alexluong]

@alexbouchardd updated, the PR description is up-to-date if you want to review the schema, etc.

[alexbouchardd]

Changes to the docs seems to be missing

[alexbouchardd]

Following the user conversation, I think we need to revisit the alert semantics altogether. Really, these are "Operator Events" and could span different topics, such as alerts. What I would propose is the following changes:

• Introduce a config/mechanism to subscribe to operator event topics such as attempt.failed and tenant.updated
• Introduce the ability to publish these events to one of the supported operator MQ like SQS/PubSub
• Rework the alert logic / destination logic to more clearly set the trigger conditions for the relevant operator events

The list of initial events would be:

• tenant.updated
• destination.disabled
• attempt.failed
• attempt.succeesed
• attempt.exhausted-retries
• alert.destination.consecutive-failures
• alert.destination.exhausted-retries
• alert.destination.failure-rate

The new alert configs would look something like

• ALERT_TRIGGER_CONSECUTIVE_FAILURES
• ALERT_TRIGGER_EXAUSTED_RETRIES_COUNT
• ALERT_TRIGGER_FAILURE_RATE

What do you think?

[alexluong]

I think the overall idea makes sense. There are some details that I may want to bring up for clarity, but nothing critical. The only high level topic I want to discuss is around publishing these Operator Events

Introduce the ability to publish these events to one of the supported operator MQ like SQS/PubSub

I assume your intention of relying on an MQ like SQS/PubSub is for reliability instead of HTTP? With that said, it's not fully guaranteed because there could be issue from Outpost side? So basically we cannot guarantee at-least-once but instead at-most-once for these events. Would that be the right take?

I'm thinking, if we want to provide more guarantee for these events, we can consider reusing the deliverymq mechanism of some sort? Treat it like a system-level destination somehow. I have an approach in mind that's not super huge in scope, just want to make sure we're on the same page first.

[alexbouchardd]

Good consideration, while I considered re-using the system level destination, my thinking is that it would leave a bunch of edge cases around having to hardcoded some exceptions for this "internal" destination, while we already have a pattern for operators to bring their own queue.

In terms of guarantee, we may be able can get there. Those events can be emitted by the log service when pulling attempts, and the message would get nacked in the log queue if the event can't be published. That would imply re-inserting into storage which should already be idempotent.