Skip to content

Recover from panicking notifier subscribers#1270

Closed
peter941221 wants to merge 2 commits into
riverqueue:masterfrom
peter941221:peter/notifier-panic-recovery
Closed

Recover from panicking notifier subscribers#1270
peter941221 wants to merge 2 commits into
riverqueue:masterfrom
peter941221:peter/notifier-panic-recovery

Conversation

@peter941221
Copy link
Copy Markdown
Contributor

  1. Failure mechanism

Notifier delivers subscriber callbacks from a shared delivery goroutine in deliverNotifications. Right now, one panicking callback can take down that goroutine and stop delivery for unrelated subscribers.

  1. Semantic change

Wrap each subscriber callback invocation in local panic recovery.

If one subscriber panics, River now logs the panic value and notification topic, then continues delivering to later subscribers and stays alive for future notifications.

  1. Why this design

This keeps the fix at the narrowest boundary that owns the risk: the callback fan-out loop.

It does not change the subscription API or the notifier control flow. It only prevents one bad subscriber from killing delivery for everyone else.

  1. Testing

Added PanicInSubscriberDoesNotBreakDelivery in internal/notifier/notifier_test.go.

The test drives notificationBuf directly so it stays unit-scoped. It proves that:

  • a panicking subscriber still sees the notification up to the panic point,
  • another subscriber still receives the same notification,
  • later notifications are still delivered after the panic.

@peter941221 peter941221 marked this pull request as ready for review June 3, 2026 11:54
Copy link
Copy Markdown
Contributor

@brandur brandur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just since all notification callbacks are internal code right now, I think we might want to leave this as is — a panic would be a clear bug, and it'd be nice to know about it ASAP instead of it getting swallowed up in a test log somewhere.

@peter941221
Copy link
Copy Markdown
Contributor Author

@brandur sure, I'll close this one.

@peter941221 peter941221 closed this Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants