Add support for :skip_retries config option to Oban.ErrorReporter

[solnic]

This is inspired by this thread on Bsky started by @katafrakt:

UPDATE: and also it's based on my experience using Sentry+Oban extensively in the past, where I had to do almost exactly what this PR does :)

https://bsky.app/profile/katafrakt.bsky.social/post/3lcuccw6zik2x

Long story short - there are cases where reporting every job failure as an error is not desired. When you expect that some jobs just fail sometimes due to some external factors, you want to be notified only if all attempts failed.

To solve this, we can introduce a config option to instruct Oban.ErrorReporter what to do when a job failed but it was not the final attempt. This PR proposes such an addition.

[whatyouhide]

@solnic what's the status on this? Something we want to pursue?

[solnic]

@whatyouhide on hold until I have time for this, which is after distributed tracing and structured logging, as these are my priorities now.

[brendon9x]

Hi all – this would be a very useful feature. A common case is AI workloads: I am finding Anthropic fails about 1/250 times so I'm getting Sentries every hour or so. But these are just transient so all Oban jobs are succeeding on the 2nd try.

@solnic – this looks quite close. I would be happy to pick this up if the effort of handing over your vision for this PR isn't going to be more complicated than just doing it yourself.

[brendon9x]

I think I am following in your footsteps Peter. I worked around this issue and then ran into tracing issues (grateful for the work there too)

[Miradorn]

Hey!
I've been sent here via #963 (comment) and have some early feedback:
For our use-case of the library a "global" config won't be enough to handle all cases.
We track hundreds of Workers with sentry and have our own wrapper already with many special handling of cases. From the experience we have with that, what we'd need would be a per-job "callback" (similar to the cron-checkin-config one where we could dynamically decide per invocation if we want to "ignore" a retryable job.

Of course we might be a special case and the global option would be sufficient for the majority of users, than we'd just continue using our wrapper, just wanted to give this perspective here for considerations.

Thanks for oyur work!

[whatyouhide]

I’d be ok with the callback but the bummer is that it wouldn't go in Source Sentry.Integrations.Oban.Cron but rather in yet another behaviour, which is kind of annoying.

However, I might be missing something but could you implement this yourselves, basically? If you do have an Oban.Worker wrapper, imagine something like this:

# This uses the behaviour
use MyApp.ObanWorkerWrapper

@impl MyApp.ObanWorkerWrapper
def report_to_sentry?(...) do
  ...
end

Now in your before_send implementation:

# The pattern matching is made up, I can't remember details off the top of my head
def before_send(%{job: job} = error) do
  {:ok, worker} = Oban.Worker.from_string(job.worker)
  worker.report_to_sentry?(error)
end

Would an approach like this work?

[Miradorn]

That's basically (implementation differs) what we're doing.
Like I said, I'm not sure if this should go in the base library or not, just wanted it to be out there for consideration!

[whatyouhide]

I see. No, I don't think we should add another mechanism for potentially skipping extensions. That would increase the API surface and not be very easy to extend. Thanks for raising the concern though!