IX-1782: Que health check surfaces unhealthy state if postgres_error on any worker#118
Merged
jcardozagc merged 1 commit intomasterfrom Mar 30, 2026
Conversation
jcardozagc
force-pushed
the
ix-1782-healthcheck-checks-workers-for-pg-error
branch
3 times, most recently
from
7f5a866 to
fcd7731
Compare
| end | ||
|
|
||
| def call(_env) | ||
| if @worker_group.workers.all?(&:healthy?) |
Contributor
There was a problem hiding this comment.
I'm not sure about if we want to say that the app is unhealthy if just one worker has a problem, that could be transient any might mean we end up restarting the pod for a single error which probably isn't what we need.
Maybe we do something like
if @worker_group.workers.any?(&:healthy?)
So we end up only triggering the else when ALL the workers are unhealthy?
During a recent incident, after a CloudSQL maintenance event, Que worker pods can end up in a stale database connection state. Workers handle this internally - on a PG::Error, they return :postgres_error, sleep for the wake interval, and retry. The health check is a hardcoded lambda that always returns 200. Hence, Kubernetes also has no signal to restart the affected pods, and the worker in the incident ended up perpetually retrying a faulty connection every 5 seconds. Let's track the result of each work cycle on the workers itself and expose it through a healthy? predicate (which purely checks whether the worker is in a postgres_error state). The health check endpoint now delegates to a new WorkerHealthCheck that checks every worker in the group and returns 503 if any of them are in a unhealthy state, otherwise returns the 200 it has always been returning. Yes, this endpoint now actually can return a non-200 and this is the only 'breaking' change, but should mean Kubernetes has a signal to restart pods when liveness probe threshold gets breached.
jcardozagc
force-pushed
the
ix-1782-healthcheck-checks-workers-for-pg-error
branch
from
fcd7731 to
66be132
Compare
Contributor
Author
|
Leaving this running on payments-service live-staging for a wee bit |
stephenbinns
approved these changes
Mar 27, 2026
Contributor
stephenbinns
left a comment
stephenbinns
left a comment
There was a problem hiding this comment.
Given its been running a while now and the idea is sound I think we can get this out next week
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During a recent incident, after a CloudSQL maintenance event, Que worker pods can end up in a stale database connection state. Workers handle this internally - on a
PG::Error, they return:postgres_error, sleep for the wake interval, and retry. The health check is a hardcoded lambda that always returns 200. Hence, Kubernetes also has no signal to restart the affected pods, and the worker in the incident ended up perpetually retrying a faulty connection every 5 seconds.Let's track the result of each work cycle on the workers itself and expose it through a
healthy?predicate (which purely checks whether the worker is in apostgres_errorstate on its lastwork_loopcycle). The health check endpoint now delegates to a newWorkerHealthCheckthat checks every worker in the group and returns 503 if any of them are in a unhealthy state, otherwise returns the 200 it has always been returning.Yes, this health check now can return a non-200 and this is the only 'breaking' change, but should mean Kubernetes has a signal to restart pods when liveness probe threshold gets breached.