We are using "unpostpone" command programmatically to control when gh-ost will cut over a migration based on a button click in the UI.
When we issue "unpostpone" and if there is a high heartbeat lag on the database, gh-ost will wait for the lag to subside before even checking if "unpostpone" was sent.
When our database experiences periods of high contention, this might delay the cut-over indefinitely.
We'd like to handle these cases on our side gracefully but are running into some limitations around how "unpostpone" command is being handled today.
Observability
Current status commands offer no way to distinguish between the following two scenarios:
- cut-over has been attempted and failed. "UserCommandedUnpostponeFlag" flag has been reset. No further attempts will be made
- cut-over is waiting for the heartbeat lag to subside and will proceed when the lag falls under the threshold
We'd love to add an indicator to the status that would tell us whether or not UserCommandedUnpostponeFlag is currently set.
Cancellation
Once we issue "unpospone", if the heartbeat lag is taking too long to subside we'd like to give up the attempt. The current interface doesn't allow us to do this, since nothing other than a cut-over attempt resets UserCommandedUnpostponeFlag.
We would like to either be able to cancel an "unpostpone" command in an interactive manner, or, when sending it initially, set a timeout after which the attempt would auto-reset
What do you think?
If you let me know which interface approach you prefer I'm happy to send a PR to implement it.
We are using "unpostpone" command programmatically to control when gh-ost will cut over a migration based on a button click in the UI.
When we issue "unpostpone" and if there is a high heartbeat lag on the database, gh-ost will wait for the lag to subside before even checking if "unpostpone" was sent.
When our database experiences periods of high contention, this might delay the cut-over indefinitely.
We'd like to handle these cases on our side gracefully but are running into some limitations around how "unpostpone" command is being handled today.
Observability
Current status commands offer no way to distinguish between the following two scenarios:
We'd love to add an indicator to the status that would tell us whether or not UserCommandedUnpostponeFlag is currently set.
Cancellation
Once we issue "unpospone", if the heartbeat lag is taking too long to subside we'd like to give up the attempt. The current interface doesn't allow us to do this, since nothing other than a cut-over attempt resets UserCommandedUnpostponeFlag.
We would like to either be able to cancel an "unpostpone" command in an interactive manner, or, when sending it initially, set a timeout after which the attempt would auto-reset
What do you think? If you let me know which interface approach you prefer I'm happy to send a PR to implement it.