Survive minor redis connection drops

[adamruzicka]

If an exception was raised when trying to acquire to lock, the process would end up in a half-dead state where it would stay "up", but would not attempt to acquire the lock again when the connection came back up.

How to test this?

Setup

1. Run postgres podman run -p 5432:5432 -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=dynflow
2. Run redis podman run -p 6379:6379 redis:6
3. Run orchestrator 1

export DB_CONN_STRING=postgresql://postgres:postgres@localhost:5432/dynflow
bundle exec sidekiq -r ./examples/remote_executor.rb -q dynflow_orchestrator

4. Run orchestrator 2

export DB_CONN_STRING=postgresql://postgres:postgres@localhost:5432/dynflow
bundle exec sidekiq -r ./examples/remote_executor.rb -q dynflow_orchestrator

5. Run a worker

export DB_CONN_STRING=postgresql://postgres:postgres@localhost:5432/dynflow
bundle exec sidekiq -r ./examples/remote_executor.rb -q default

6. Start spawning things

export DB_CONN_STRING=postgresql://postgres:postgres@localhost:5432/dynflow
bundle exec ruby ./examples/remote_executor.rb client

The actual test

1. Stop the redis container
2. Observe how everything starts spewing errors
3. Stop orchestrator 1
4. Start the redis container again

Expected results:
Things may appear stuck until orchestrator 1's lock expires (up to 1 minute), after that orchestrator 2 should kick in.

Includes #462

[adamruzicka]

not ok 8 active orchestrator can survive a longer redis connection drop
...
# o1: [2025-11-26 15:29:35.743 #6666] FATAL -- dynflow: The orchestrator lock was stolen by 24ade3e2-60a6-41e8-acbf-076998697081, aborting.

waiiit, what? In a test where there's only one orchestrator running?

[adamruzicka]

Guess a SIGTERM wasn't enough

[ofedoren]

Thanks, @adamruzicka, LGTM, just one probably not related question, but since we're at redis/sidekiq world...

I've just noticed that dynflow uses gitlab-sidekiq-fetcher, which seems to be no longer maintained for others: https://gitlab.com/gitlab-org/ruby/gems/sidekiq-reliable-fetch#this-gem-is-no-longer-updated. Is it something that affects us aside from using gem that is not maintained anymore? Note that this gem locks used version of sidekiq to be ~> 6.1, thus it might need some refactoring if we'll end up using latest sidekiq/redis versions since we don't seem to have any locks on these.

[adamruzicka]

Is it something that affects us aside from using gem that is not maintained anymore?

It is holding us back from moving to newer sidekiq, but nothing else apart from that. I have some ideas how we could get out of this, but resolving this situation isn't an immediate priority right now.

[adamruzicka]

Thank you @ofedoren !