Skip to content

Deadlock condition in graphclient on subscribe/unsubscribe#94

Merged
woswos merged 7 commits into
masterfrom
mw-graphclient-race
May 12, 2026
Merged

Deadlock condition in graphclient on subscribe/unsubscribe#94
woswos merged 7 commits into
masterfrom
mw-graphclient-race

Conversation

@monaresh
Copy link
Copy Markdown

@monaresh monaresh commented May 8, 2026

Found a deadlock condition when trying to unsubscribe using the client

  1. Main thread enters UnsubscribeGraphAPI -> acquires self._subscriptionLock
  2. Main thread calls self._backgroundThread.RunCoroutine(_Unsubscribe()).result() - still holding _subscriptionLock.

.result() blocks the main thread until the coroutine returns -> we are waiting for this under lock
3. On the background thread's event loop, _Unsubscribe runs and reaches await self._CloseWebSocket() -> await self._webSocket.close(). That await yields control back to the event loop.
4. now yielded, the sleeping _ListenToWebSocket function wakes up because the websocket just

  1. _ListenToWebSocket exception handler executes with self._subscriptionLock: - synchronous threading.Lock. The main thread holds it, so the background thread blocks here and freezes. The entire event loop is now blocked - no further awaits can move now

  2. _Unsubscribe await self._webSocket.close() never gets to return, so .result() on the main thread never completes. Main thread keeps holding _subscriptionLock. Stuck.

Error on sigterm

2026-05-08 14:14:48,914 mujinwebstackclient.controllerwebclientraw [ERROR] [controllerwebclientraw.py:598 _ListenToWebSocket] caught WebSocket exception: sent 1000 (OK); no close frame received
Traceback (most recent call last):
 File "/opt/lib/python3.13/site-packages/mujinwebstackclient/controllerwebclientraw.py", line 539, in _ListenToWebSocket
   async for response in self._webSocket:
   ...<55 lines>...
           subscription.GetSubscriptionCallbackFunction()(error=None, response=content.get('payload') or {})
 File "/opt/lib/python3.13/site-packages/websockets/asyncio/connection.py", line 242, in __aiter__
   yield await self.recv()
         ^^^^^^^^^^^^^^^^^
 File "/opt/lib/python3.13/site-packages/websockets/asyncio/connection.py", line 322, in recv
   raise self.protocol.close_exc from self.recv_exc
websockets.exceptions.ConnectionClosedError: sent 1000 (OK); no close frame received
ConnectionClosedError(None, Close(code=1000, reason=''), None)
                        

@monaresh monaresh self-assigned this May 8, 2026
@monaresh monaresh requested review from gcccai, rschlaikjer and woswos May 8, 2026 07:11
Comment thread python/mujinwebstackclient/controllerwebclientraw.py Outdated
Comment thread python/mujinwebstackclient/controllerwebclientraw.py
@monaresh monaresh requested a review from woswos May 8, 2026 13:46
Comment thread python/mujinwebstackclient/controllerwebclientraw.py
Comment thread python/mujinwebstackclient/controllerwebclientraw.py Outdated
Comment thread python/mujinwebstackclient/controllerwebclientraw.py Outdated
Comment thread python/mujinwebstackclient/controllerwebclientraw.py
@woswos
Copy link
Copy Markdown
Member

woswos commented May 11, 2026

Pipeline #1903871

@monaresh monaresh requested a review from woswos May 11, 2026 07:06
@woswos woswos merged commit 83cd0ad into master May 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants