[CRASH]: "main thread terminated" due to fastlock lock/unlock mismatch (KeyDB 6.3.4) #924
Description
Environment
KeyDB Version: 6.3.4, bits=64, commit=00000000, modified=0
Mode: Standalone
Platform: Linux 5.10.240-238.966.amzn2.x86_64
Containerization: Running in Kubernetes, Bitnami Helm chart
ConfigFile: /opt/bitnami/keydb/etc/keydb.conf
Deployment Context: High connection count, background saves, frequent replication
Problem Summary
KeyDB crashed with the log line main thread terminated. The crash report includes a "Guru Meditation" error, pointing to a lock/unlock mismatch in fastlock.cpp:295. The stack trace identifies a concurrency or internal locking bug, but fast memory test passed, suggesting the RAM itself is not the immediate cause. The error occurred during high load/background saving intervals.
Relevant Log Extracts
115S 29 Oct 2025 184500.980 ------------------------------------------------
115S 29 Oct 2025 184500.980 !!! Software Failure. Press left mouse button to continue
115S 29 Oct 2025 184500.980 Guru Meditation: fastlock lock/unlock mismatch for v [fastlock.cpp:295]
------ STACK TRACE ------
Backtrace:
keydb-server 0.0.0.0:6379 handleBlockedClientsTimeout+0x23b (0x55a505bc3dab)
keydb-server 0.0.0.0:6379 beforeSleep+aeEventLoop+0x190 (0x55a505ab5270)
keydb-server 0.0.0.0:6379 aeProcessEvents+0x3a8 (0x55a505aac068)
keydb-server 0.0.0.0:6379 aeMain+0x3e (0x55a505aac6be)
keydb-server 0.0.0.0:6379 workerThreadMain(void)+0x105 (0x55a505ac31b5)
libx86_64-linux-gnu/libc.so.6+0x891c4 (0x7f76e7f5e1c4)
libx86_64-linux-gnu/libc.so.6+clone+0x40 (0x7f76e7fddac0)
------ INFO OUTPUT ------
main thread terminated
Bio thread for job type 0 terminated
Bio thread for job type 1 terminated
Bio thread for job type 2 terminated
Fast memory test PASSED, however your memory can still be broken.
Please run a memory test for several hours if possible.
KEYDB BUG REPORT END. Make sure to include from START to END.
Please report the crash by opening an issue on github: https://github.com/JohnSully/KeyDB/issues
Suspect RAM error? Use keydb-server --test-memory to verify it
Timeline of Events (at crash)
- Multiple background saves every few seconds: RDB files ~6-11 MB.
- Notice: Detuning locks due to high load per core (values ~90–103).
- Role: active-replica, connected to master. Replication events, sync and partial resync.
- 560+ connected clients; 1 master, 1 replica.
- Server uptime: ~1.1 days; Redis buildID 37939c54a10461d4.
Replication and Persistence Details
- Partial/failure resync events logged.
- Replication backlog resizing events.
- Disk persistence and background saves completed successfully until crash.
Additional Diagnostics and Actions
- Fast memory test PASSED.
- No custom modules loaded at time of crash.
- No other fatal errors in logs.
- High lock detuning notices; possible relationship to concurrency or contention.
Steps to Reproduce
- Not reproducible on-demand but typically happens under heavy background saves + high concurrency:
- Multiple clients connected.
- Frequent background saves (RDB).
- High load per core notifications present right before crash.
- Heavy replication/persistence activity.