fix(controller): resolve mutex deadlock between agent and remotemcpserver reconcilers

[joshadambell]

Summary

Fixes controller reconciliation deadlock by removing the application-level upsertLock mutex. Database-level concurrency control (atomic upserts and transactions) now handles synchronization.

Problem

The shared upsertLock mutex blocked all Agent reconciliations when RemoteMCPServer reconciliation performed slow network I/O while holding the lock.

Solution

Phase 1: Reduced lock scope to exclude network I/O
Phase 2: Fixed Agent predicate to ensure Create events are always processed
Phase 3: Removed mutex entirely - database handles concurrency via:

• Atomic upserts: INSERT ... ON CONFLICT DO UPDATE
• Transactions: Atomic tool replacement in RefreshToolsForServer()

Phase 4: Documentation and tests

• Added architecture doc explaining concurrency model
• Added idempotence tests for StoreAgent and StoreToolServer
• Added IMPORTANT comment to RefreshToolsForServer about transaction scope

[EItanya]

Great catch on the predicate issues with agents! That's so weird, I'll need to dig into the impl of those predicates.

In terms of the lock, I hate to say it but I think we can actually get rid of those locks altogether. Those are a relic of an older impl which didn't have a DB. The DB should function as the atomic record for us.

[joshadambell]

Great catch on the predicate issues with agents! That's so weird, I'll need to dig into the impl of those predicates.

In terms of the lock, I hate to say it but I think we can actually get rid of those locks altogether. Those are a relic of an older impl which didn't have a DB. The DB should function as the atomic record for us.

@EItanya you are right. I completely removed the app level locks.