Implement global queue for coordinated ingestion across sources

[anirudhaacharyap]

Problem

Currently, ingestion requests can originate from multiple sources (MCP, SDK, browser extension, etc.). When these requests are processed concurrently for the same user, it can lead to race conditions during memory updates.

Even with batch ingestion, overlapping requests across interfaces may:

• Cause inconsistent memory state
• Overwrite updates
• Break ordering guarantees

Goal

Ensure that ingestion requests for a single user are processed in a controlled, ordered manner across all sources.

Proposed Approach

• Introduce a per-user coordination mechanism:

  • Option 1: In-memory queue keyed by user_id
  • Option 2: Distributed queue (future) using Redis or similar

• Guarantee:

  • Only one ingestion pipeline (or critical section) runs per user at a time
  • Requests are processed in FIFO order

Scope

• Define abstraction for per-user coordination
• Integrate with existing ingestion endpoint(s)
• Ensure compatibility with batch ingestion

Out of Scope (for this issue)

• Full distributed queue implementation (can be separate enhancement)
• Infrastructure-level decisions (Redis, Celery, etc.)

Acceptance Criteria

• No overlapping ingestion updates for the same user
• Requests maintain order consistency
• Works across multiple API entry points