Skip to content

Deduplicate ingestion cursor and state management #7

@BunsDev

Description

@BunsDev

Problem

Cursor tracking logic for ingestion is repeated across import-openclaw.ts, import-cron.ts, and ingestion-state.ts. Each ingestion source implements its own cursor read/write pattern with no shared abstraction. There is also no deduplication logic for events that may be ingested more than once.

Proposed Solution

  1. Shared IngestionCursor class — Encapsulates cursor read, advance, and persist operations
  2. Event deduplication — Check for existing event IDs before insert (idempotent ingestion)
  3. Consistent error handling — Shared retry/skip logic for malformed records
  4. Ingestion registry — Central registry of ingestion sources with their cursor states

Acceptance Criteria

  • A shared IngestionCursor utility handles all cursor state operations
  • import-openclaw.ts and import-cron.ts use the shared utility
  • Duplicate events are detected and skipped (idempotent)
  • Unit tests cover cursor advancement and deduplication
  • No behavioral changes to existing ingestion flows

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    architectureStructural and design improvementsenhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions