Skip to content

feat: add LanceDB version snapshot auto-cleanup#793

Open
njuboy11 wants to merge 1 commit into
CortexReach:masterfrom
njuboy11:feat/lancedb-storage-maintenance
Open

feat: add LanceDB version snapshot auto-cleanup#793
njuboy11 wants to merge 1 commit into
CortexReach:masterfrom
njuboy11:feat/lancedb-storage-maintenance

Conversation

@njuboy11
Copy link
Copy Markdown

Summary

LanceDB stores a new version snapshot on every write operation but never cleans them up. After a few days the database directory grows to 300+ MB despite only ~5-30 MB of actual data.

This PR adds a storageMaintenance.autoCleanup config block that periodically calls Table.optimize() to remove old version snapshots.

Config

{
  "storageMaintenance": {
    "autoCleanup": {
      "enabled": true,
      "intervalHours": 24,
      "retentionDays": 1
    }
  }
}

Changes

  • Add storageMaintenance.autoCleanup to plugin config schema (enabled, intervalHours, retentionDays)
  • Add MemoryStore.runMaintenance(retentionDays) → calls table.optimize({ cleanupOlderThan })
  • Add non-blocking setInterval timer in registerService.start (fires 5 min after startup, then every intervalHours)
  • Latest version is NEVER deleted by LanceDB

Closes #792

- Add storageMaintenance.autoCleanup config:
  - enabled: bool (default false)
  - intervalHours: int (default 24)
  - retentionDays: int (default 1)
- Add MemoryStore.runMaintenance() to call table.optimize()
- Add non-blocking setInterval timer for periodic cleanup
- Add JSON schema for new config in openclaw.plugin.json
- Closes CortexReach#792
@TurboTheTurtle
Copy link
Copy Markdown
Contributor

Review findings:

P1: maintenanceTimer is scoped inside start() but used in stop()

maintenanceTimer is declared inside the start callback, but stop is a sibling callback and references that variable later (index.ts:4176, index.ts:4210). That variable is out of scope for stop, so this either fails type checking or throws when the service stops. Please hoist maintenanceTimer to the same scope as backupTimer so start and stop share the same timer handle.

P1: accidental /tmp/rerank.log write leaks recall queries

The auto-recall path now does a synchronous append to /tmp/rerank.log with the recall query text (index.ts:2596). That leaks user memory queries to a fixed temp file and adds sync filesystem I/O on a hot recall path. This looks like debug instrumentation and should be removed before merge.

P1: lock ENOENT falls back to lock-free writes

runWithFileLock() now catches ENOENT from proper-lockfile.lock() and proceeds with the write without acquiring any cross-process lock (src/store.ts:242-272). That breaks the serialization guarantee around LanceDB writes, and it is unrelated to the storage-cleanup feature. If lock acquisition fails, the write should fail or retry under the lock rather than entering the critical section lock-free.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add LanceDB version snapshot auto-cleanup

2 participants