feat: enhance deduplicateItems with cross-batch/execution tracking

[restareaByWeezy]

🎯 Changes

This PR enhances the deduplicateItems option to support cross-batch/execution duplicate detection, simplifying the API by consolidating all deduplication functionality under a single option.

What Changed

Batcher & Queuer Enhancements:

• ✅ deduplicateItems now automatically tracks processedKeys across batches/executions
• ✅ Added maxTrackedKeys option (default: 1000) with FIFO eviction
• ✅ Enhanced onDuplicate callback with existingItem parameter to distinguish duplicate types:
  • existingItem === undefined: cross-batch/execution duplicate (already processed)
  • existingItem !== undefined: in-batch/queue duplicate
• ✅ Added clearProcessedKeys() method for manual key management

Usage Example

const batcher = new Batcher(processFn, {
  deduplicateItems: true,  // Single option enables both in-batch and cross-batch deduplication
  maxTrackedKeys: 100,
  getItemKey: (item) => item.id,
  onDuplicate: (newItem, existingItem, batcher) => {
    if (existingItem === undefined) {
      console.log('Already processed in previous batch')
    } else {
      console.log('Duplicate in current batch')
    }
  }
})

batcher.addItem({ id: 1, data: 'first' })
batcher.addItem({ id: 1, data: 'second' })  // in-batch duplicate
await batcher.flush()
batcher.addItem({ id: 1, data: 'third' })   // cross-batch duplicate (skipped)

// Clear processed keys when needed (e.g., for persistence restoration)
batcher.clearProcessedKeys()

Design Rationale

This follows the pattern of RateLimiter's executionTimes tracking:

• Single option to enable the feature
• Automatic state management
• Persistence support through state serialization
• Clear and intuitive API

Test Coverage

• Batcher: 51 new test cases (109 total passing ✅)
• Queuer: 58 new test cases (553 total passing ✅)
• Comprehensive coverage of:
  • Cross-batch/execution duplicate detection
  • In-batch/queue deduplication with keep-first/keep-last strategies
  • maxTrackedKeys FIFO eviction behavior
  • clearProcessedKeys() method
  • Persistence scenarios
  • onDuplicate callback invocations

Documentation & Examples

• Added examples/react/useBatcherDedup demonstrating:
  • Cross-batch duplicate tracking visualization
  • processedKeys state display
  • maxTrackedKeys and clearProcessedKeys() usage
  • Visual indicators for duplicate types
• Updated changeset with detailed migration guide

---

✅ Checklist

• I have followed the steps in the Contributing guide.
• I have tested this code locally with pnpm run test:pr.

🚀 Release Impact

• This change affects published code, and I have generated a changeset.
• This change is docs/CI/dev-only (no release).

Changeset: .changeset/add-deduplication-feature.md

[changeset-bot]

🦋 Changeset detected

Latest commit: 36189fe

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@tanstack/pacer	Minor
@tanstack/preact-pacer	Patch
@tanstack/react-pacer	Patch
@tanstack/solid-pacer	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR