feat: enhance deduplicateItems with cross-batch/execution tracking

.changeset/add-deduplication-feature.md

@@ -0,0 +1,55 @@
+---
+"@tanstack/pacer": minor
+---
+
+Add cross-batch/cross-execution deduplication support to Batcher and Queuer
+
+This feature extends the existing `deduplicateItems` option to track processed items across batch/execution cycles. When enabled, items that have already been processed will be automatically skipped.
+
+### Enhanced Options
+
+- `deduplicateItems: boolean` - Now prevents duplicates **both within and across batches** (default: false)
+- `deduplicateStrategy: 'keep-first' | 'keep-last'` - Only affects in-batch duplicates (default: 'keep-first')
+- `getItemKey: (item) => string | number` - Extract unique key from item
+- `maxTrackedKeys: number` - Maximum keys to track with FIFO eviction (default: 1000)
+- `onDuplicate: (newItem, existingItem?, instance) => void` - Called for both in-batch and cross-batch duplicates
+
+### New Methods
+
+- `hasProcessedKey(key)` - Check if a key has been processed
+- `peekProcessedKeys()` - Get a copy of all processed keys
+- `clearProcessedKeys()` - Clear the processed keys history
+
+### New State Properties
+
+- `processedKeys: Array<string | number>` - Keys that have been processed (similar to RateLimiter's executionTimes)
+
+### Behavior
+
+When `deduplicateItems` is enabled:
+1. **In-batch duplicates**: Merged based on `deduplicateStrategy` ('keep-first' or 'keep-last')
+2. **Cross-batch duplicates**: Skipped entirely (already processed)
+3. `onDuplicate` called with `existingItem` for in-batch, `undefined` for cross-batch
+
+### Use Case
+
+Prevents redundant processing when the same data is requested multiple times:
+- API calls: Don't fetch user-123 if it was already fetched
+- No-code tools: Multiple components requesting the same resource
+- Event processing: Skip events that have already been handled
+
+Similar to request deduplication in TanStack Query, but at the batching/queuing level.
+
+### Persistence Support
+
+The `processedKeys` can be persisted via `initialState`, following the existing Pacer pattern (similar to RateLimiter):
+
+```typescript
+const savedState = localStorage.getItem('batcher-state')
+const batcher = new Batcher(fn, {
+  deduplicateItems: true,
+  initialState: savedState ? JSON.parse(savedState) : {},
+})
+```
+
+Fully opt-in with no breaking changes to existing behavior.

examples/react/useBatcherDedup/.eslintrc.cjs

@@ -0,0 +1,13 @@
+// @ts-check
+
+/** @type {import('eslint').Linter.Config} */
+const config = {
+  settings: {
+    extends: ['plugin:react/recommended', 'plugin:react-hooks/recommended'],
+    rules: {
+      'react/no-children-prop': 'off',
+    },
+  },
+}
+
+module.exports = config

examples/react/useBatcherDedup/.gitignore

@@ -0,0 +1,27 @@
+# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
+
+# dependencies
+/node_modules
+/.pnp
+.pnp.js
+
+# testing
+/coverage
+
+# production
+/build
+
+pnpm-lock.yaml
+yarn.lock
+package-lock.json
+
+# misc
+.DS_Store
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*

examples/react/useBatcherDedup/README.md

@@ -0,0 +1,80 @@
+# TanStack Pacer - Cross-Batch Deduplication Example
+
+This example demonstrates the `trackProcessedKeys` feature in TanStack Pacer, which prevents duplicate processing **across batches**.
+
+## What This Example Shows
+
+- **Cross-Batch Deduplication**: Items that have been processed won't be processed again until cleared
+- **Processed Keys Tracking**: Similar to `RateLimiter.executionTimes`, tracks which keys have been processed
+- **Custom Key Extraction**: Use `getItemKey` to define what makes an item unique
+- **Memory Management**: `maxTrackedKeys` limits memory usage with FIFO eviction
+- **Skip Callback**: Track when items are skipped with `onSkip`
+- **State Persistence**: Use `initialState` to restore processed keys from storage
+
+## Use Case
+
+In applications where the same data might be requested multiple times (e.g., no-code tools, component-based UIs), you want to:
+
+- **Prevent redundant API calls** - If user-123's data was fetched, don't fetch it again
+- **Reduce server load** - Avoid duplicate processing even across different batch cycles
+- **Improve performance** - Skip items that have already been handled
+
+This is similar to request deduplication in TanStack Query, but at the batching level.
+
+## Running the Example
+
+```bash
+pnpm install
+pnpm dev
+```
+
+Then open http://localhost:3007
+
+## Key Configuration
+
+```typescript
+const batcher = useBatcher(
+  (userIds: string[]) => {
+    // API call to fetch users
+    console.log("Fetching users:", userIds);
+  },
+  {
+    maxSize: 5,
+    wait: 2000,
+    trackProcessedKeys: true,       // Enable cross-batch deduplication
+    getItemKey: (userId) => userId, // Define uniqueness
+    maxTrackedKeys: 100,            // Limit memory (FIFO eviction)
+    onSkip: (item) => {
+      console.log(`Skipped (already processed): ${item}`);
+    },
+  }
+);
+```
+
+## Try It Out
+
+1. Click "Fetch user-123" - It will be added to the batch
+2. Wait for the batch to process (or click "Flush Batch Now")
+3. Click "Fetch user-123" again - It will be **skipped** because it was already processed!
+4. Watch the "Items Skipped" counter increase
+5. Click "Clear Processed Keys" to allow re-processing
+
+## API Reference
+
+### Options
+
+- `trackProcessedKeys: boolean` - Enable cross-batch deduplication (default: false)
+- `getItemKey: (item) => string | number` - Extract unique key from item
+- `maxTrackedKeys: number` - Maximum keys to track (default: 1000, FIFO eviction)
+- `onSkip: (item, batcher) => void` - Callback when an item is skipped
+
+### Methods
+
+- `hasProcessedKey(key)` - Check if a key has been processed
+- `peekProcessedKeys()` - Get a copy of all processed keys
+- `clearProcessedKeys()` - Clear the processed keys history
+
+### State
+
+- `processedKeys: Array<string | number>` - Keys that have been processed
+- `skippedCount: number` - Number of items skipped due to deduplication

examples/react/useBatcherDedup/index.html

@@ -0,0 +1,16 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8" />
+    <link rel="icon" type="image/svg+xml" href="/emblem-light.svg" />
+    <meta name="viewport" content="width=device-width, initial-scale=1" />
+    <meta name="theme-color" content="#000000" />
+    <script src="https://unpkg.com/react-scan/dist/auto.global.js"></script>
+    <title>TanStack Pacer Example</title>
+  </head>
+  <body>
+    <noscript>You need to enable JavaScript to run this app.</noscript>
+    <div id="root"></div>
+    <script type="module" src="/src/index.tsx"></script>
+  </body>
+</html>

examples/react/useBatcherDedup/package.json

@@ -0,0 +1,34 @@
+{
+  "name": "@tanstack/pacer-example-react-use-batcher-dedup",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "vite --port=3007",
+    "build": "vite build",
+    "preview": "vite preview",
+    "test:types": "tsc"
+  },
+  "dependencies": {
+    "@tanstack/react-pacer": "^0.19.3",
+    "react": "^19.2.3",
+    "react-dom": "^19.2.3"
+  },
+  "devDependencies": {
+    "@types/react": "^19.2.8",
+    "@types/react-dom": "^19.2.3",
+    "@vitejs/plugin-react": "^5.1.2",
+    "vite": "^7.3.1"
+  },
+  "browserslist": {
+    "production": [
+      ">0.2%",
+      "not dead",
+      "not op_mini all"
+    ],
+    "development": [
+      "last 1 chrome version",
+      "last 1 firefox version",
+      "last 1 safari version"
+    ]
+  }
+}