Skip to content

Feat/issue 72 replication plugin#105

Open
jeid12 wants to merge 2 commits intoouterbase:mainfrom
jeid12:feat/issue-72-replication-plugin
Open

Feat/issue 72 replication plugin#105
jeid12 wants to merge 2 commits intoouterbase:mainfrom
jeid12:feat/issue-72-replication-plugin

Conversation

@jeid12
Copy link

@jeid12 jeid12 commented Mar 19, 2026

Purpose

Implements a replication plugin that pulls data from the configured external source into internal Durable Object SQLite on a schedule, enabling close-to-edge read replicas.

Closes #72
/claim #72

Tasks

  • Add new ReplicationPlugin with task management endpoints
  • Add internal replication task table (tmp_replication_tasks) and cursor state
  • Implement cursor-based pull (cursor_column + cursor_value) from external source
  • Implement write path into internal source with INSERT OR REPLACE
  • Support per-task interval and batch size controls
  • Trigger replication execution via Durable Object alarm callback route
  • Add replication alarm continuation in Durable Object alarm()
  • Register plugin in worker startup plugin list
  • Add plugin unit tests
  • Add plugin README usage guidance

Verify

  • Run tests:
pnpm vitest run \
  plugins/replication/index.test.ts \
  src/do.test.ts \
  src/handler.test.ts
  • Manual flow:
# Create replication task
curl --location --request POST 'https://starbasedb.YOUR-ID-HERE.workers.dev/replication/tasks' \
  --header 'Authorization: Bearer ABC123' \
  --header 'Content-Type: application/json' \
  --data '{
    "sourceTable":"orders",
    "targetTable":"orders",
    "cursorColumn":"id",
    "intervalSeconds":60,
    "batchSize":500
  }'

# List tasks
curl --location 'https://starbasedb.YOUR-ID-HERE.workers.dev/replication/tasks' \
  --header 'Authorization: Bearer ABC123'

# Run once manually
curl --location --request POST 'https://starbasedb.YOUR-ID-HERE.workers.dev/replication/tasks/TASK_ID/run' \
  --header 'Authorization: Bearer ABC123'

# Delete task
curl --location --request DELETE 'https://starbasedb.YOUR-ID-HERE.workers.dev/replication/tasks/TASK_ID' \
  --header 'Authorization: Bearer ABC123'

Before

No built-in plugin mechanism to regularly pull selected external tables into internal SQLite for edge-local reads.

After

Replication tasks can be configured per table with interval + cursor strategy, and are executed on schedule through DO alarms.

Demo Video

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replicate data from external source to internal source with a Plugin

1 participant