Skip to content

Add background pull-sync dispatch for organic routes#559

Draft
ChristianPavilonis wants to merge 1 commit intofeature/ec-s2s-batch-syncfrom
feature/ec-pull-sync-dispatch
Draft

Add background pull-sync dispatch for organic routes#559
ChristianPavilonis wants to merge 1 commit intofeature/ec-s2s-batch-syncfrom
feature/ec-pull-sync-dispatch

Conversation

@ChristianPavilonis
Copy link
Collaborator

Summary

Implements Story 9 (#542): server-to-server pull sync that runs after send_to_client() on organic traffic only, ensuring zero client-facing latency impact.

  • Refactors Fastly adapter entrypoint from #[fastly::main] to explicit Request::from_client() + send_to_client() to enable post-send background work
  • Adds pull_sync module: enumerates pull-enabled partners, checks staleness via pull_sync_ttl_sec, validates URL hosts against partner allowlist, enforces hourly rate limits, and dispatches concurrent outbound GETs with Bearer auth
  • Validates EC ID format before dispatch to prevent amplification from spoofed values
  • Adds PartnerStore::list_registered() for KV store enumeration
  • Adds ec.pull_sync_concurrency setting (default 3)

Acceptance Criteria Coverage

Criteria Status
Dispatch only on organic routes handle_publisher_request, handle_proxy — never /sync, /identify, /auction, /admin/*
Fire when EC present, consent granted, pull_sync_enabled, partner missing/stale
Rate limit key pull:{partner_id}:{ec_hash}, 1-hour window
Concurrency cap settings.ec.pull_sync_concurrency default 3
Validate pull_sync_url host in pull_sync_allowed_domains
Outbound GET with ec_hash + ip query params + Bearer token
uid: null and 404 → no-op (debug only)
Non-200 → warn, no retry
Run after send_to_client()

Known Follow-ups

  • Performance: list_registered() scans all partner keys per eligible request. A pull-sync partner index would reduce this to O(1) for deployments with many partners.

Test Results

  • cargo fmt
  • cargo clippy
  • cargo test --workspace — 812 tests pass
  • npx vitest run — 282 tests pass

Closes #542

Implement Story 9 (#542): server-to-server pull sync that runs after
send_to_client() on organic traffic only. Refactors the Fastly adapter
entrypoint from #[fastly::main] to explicit Request::from_client() +
send_to_client() to enable post-send background work.

Pull sync enumerates pull-enabled partners, checks staleness against
pull_sync_ttl_sec, validates URL hosts against the partner allowlist,
enforces hourly rate limits, and dispatches concurrent outbound GETs
with Bearer auth. Responses with uid:null or 404 are no-ops; valid
UIDs are upserted into the identity graph.

Includes EC ID format validation to prevent dispatch on spoofed values,
partner list_registered() for KV store enumeration, and configurable
pull_sync_concurrency (default 3).
@ChristianPavilonis ChristianPavilonis force-pushed the feature/ec-pull-sync-dispatch branch from 01e9180 to a4ae046 Compare March 25, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant