Skip to content

Add proxy integration tests and missing UTS spec coverage#462

Open
paddybyers wants to merge 15 commits intouts-integrationfrom
uts-integration-proxy
Open

Add proxy integration tests and missing UTS spec coverage#462
paddybyers wants to merge 15 commits intouts-integrationfrom
uts-integration-proxy

Conversation

@paddybyers
Copy link
Copy Markdown
Member

@paddybyers paddybyers commented May 3, 2026

Summary

  • Proxy-based integration test specs: A programmable Go test proxy sits between the SDK and the Ably sandbox, enabling fault injection (drop connections, inject errors, delay responses) for testing connection recovery, heartbeat, auth reauth, presence reentry, and REST fault handling. The proxy is controlled via a REST API on a control port — each test creates a session with declarative rules, then inspects an event log after the test completes.
  • Reorganised UTS docs: Moved guides into docs/, rewrote README as a concise entry point, added writing-derived-tests.md for translating UTS specs to language-specific tests.
  • Added missing unit/integration UTS specs covering 14 spec areas that previously had no test coverage.

Proxy approach

The test proxy (uts/proxy/) is a Go binary that:

  1. Accepts sessions via POST /sessions — each session gets its own listener port and targets a specific upstream (sandbox-realtime/sandbox-rest)
  2. Supports declarative rules that match on message type, action, channel, HTTP method/path, and fire actions (drop, delay, inject message, return HTTP error)
  3. Supports imperative actions via POST /sessions/:id/actions for ad-hoc fault injection mid-test
  4. Records all events to a queryable log via GET /sessions/:id/log
  5. Sessions auto-cleanup on DELETE or timeout

SDK test harnesses (e.g. ably-js) build and auto-launch the proxy from their test setup — no manual proxy startup needed.

Proxy integration test specs added

Spec file Spec points Description
connection_resume.md RTN15a-j, RTN16d/k/l Connection resume, recovery key, fatal resume errors
channel_faults.md RTL2f, RTL4f, RTL13b Channel detach/reattach on transport failure
heartbeat.md RTN23a/b Heartbeat timeout detection
rest_faults.md RSC15a/d, TO3l8 REST fallback host selection
auth_reauth.md RSA4b/c/d, RTN22 Auth token renewal and server-initiated reauth
presence_reentry.md RTP17a-f Presence auto re-entry after resume
connection_open_failures.md (existing, extended) Connection open failure scenarios

Missing unit/integration specs added

Spec file Spec points Description
backoff_jitter_test.md RTB1a/b Backoff coefficient and jitter range
token_expiry_non_renewable_test.md RSA4a1/a2 Non-renewable token expiry handling
auth_callback_errors_test.md RSA4c/d/e/f Auth callback error scenarios
connection_recovery_test.md RTN16d-l Recovery key format and usage
forwards_compatibility_test.md RSF1/RTF1 Unknown protocol fields/actions
network_change_test.md RTN20a-c Network change events
push_channels.md (unit + integration) RSH7a-e Push channel subscriptions
channel_subscribe.md (extended) RTL22/MFI Message filter subscriptions
channel_publish.md (extended) RTN7e Publish error matches errorReason
rest_channel_attributes.md (extended) CHD2/CHM2 All ChannelMetrics fields

Test plan

  • Verify proxy binary builds: cd uts/proxy && go build -o test-proxy .
  • Review proxy test specs match features.md requirements
  • Verify completion-status.md reflects all new specs
  • Cross-check with ably-js test implementations (separate PR in ably-js)

🤖 Generated with Claude Code

paddybyers and others added 4 commits May 3, 2026 09:55
@paddybyers @claude
Fix mislabeled spec points in proxy integration test specs
9031eb7 
- channel_faults.md: Remove incorrect RTL4h reference; the test that
  replaces ATTACHED with ERROR is RTL14, not RTL4h. Update test name,
  channel name prefix, and rule comment accordingly.

- rest_faults.md: Fix spec point references from RSC15a to RSC15m/REC2c2
  (the correct spec points for fallback host behaviour when the fallback
  domain set is empty). Update test title, comments, and channel names.

- Update proxy binary.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@paddybyers @claude
Add tier 1+2 proxy integration test specs and proxy writing guidance
7f85583 
New proxy test specs:
- Test 21 (RTN15j): Fatal ERROR mid-session → FAILED state
- Test 22 (RTN15g/g2): connectionStateTtl expiry prevents resume
- Test 23 (RTN19a/a2): Unacked messages resent after resume
- Test 24 (RTL12): ATTACHED with resumed=false → channel UPDATE event
- Test 25 (RTL3d): Channels reattach after connection recovery
- Test 26 (RTN22/RTC8a): Server-initiated re-authentication
- Test 27 (RTP17i/RTP17g): Presence re-enter on non-resumed reattach

Also adds "Writing Proxy Tests" guidance to integration-testing.md
covering late fault injection and two approaches for early faults.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@paddybyers @claude
Reorganise UTS docs: move guides to docs/, rewrite README
f4751ed 
Move completion-status.md, integration-testing.md, and the two skill
files (write-test-spec, write-derived-tests) into a new uts/docs/
directory. Strip skill frontmatter from the writing guides. Fix stale
integration-proxy/ references to match actual integration/proxy/ layout.
Rewrite README to reflect current spec counts and link to docs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@paddybyers @claude
Add UTS specs for missing test coverage
c8f5b47 
New specs:
- RTB1: backoff and jitter for connection retries
- RSA4a: token expiry with non-renewable tokens
- RSA4c/d/f: auth callback error handling
- RTN16: connection recovery (recovery key, msgSerial, channelSerials)
- RTN20: network change events (browser-only)
- RSF1/RTF1: forwards compatibility (unknown fields/actions)
- RSH7: push channel subscriptions (unit + integration)

Extended specs:
- RTL22/MFI: message filter subscriptions (channel_subscribe)
- RTN7e: error reason on publish failure (channel_publish)
- CHD2/CHM2: all ChannelMetrics fields (rest_channel_attributes)
- RTN16d/RTN16l: proxy-based recovery tests (connection_resume)

Updates completion-status.md and README spec counts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 3, 2026 19:38 Inactive
@paddybyers paddybyers mentioned this pull request May 3, 2026
Open
4 tasks
@paddybyers @claude
Add RSC15f test: expired fallback not resurrected by late in-flight s…
fcfb8a5 
…uccess

Tests that a request completing successfully against a cached fallback
host after fallbackRetryTimeout has expired does not re-pin that host.
Uses the existing onRequest handler with a held PendingRequest pattern
rather than introducing a new mock primitive.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 3, 2026 19:50 Inactive
@paddybyers @claude
Add RTN15a proxy test variant for TCP close without WebSocket close f…
22e5304 
…rame

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 4, 2026 08:11 Inactive
@paddybyers @claude
Remove proxy source code; now lives at ably/uts-proxy
f4cebc8 
The Go test proxy has moved to https://github.com/ably/uts-proxy.
Updated references in README, writing guide, and proxy infrastructure
spec to point to the external repo.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 4, 2026 08:39 Inactive
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 4, 2026 10:22 Inactive
@paddybyers @claude
Fix RSA4c3 test: auth failure while CONNECTED should not set errorReason
021fdb3 
Per specification#466, RSA4c1 (setting errorReason with code 80019) should
only apply to RSA4c2 (CONNECTING case), not RSA4c3 (CONNECTED case).

When auth fails while CONNECTED, the connection is still healthy — the
existing token is valid. Setting errorReason with no state change is
misleading. The failure will naturally surface when the token expires.

Changes:
- RSA4c3 test now asserts errorReason is null and no events are emitted
- RSA4c1 references replaced with RSA4c2 throughout (error definition
  absorbed into RSA4c2)
- Notes updated to explain the rationale

Ref: #466

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 5, 2026 21:12 Inactive
@paddybyers @claude
Fix RTN24 tests: connectionId is not inside connectionDetails
ad58a63 
connectionId is a top-level ProtocolMessage field, not inside
connectionDetails. RTN24's "connectionDetails must override stored
details" does not apply to it — connection.id never changes for an
in-progress connection.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 5, 2026 21:48 Inactive
@paddybyers @claude
Fix TM3/TM4 UTS tests: correct spec point interpretations
007894c 
TM3 is about fromEncoded/fromEncodedArray, not generic fromJson.
TM4 is about Message constructors, not toJson serialization.
TM5 is about MessageAction enum, not message equality.

Removed toJson tests (no spec point requires a toJson method) and
replaced TM4 section with constructor tests per spec.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 5, 2026 22:04 Inactive
paddybyers and others added 4 commits May 5, 2026 23:42
@paddybyers @claude
Fix batch operation response format in UTS specs (RSC24, RSA17c)
1f9efc6 
With X-Ably-Version >= 3, the server returns a BatchResult envelope
({successCount, failureCount, results}) with HTTP 200 for all batch
responses including mixed success/failure. The UTS specs were
incorrectly mocking the legacy format (plain arrays + batchResponse
with HTTP 400) which is only used without a version header.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@paddybyers @claude
Add UTS spec for REST fallback proxy integration tests (RSC15l2, RSC1…
7fac103 
…5l4)

Proxy-based tests that exercise fallback behaviour through the real HTTP
client: request timeout triggers fallback (RSC15l2), and CloudFront
Server header should trigger fallback (RSC15l4).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@paddybyers @claude
Use endpoint instead of deprecated restHost in proxy fallback spec
12b06c9 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@paddybyers @claude
Add comprehensive REST proxy integration tests for HTTP error handling
2286d2a 
Adds tests for unreachable endpoint, connection drop, 5xx with/without
error body, 4xx not retried, and RSL1k4 idempotent publish (pending
proxy enhancement). Extracts token auth helper used across all tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@paddybyers